searching for experimental data

Searching for and retrieving data

Version 5

R. E. Ghosh, November 2006

Data from Instruments - ASCII format (post 1994)

The raw data are automatically transferred and stored on a data-server serdon. This also holds data from the previous reactor cycle. The server path is defined on standard workstations and the directories can be examined with standard commands.

IDA - Internet Data Access

The easiest way to examine these directories of data is to use barns.ill.fr from your web browser.

On standard workstations data may be found in the directory trees from

 
/usr/illdata/data/instrument         for the current cycle
/usr/illdata/data-1/instrument       for the preceding cycle
/usr/illdata/975/instrument          for cycle 975 (compressed data)
                                            etc.

Certain instruments, e.g. D16 have very large numbers of files, which are subdivided:

/usr/illdata/data/d16/d16_0    for files 000001 to 009999
/usr/illdata/data/d16/d16_1              010000 to 019999  etc

Older data are stored in compressed format. Filenames terminate in .Z, and must be copied onto your own disks before using the uncompress command: the uncompressed file replaces the .Z file locally e.g. for a short sequence...

        %cp /usr/illdata/951/d11/00437*.Z .
        %uncompress 00437*.Z

Catalogue

The regularly updated file /usr/illdata/data/DATA_CATALOG permits experimenters to find where their data files are stored.

Experimenters may use unix commands (cat,more,grep,awk...) to search files in the fashion of an SQL command.

Note: all input is case sensitive

Examples

example 1:to find where are in5 data files with date:JUN
---------
Use grep command:  
grep "in5" /usr/illdata/data/DATA_CATALOG | grep "JUN"
-----=====---------------------------=====
cycle   exp   numor   numor   from date        to date          identifier

  953   in5   002656  002688  10-JUN-95 10:29  13-JUN-95 09:04  PRAGERH
  953   in5   002690  002690  13-JUN-95 19:49  13-JUN-95 19:49  Herma
  953   in5   002691  002694  13-JUN-95 20:51  14-JUN-95 07:54  HERMA
  953   in5   002695  002722  14-JUN-95 14:25  19-JUN-95 08:45  DERIU MF
  953   in5   002723  002770  19-JUN-95 11:28  23-JUN-95 08:34  COULOMMF
  953   in5   002771  002796  23-JUN-95 14:43  27-JUN-95 09:03  J SuckJCC
  953   in5   002797  002831  27-JUN-95 10:42   3-JUL-95 09:03  FGJPR AJD   

example 2:to find in5 data files with date:JUN and experimenter name Herma.
---------
Use grep command:    
grep "in5" /usr/illdata/data/DATA_CATALOG | grep "JUN" | grep "Herma"
-----=====---------------------------=====--------=======
  953   in5   002689  002689  13-JUN-95 18:02  13-JUN-95 18:02  Herma HERM
  953   in5   002690  002690  13-JUN-95 19:49  13-JUN-95 19:49  Herma     

awk -F '/in5/&&/JUN/{print $0}'  /usr/illdata/data/DATA_CATALOG
--------============------------------------------
  953   in5   000009  002629   6-JUN-95 15:34   7-JUN-95 08:33  Cook  Ferr
  953   in5   002630  002655   7-JUN-95 12:39  10-JUN-95 08:50  ENSS  MF
  953   in5   002656  002688  10-JUN-95 10:29  13-JUN-95 09:04  PRAGERHERM
  953   in5   002689  002689  13-JUN-95 18:02  13-JUN-95 18:02  Herma HERM
  953   in5   002690  002690  13-JUN-95 19:49  13-JUN-95 19:49  Herma
  953   in5   002691  002694  13-JUN-95 20:51  14-JUN-95 07:54  HERMA
  953   in5   002695  002722  14-JUN-95 14:25  19-JUN-95 08:45  DERIU MF
  953   in5   002723  002770  19-JUN-95 11:28  23-JUN-95 08:34  COULOMMF
  953   in5   002771  002796  23-JUN-95 14:43  27-JUN-95 09:03  J SuckJCC
  953   in5   002797  002831  27-JUN-95 10:42   3-JUL-95 09:03  FGJPR AJD
 
example 4:to find in5 data files with date:JUN or with date:JUL
---------
Use the awk unix command:

awk -F '((/in5/&&/JUN/)||(/in5/&&/JUL/)){print $0}' /usr/illdata/data/DATA_CATALOG
--------================================------------------------------
  953   in5   000009  002629   6-JUN-95 15:34   7-JUN-95 08:33  Cook  Ferr
  953   in5   002630  002655   7-JUN-95 12:39  10-JUN-95 08:50  ENSS  MF
  953   in5   002656  002688  10-JUN-95
  953   in5   002689  002689  13-JUN-95 18:02  13-JUN-95 18:02  Herma HERM
  953   in5   002690  002690  13-JUN-95 19:49  13-JUN-95 19:49  Herma
  953   in5   002691  002694  13-JUN-95 20:51  14-JUN-95 07:54  HERMA
  953   in5   002695  002722  14-JUN-95 14:25  19-JUN-95 08:45  DERIU MF
  953   in5   002723  002770  19-JUN-95 11:28  23-JUN-95 08:34  COULOMMF
  953   in5   002771  002796  23-JUN-95 14:43  27-JUN-95 09:03  J SuckJCC
  953   in5   002797  002831  27-JUN-95 10:42   3-JUL-95 09:03  FGJPR AJD
  953   in5   002832  002850   3-JUL-95 10:25   4-JUL-95 22:54  BERND MF
  953   in5   002851  002929   5-JUL-95 02:01   7-JUL-95 20:12  BA,JZLMF
  953   in5   002930  002952  10-JUL-95 02:55  13-JUL-95 07:24  VRGZ  MF
  953   in5   002953  003007  13-JUL-95 10:38  19-JUL-95 15:27  CORNELGJK
  953   in5   003008  003017  19-JUL-95 20:09  21-JUL-95 07:59  VRGZ  MF
  953   in5   003018  003020  21-JUL-95 08:57  21-JUL-95 11:26  HGBGJKHGB
  953   in5   003021  003037  21-JUL-95 14:06  23-JUL-95 08:27  COOK  COOK

example 5: to find all experimenters with name beginning by "O" or "o"
---------
use awk unix command:

awk -F' ' '($9~/^O/)||($9~/^o/){print $0}' /usr/illdata/data/DATA_CATALOG  
-----------------=----------=--------------------------
  951   d17   003411  003583  10-MAR-95 18:21  13-MAR-95 10:28  OTTEWILL
  951   d7    001284  001286  04-MAR-95 15:28  04-MAR-95 20:18  OVCUR0
  952   d17   004547  004598  18-APR-95 17:01  20-APR-95 08:23  oeser/mont
  952   d1b   021515  022229  25-APR-95 10:46  27-APR-95 08:43  OULADDGOME
  952   d1b   022230  022419  27-APR-95 10:14  29-APR-95 08:19  OULADDHUBE
  952   d1b   022420  023238  29-APR-95 10:22   2-MAY-95 08:30  OULADDSAYE
  952   d22   012759  012761  19-APR-95 22:25  19-APR-95 22:52  oeser
  952   d22   012762  013159  20-APR-95 18:43  24-APR-95 19:43  Ottewill
  952   d7    001534  001534  02-MAY-95 20:54  02-MAY-95 20:54  OFFENV
  952   d17   004547  004598  18-APR-95 17:01  20-APR-95 08:23  oeser/mont
  952   d1b   021515  022229  25-APR-95 10:46  27-APR-95 08:43  OULADDGOME
  952   d1b   022230  022419  27-APR-95 10:14  29-APR-95 08:19  OULADDHUBE
  952   d1b   022420  023238  29-APR-95 10:22   2-MAY-95 08:30  OULADDSAYE
  952   d22   012759  012761  19-APR-95 22:25  19-APR-95 22:52  oeser
  952   d22   012762  013159  20-APR-95 18:43  24-APR-95 19:43  Ottewill
  953   d11   015394  015632  16-JUL-95 09:47  18-JUL-95 09:21  Ottewill

Data transcribed from pre-1994 binary

The archive and transcription tools were maintained on the Alpha-VMS cluster, and data have been progressively transcribed to ASCII files similar to those in current use. These data are held on-line in directories as described above. Notably for SANS, this archive matches current standards and data may be examined with present tools from year/cycle 743 and 771 for D11 and D17 respectively. Since these data have not all been catalogued requests should be sent to the Service Informatique concerning these data.