How to use LSTOSA#

The main script is osa/scripts/sequencer.py, which reads the Night Summary, creates the analysis directory, extract run sequences and launches them using sbatch.

First, activate osa environment and go to the then run sequencer.py specifying the date and the telescope, for the time being only LST1 is available.

(osa) [daniel.morcuende@cp02 lstosa]$ python osa/scripts/sequencer.py -c cfg/mysequencer.cfg -d 2020_01_27 LST1
============================== Starting sequencer.py at 2021-01-01 20:10:32 UTC for LST, Telescope: LST1, Night: 2020_01_27 ==============================
Tel   Seq  Parent  Type  Run   Subruns  Source  Wobble  Action  Tries  JobID  State  Host  CPU_time  Walltime  Exit  DL1%  DATACHECK%  MUONS%  DL2%
LST1    0  None    CALI  1873        5  None    None    None        0  None   None   None  None      None      None  None  None        None    None
LST1    1       0  DATA  1874      194  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    2       0  DATA  1875      209  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    3       0  DATA  1876      225  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    4       0  DATA  1877      202  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    5       0  DATA  1878       74  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    6       0  DATA  1879      207  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    7       0  DATA  1880      203  None    None    None        0  None   None   None  None      None      None     0           0       0     0
LST1    8       0  DATA  1881      207  None    None    None        0  None   None   None  None      None      None     0           0       0     0

A table containing all the sequences (or runs) for a given day is shown. You can also monitor the status of the processing of the file, stated in the last columns. When launching the sequencer script, jobs are submitted for each run listed in the table. First a CALIBRATION sequence is executed (it takes around 1 hour) and then rest of the DATA sequences (they also take around 1 hour in total to go from R0 to DL2). Data sequences depend on the calibration sequence and they only start whenever the parent calibration sequence finishes without problems.

If you want to use LSTOSA from your personal account @ LST-IT container, you just need to set the proper paths in the sequencer.cfg config file.

Useful options:

  • Before executing sequencer to submit any job, it is recommended to use first the simulate option -s. With this option enabled, the sequencer is just simulated and no jobs are launched.

  • To know more about what happens internally when you run sequencer.py use the verbose mode -v. An example output is shown below:

(osa) [daniel.morcuende@cp02 lstosa]$ python osa/scripts/sequencer.py -v -c cfg/mysequencer.cfg -s -d 2020_01_27 LST1
============================== Starting sequencer.py at 2021-01-01 20:10:32 UTC for LST, Telescope: LST1, Night: 2020_01_27 ==============================
2021-01-01 20:10:32,473 DEBUG [osa.utils.utils] (utils.getnightdirectory): Getting analysis path for tel_id LST1
2021-01-01 20:10:32,473 DEBUG [osa.utils.utils] (utils.get_prod_id): Getting the prod ID for the running analysis directory: v0.6.3_v05
2021-01-01 20:10:32,473 DEBUG [osa.utils.utils] (utils.getnightdirectory): Analysis directory: /fefs/aswg/workspace/daniel.morcuende/data/real/running_analysis/20200127/v0.6.3_v05
2021-01-01 20:10:32,474 DEBUG [osa.nightsummary.nightsummary] (nightsummary.read_nightsummary): Night summary file path: /home/daniel.morcuende/lstosa/NightSummary/NightSummary_20200127.txt
2021-01-01 20:10:32,474 DEBUG [osa.nightsummary.nightsummary] (nightsummary.read_nightsummary): Night summary:
 01872    5 DRS4  2020-01-27 19:51:44 0001 1580154753739954334 5739954100 0001 1580154753739954334 5739951300
 01873    5 CALI  2020-01-27 20:23:43 0001 1580156670887160057 1887159800 0001 1580156670887160057 1887158800
 01874  194 DATA  2020-01-27 20:44:13 0003 1580157904186709543 5186709300 0003 1580157904186709543 5186708700
 01875  209 DATA  2020-01-27 21:05:44 0003 1580159197411578464 7411578200 nan nan nan
 01876  225 DATA  2020-01-27 21:27:20 0001 1580160490575729635 7575729400 nan nan nan
 01877  202 DATA  2020-01-27 21:51:28 0001 1580161935735383476 1735383200 nan nan nan
 01878   74 DATA  2020-01-27 22:13:34 0001 1580163263237149740 2237149500 nan nan nan
 01879  207 DATA  2020-01-27 22:33:06 0003 1580164436408793971 3408793700 nan nan nan
 01880  203 DATA  2020-01-27 22:55:31 0003 1580165786211720504 7211720200 nan nan nan
 01881  207 DATA  2020-01-27 23:17:52 0001 1580167122989548546 3989548300 nan nan nan

2021-01-01 20:10:32,477 DEBUG [osa.nightsummary.extract] (extract.extractsubruns): Subrun list extracted
2021-01-01 20:10:32,477 DEBUG [osa.nightsummary.extract] (extract.extractruns): Run list extracted
2021-01-01 20:10:32,477 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new PED run 1872 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new CAL run 1873 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1874 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1875 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1876 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1877 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1878 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1879 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1880 for None
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Detected a new DATA run 1881 for None
...
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): The storage contains 8 data sequences
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1872, type DRS4 to sequence 0
2021-01-01 20:10:32,478 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1873, type CALI to sequence 0
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Sequence 0 assigned to run 1873 whose parent is None with run 1872
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1874, type DATA to sequence 1
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Sequence 1 assigned to run 1874 whose parent is 0 with run 1873
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1875, type DATA to sequence 2
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Sequence 2 assigned to run 1875 whose parent is 0 with run 1873
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1876, type DATA to sequence 3
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Sequence 3 assigned to run 1876 whose parent is 0 with run 1873
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1877, type DATA to sequence 4
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Sequence 4 assigned to run 1877 whose parent is 0 with run 1873
2021-01-01 20:10:32,479 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): trying to assign run 1878, type DATA to sequence 5
...
2021-01-01 20:10:32,480 DEBUG [osa.nightsummary.extract] (extract.generateworkflow): Workflow completed
2021-01-01 20:10:32,480 DEBUG [osa.nightsummary.extract] (extract.extractsequences): Sequence list extracted
2021-01-01 20:10:32,480 DEBUG [osa.jobs.job] (job.preparejobs): Creating sequence.txt and sequence.py for sequence 0
2021-01-01 20:10:32,480 DEBUG [osa.jobs.job] (job.createsequencetxt): SIMULATE Creating sequence txt /fefs/aswg/workspace/daniel.morcuende/data/real/running_analysis/20200127/v0.6.3_v05/sequence_LST1_01873.txt
2021-01-01 20:10:32,480 DEBUG [osa.jobs.job] (job.preparejobs): Creating sequence.txt and sequence.py for sequence 1
...
2021-01-01 20:10:32,596 DEBUG [osa.jobs.job] (job.submitjobs): SIMULATE Launching scripts
2021-01-01 20:10:32,596 DEBUG [osa.jobs.job] (job.submitjobs): ['sbatch', '--parsable', '--export=ALL,MPLBACKEND=Agg', '/fefs/aswg/workspace/daniel.morcuende/data/real/running_analysis/20200127/v0.6.3_v05/sequence_LST1_01873.py']
2021-01-01 20:10:32,596 DEBUG [osa.jobs.job] (job.submitjobs): Adding dependencies to job submission