Components#
LSTOSA is a set of python scripts and cron jobs that pipes the different analysis steps of lstchain, using SLURM as
resource manager. It is thought to run on the LST IT container at La Palma using the lstanalyzer
account.
lstchain#
lstchain is the analysis library developed for the commissioning of the LST-1 prototype. It is heavily based on the the Prototype CTA Pipeline Framework ctapipe.
SLURM#
SLURM is the resource manager used at the LST IT container. Most LSTOSA scripts, specially those devoted to analyze
data (heavy duty ones) run through SLURM. Its capabilities are used to send chained jobs, that are executed only after
the parent job finished. This is used for calibration sequences, whose end triggers the analysis sequences for the
DATA
runs. The code in job.py
interacts with the resource manager.
More information: https://slurm.schedmd.com/
Cron Jobs#
A set of cron jobs automatize LSTOSA execution.
Production of NightSummary file, launched each morning at 7 UTC. They are currently not produced by LSTOSA, but generated independently by the Data-Check software being developed by I. Aguado. These files sometimes need to be modified to assure that only one DRS4 and CALIBRATION runs are present each day. Therefore next steps cannot be fully automatized until runs are properly and reliably tagged.
Sequencer: Cron job for sequencer script to be implemented.
Closer: Cron job for closer script to be implemented.
Sequencer#
sequencer.py
is the main script controlling LSTOSA execution. It takes as input a configuration file and a
NightSummary.txt
file. The first one contains all the needed parameters and paths. The NightSummary
file
contains the list of runs to process, tagged as DRS4
, CALIBRATION
or DATA
based on the percentage of
pedestal events in each sub-run file. Each run forms a so-called sequence. Sequencer builds a job for each run,
depending on its type the job will call either calibrationsequence.py
or datasequence.py
. Each job is an array
job which processes all the sub-runs contained in the run in parallel.
It uses nightsummary.py
and extract.py
to read the NightSummary.txt
file and extract the sequences.
Closer#
The closer
is an error handler and closer for LSTOSA. It processes the sequencer table and closes the
successfully analyzed sequences or tries to solve known issues in an automatic way. For details see closer.py
.
Provenance#
The data analysis steps executed to create DL1 and DL2 level data are captured for each run, together with the
configuration parameters and files needed as well as intermediate files produced. This information is serialized in
.json
formatted files, following the IVOA Provenance Model Recommendation [IVOAProvenance]. Provenance
graphs are also provided in .pdf
formatted files, rendering a detailed complete view of the data analysis
process which improves process inspection and helps achieving reproducibility. Tracking of the calibration steps
will be implemented shortly, and a more detailed provenance query tool is also foreseen, which would need to store
the provenance information in a database.
HighLevel#
The production of DL3 file is not implemented yet. Further high-level analysis as obtaining the significance level of the source detection, sky maps, spectra and light curves are to be implemented.
MySQL Database#
Not implemented yet.