DL1 to DL2 tool

[vuillaut]

In an attempt to solve #1340 , I transformed the DL1 to DL2 script into a ctapipe tool.

I also added functions to write and read the provenance into an HDF5 file.
Advantages:

• the provenance can't be lost, it's inside the file
• it add an unambiguous 'provenance' group where each tool can write, without overwriting other provenances, and keeping track of older ones

Drawbacks:

• the provenance can't be read using pytables (or by extension vitables) because it does not handle variable length strings:

  variable length strings are not supported yet

The leaf will become an ``UnImplemented`` node.

• Other kind of files are produced by lstchain (.fits, .sav for the models...) and might not be suitable to save the provenance

[morcuended]

Shall we move it to the tools/ folder?

[morcuended]

For the CI/docs action, remove the script entry from docs/lstchain_api/scripts/index.rst

.. _lstchain_dl1_to_dl2:

lstchain_dl1_to_dl2
+++++++++++++++++++

.. automodule:: lstchain.scripts.lstchain_dl1_to_dl2

Usage
-----
.. argparse::
   :module: lstchain.scripts.lstchain_dl1_to_dl2
   :func: parser
   :prog: lstchain_dl1_to_dl2

and add to docs/lstchain_api/tools/index.rst

.. automodapi:: lstchain.tools.lstchain_dl1_to_dl2
   :no-inheritance-diagram:

[moralejo]

Shall we move it to the tools/ folder?

I see no clear advantage in that, and it might make some doc (or school presentation) obsolete. The way of executing it is the same (well, it has additional options, but the basic commands are the same), so the explanations on how to run it will still be ok.

[moralejo]

Drawbacks:

• the provenance can't be read using pytables (or by extension vitables) because it does not handle variable length strings:

  variable length strings are not supported yet

The leaf will become an ``UnImplemented`` node.

How does one read the provenance, then?

[moralejo]

I think tests fail because the case in which no config file is explicitly provided the default is not taken.
Seems it should be, but somehow does not work

[morcuended]

How does one read the provenance, then?

I'd say it is:

from lstchain.io.provenance import read_dl2_provenance
read_dl2_provenance(dl2_hdf5_file_path)

I would add it to the documentation.

I see no clear advantage in that, and it might make some doc (or school presentation) obsolete. The way of executing it is the same (well, it has additional options, but the basic commands are the same), so the explanations on how to run it will still be ok.

I would still do it for internal consistency with the rest of scripts/tools. The change will not have any impact on the users' side. Just some modifications in the API docs would be needed.

[moralejo]

traits.Path does not take None as default argument... Any idea how to solve this?

[moralejo]

traits.Path does not take None as default argument... Any idea how to solve this?

Should be solved now

[moralejo]

How does one read the provenance, then?

I'd say it is:

from lstchain.io.provenance import read_dl2_provenance
read_dl2_provenance(dl2_hdf5_file_path)

I would add it to the documentation.

Ok!

I see no clear advantage in that, and it might make some doc (or school presentation) obsolete. The way of executing it is the same (well, it has additional options, but the basic commands are the same), so the explanations on how to run it will still be ok.

I would still do it for internal consistency with the rest of scripts/tools. The change will not have any impact on the users' side. Just some modifications in the API docs would be needed.

Ook... @vuillaut , can you do these two changes, so we can move on?

[moralejo]

There is also the issue of docs failing. @morcuended, will that be solved also with the suggestions you made?

[morcuended]

Let me check locally

[moralejo]

traits.Path does not take None as default argument... Any idea how to solve this?

Should be solved now

CI tests fail now somewhere else... seems a dl2 test file is not produced or something like that.
Update: it seems that unlike with the old script, with the tool one test in which two input files are passed (instead of just one) fails.

[codecov]

Codecov Report

Attention: Patch coverage is 87.39496% with 15 lines in your changes missing coverage. Please review.

Project coverage is 72.87%. Comparing base (7f3979e) to head (57af1bb).
Report is 23 commits behind head on main.

Files with missing lines	Patch %	Lines
lstchain/tools/lstchain_dl1_to_dl2.py	83.33%	11 Missing ⚠️
lstchain/io/provenance.py	83.33%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1343      +/-   ##
==========================================
+ Coverage   72.77%   72.87%   +0.10%     
==========================================
  Files         135      137       +2     
  Lines       14429    14511      +82     
==========================================
+ Hits        10500    10575      +75     
- Misses       3929     3936       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[vuillaut]

Hi @moralejo, @morcuended,
Sorry I was on leave until now, thank you for taking care of last changes and merging !