Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement suggestions for CMSSW_14_1_X PR #45665 #137

Merged
merged 25 commits into from
Aug 27, 2024

Conversation

IzaakWN
Copy link

@IzaakWN IzaakWN commented Aug 20, 2024

[Work in progress.]

Implement suggestions by @makortel in PR cms-sw#45665 to official CMSSW:

Needs to be run together with hgcal-dpg/hgcal-comm!45.

@IzaakWN
Copy link
Author

IzaakWN commented Aug 21, 2024

@yulunmiao & @hqucms, can you have a look at specifically the changes to the enumeration/constants, please?

Following Matti's comments, I changed the enumerators

 enum ECONDFlag {
     BITT_POS = 0,
     BITM_POS = 1,
     EBO_POS = 2,
     EBO_MASK = 0b11,
     HT_POS = 4,
     HT_MASK = 0b11,
     BITE_POS = 6,
     BITS_POS = 7
   };

to constexpr uint8_t in the following way.

  namespace ECONDFlag {
    constexpr uint8_t BITT_POS = 0, BITM_POS = 1, EBO_POS = 2, EBO_MASK = 0b11, HT_POS = 4, HT_MASK = 0b11,
                      BITE_POS = 6, BITS_POS = 7;
  }

(Unfortunately, the CMSSW code formatting converts the line breakes after commas to a single wrapped line as above, which is a bit ugly. Alternatively, we can write each constant as constexpr uint8_t ... = ...;.)

Choices of data types for the others are as follows:

namespace hgcal {
  namespace econd {
    namespace ToTStatus {
      constexpr uint8_t ZeroSuppressed = 0x0, ...;
    }  // namespace ToTStatus
  }    // namespace econd

  namespace backend {
    namespace ECONDPacketStatus {
      constexpr uint8_t Normal = 0x0, ...;
    }  // namespace ECONDPacketStatus
  }    // namespace backend

  namespace ECOND_FRAME {
    constexpr uint32_t HEADER_POS = 23, ...;
  }  // namespace ECOND_FRAME

  namespace BACKEND_FRAME {
    constexpr uint32_t CAPTUREBLOCK_RESERVED_MASK = 0x7f, ...;
  }  // namespace BACKEND_FRAME

  namespace DIGI_FLAG {
    constexpr uint16_t
        FULL_READOUT = 0x0000, ...;
  }  // namespace DIGI_FLAG

}  // namespace hgcal

@yulunmiao
Copy link

Hi @IzaakWN, thank you for making the changes, I think the changes look good to me

Copy link

@pfs pfs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All looks reasonable to me, thanks for implementing!

@IzaakWN IzaakWN mentioned this pull request Aug 21, 2024
@IzaakWN
Copy link
Author

IzaakWN commented Aug 27, 2024

Tested with

ssh lxplus9
source /cvmfs/cms.cern.ch/cmsset_default.sh

# install
export SCRAM_ARCH="el9_amd64_gcc12"
cmsrel CMSSW_14_1_0_pre4
cd CMSSW_14_1_0_pre4/src/
cmsenv

# checkout our development branch
git cms-merge-topic -u CMS-HGCAL:dev/CMSSW_14_1_X_PR_suggestions

# load external tools & data
git clone https://gitlab.cern.ch/hgcal-dpg/hgcal-comm.git HGCalCommissioning
git fetch origin CMSSW_14_1_X_PR_suggestions
git checkout CMSSW_14_1_X_PR_suggestions

# compile
scram b -j 10

# test that things are up and running (run unit tests)
scram b runtests

# if debug information is needed, instead of `scram b -j 10`, do
#USER_CXXFLAGS=-DEDM_ML_DEBUG scram b -j 10

cmsRun $CMSSW_BASE/src/HGCalCommissioning/SystemTestEventFilters/test/test_raw2reco.py charMode=0 dqmOnly=True prescale=100 maxEvents=5000 output=output_RunRUN.root inputFiles=/eos/cms/store/group/dpg_hgcal/tb_hgcal/2024/BeamTestAug/HgcalBeamtestAug2024/Relay1722965128/Run1722965131_Link1_File000000000[01].bin

@pfs pfs merged commit 7ea653e into dev/hackathon_base_CMSSW_14_1_X Aug 27, 2024
@pfs pfs deleted the dev/CMSSW_14_1_X_PR_suggestions branch August 27, 2024 13:32
pfs added a commit that referenced this pull request Aug 27, 2024
pfs added a commit that referenced this pull request Sep 30, 2024
* Implement TH2Poly in DQM Services for HGCal DQM
* Apply changes from code quality checks
* Add a patch based on makortel's comments
* enum -> namespace + constexpr uint{8,16,32}_t
* Matti suggestion: remove '_t' postfix from HGCalFEDReadoutSequence_t, HGCalROCConfig_t, HGCalECONDConfig_t, HGCalFedConfig_t structs
* Matti suggestions & code checks: remove redundant comments; silence debugging cout or use MessageLogger; move includes; remove trailing '_' from function arguments
* Matti suggestion: reuse typecodeMap_.find() to avoid searching twice (TODO: throw exception if not found?)
* Matti suggestion: use SET_PORTABLEHOSTCOLLECTION_READ_RULES; add ClassVersion+checksum
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.; rm redundant includes & couts; pass reference to fill_SoA_column to avoid copy
* Matti suggestions: use SET_PORTABLEHOSTCOLLECTION_READ_RULES for HGCalRecHitHost; Remove deprecated HGCalFlaggedECONDInfo* & comments;
* Matti suggestion: throw exception if type code not found; replace for-loop with std::accumulate
* Matti suggestion: remove TestObjects
* Matti suggestions: remove config as member data and produce unique_ptr instead of shared_ptr; NULL → nullptr; initialize optional override; memcp -> std::copy
* Matti suggestions: use file in path
* removing soafiller and updating buildfile (#139)
* address comments on HGCalRecHitCalibrationAlgorithms.dev.cc
* irot as uint8_t instead of char
* restore DQM without TH2Poly
* apply code formats

---------

Co-authored-by: ywkao <ywkao@hep1.phys.ntu.edu.tw>
Co-authored-by: Pedro <psilva@cern.ch>
pfs pushed a commit that referenced this pull request Sep 30, 2024
Code format HGCalMappingESSourceTester.cc

Removing unused includes and delarations after PR comments

Correction: missing :: for namespace

Fix code style indentation

record descriptions

Re-introducing headers.h file needed for serialization classes

remove data and change locations to cms-data folder

moving to single record as all conds produced have the same IOV

re-introducing 1 record per payload, addressing additional code comments

moving to printf, removing unused product, and message logger

code checks

remove unused iomanip

adding digi SoAs

update collection names

moving emulators to SimCalorimetry/HGCalSimAlgos, adapting paths

moving digi soa filler to alpaka and adding extra tools

some progress with soafiller

updating

fixing but in module es producer (assignment of capture block index instead of capture block)

add dense index map for HGCalElectronicsId

add getMaxDataSize & getMaxERxSize functions

fix getDenseIndex bug; add typcode map to ModuleIndexer (to retrieve module dense index offset); use regexp for type code

add getFedAndModuleIndex; rename getDenseIndex -> getIndexForModule(Data) for elecID

add calibrations Alpaka ESProducer

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

int -> uint32_t; uncomment debugging print out

clean

add HGCal configuration ESProducers + record + tester

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

rename HGCal*ParameterHostCollection.cc -> HGCal*ParameterDeviceCollection.cc for clarity

rename 'HGCal.*(Device|Host)Collection -> (Device|Host)' for readability; move calib SoAs to CondFormats/HGCalObjects/; move TestHGCalRecHitESProducers to test

rename 'HGCal.*(Device|Host)Collection -> HGCal.*(Device|Host)' for readability

add HGCalRecHit

add DIGI2RECO calibration algorithms (see https://gitlab.cern.ch/hgcal-dpg/hgcal-comm/-/merge_requests/3)[C; remove detId/elecId from SoAs (should be redundant since common indexer); add HGCROCChannelDataFrame<HGCalDetId> to classes_def.xml

Move HGCalRawDataDefinitions.h.

[WIP] Refactor unpacker.

parse arrays of gain-dependent calibration parameters in eRx-blocks

add missing Noise attribute

updating usage of Rcd

fixing cfg files

Convert ECON-D payload to 32b words.

[Unpacker] rebase the unpacker on denseIdx and SoA digis

Update unpacker code.

SoA format of HGCal ECON-D header

Unpack header information to HGCalECONDInfoSoA; Add in quality checks for ECON-D

remove unused

add typecode regexp; pass typecode to modIndexer_.processNewModule

update ESProducers

update record

fix ESInputTags

prefix filename with T_EventSetup_

add HGCalConfiguration & HGCalConfigurationESProducer

Update and rename HGCalRecHitProducer.cc to HGCalRecHitProducers.cc

fill HGCal config structs for unpacker (header marks, charMode)

update calibration kernels

read HGCal FED & ECON-D configuration from JSON

Update unpacker to use HGCalConfiguration structure

make manual overrides truly optional

fix compilation issue due to initialization of const gain

Change to LogDebug

adding dense index info producer

adding hgcalmapping customise

fix misunderstanding in max data size expected from indexer

Implement TH2Poly in DQM based on 14e856f

Update HGCalConfigurationESProducer.cc

update unpacker and HGCalConfiguration

Add definitions of flags in digi SOA

update the other part to most recent commit

Add flags for passthrough ECON-Ds

Debug: loop through every denseIdx

Fix the number of channel in eRx;Add default flag for digi, representing digi not read in raw data

fully read config parameters from JSONs for config SoA; add sanity checks; add HGCalMappingModuleIndexer::getNFED; rename getFedAndModuleIndex -> getIndexForFedAndModule; clean code; ...

rename macro

silence debugging message in ESProducers

silence cout

add HGCalMappingModuleIndexer::getMaxModuleSize(fedid)

typo; SystemTestEventFilters/data -> Configuration/data

bugfix in the cm Sum

fixing dense index issue

Update comment in HGCalMappingParameterSoA.h

Some additional fixes (#132)

- remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester
- do not use moduleLUT in DenseIndexInfoESProducer!! it has the full dimensions before trimming to the actual number of ECON-Ds in a FED

Add parsing function for econd flag; renaming HGCalECONDInfo to HGCalECONDPacketInfo

Add irot to module mapper (#134)

* - remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester

* do not use moduleLUT in DenseIndexInfoESProducer!!

* Adding optional irot to module mapper file
- If existing it will be put in the module info SoA
- othwerwise a default irot=0 (baseline is used)

Suppress "unused variable" warning: remove CellIndexer in HGCalDigiSoaFiller.cc

apply code formats (#135)

Apply suggestions from code review

Apply easy suggestions by Matti. (Looking into other comment & suggestions.)

Co-authored-by: Matti Kortelainen <matti.kortelainen@cern.ch>

'const json' to fix compilation issue for Matti's suggestion

gain index should be taken as is, not subtracted by 1

make use of valid at rechit level / set rechit valids

fill rec hitflags

update kernel after closure

Implement suggestions for CMSSW_14_1_X PR cms-sw#45665 (#137)

* Implement TH2Poly in DQM Services for HGCal DQM
* Apply changes from code quality checks
* Add a patch based on makortel's comments
* enum -> namespace + constexpr uint{8,16,32}_t
* Matti suggestion: remove '_t' postfix from HGCalFEDReadoutSequence_t, HGCalROCConfig_t, HGCalECONDConfig_t, HGCalFedConfig_t structs
* Matti suggestions & code checks: remove redundant comments; silence debugging cout or use MessageLogger; move includes; remove trailing '_' from function arguments
* Matti suggestion: reuse typecodeMap_.find() to avoid searching twice (TODO: throw exception if not found?)
* Matti suggestion: use SET_PORTABLEHOSTCOLLECTION_READ_RULES; add ClassVersion+checksum
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.; rm redundant includes & couts; pass reference to fill_SoA_column to avoid copy
* Matti suggestions: use SET_PORTABLEHOSTCOLLECTION_READ_RULES for HGCalRecHitHost; Remove deprecated HGCalFlaggedECONDInfo* & comments;
* Matti suggestion: throw exception if type code not found; replace for-loop with std::accumulate
* Matti suggestion: remove TestObjects
* Matti suggestions: remove config as member data and produce unique_ptr instead of shared_ptr; NULL → nullptr; initialize optional override; memcp -> std::copy
* Matti suggestions: use file in path
* removing soafiller and updating buildfile (#139)
* address comments on HGCalRecHitCalibrationAlgorithms.dev.cc
* irot as uint8_t instead of char
* restore DQM without TH2Poly
* apply code formats

---------

Co-authored-by: ywkao <ywkao@hep1.phys.ntu.edu.tw>
Co-authored-by: Pedro <psilva@cern.ch>

restore release base DQM classes (#140)

move to chrono calls instead of struct inside namespace (#141)

address comments (#142)

update include (#143)

remove ECON-D tests as they'll need to be re-written in a different form (#144)

Declare HGCalRecHitCalibrationAlgorithms::calibrate as const method (#145)

* remove ECON-D tests as they'll need to be re-written in a different form

* enable const

patching unpacker for padding of capture blocks within FED

move check of padding to start of capture block

Fix the throw, so the processing won't stop at the first event with error

Update HGCalRecHitCalibrationAlgorithms.dev.cc

CM sum -> 0.5*CM sum

Change the throws to specific returns

Optimize the producer

Debugging the padding of eRx subpackets

roll back (#149)

apply code checks and remove dqm
IzaakWN pushed a commit that referenced this pull request Oct 8, 2024
Code format HGCalMappingESSourceTester.cc

Removing unused includes and delarations after PR comments

Correction: missing :: for namespace

Fix code style indentation

record descriptions

Re-introducing headers.h file needed for serialization classes

remove data and change locations to cms-data folder

moving to single record as all conds produced have the same IOV

re-introducing 1 record per payload, addressing additional code comments

moving to printf, removing unused product, and message logger

code checks

remove unused iomanip

adding digi SoAs

update collection names

moving emulators to SimCalorimetry/HGCalSimAlgos, adapting paths

moving digi soa filler to alpaka and adding extra tools

some progress with soafiller

updating

fixing but in module es producer (assignment of capture block index instead of capture block)

add dense index map for HGCalElectronicsId

add getMaxDataSize & getMaxERxSize functions

fix getDenseIndex bug; add typcode map to ModuleIndexer (to retrieve module dense index offset); use regexp for type code

add getFedAndModuleIndex; rename getDenseIndex -> getIndexForModule(Data) for elecID

add calibrations Alpaka ESProducer

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

int -> uint32_t; uncomment debugging print out

clean

add HGCal configuration ESProducers + record + tester

rename ES_HGCal*Parameter.cc -> HGCal*ParameterHostCollection.cc for clarity

rename HGCal*ParameterHostCollection.cc -> HGCal*ParameterDeviceCollection.cc for clarity

rename 'HGCal.*(Device|Host)Collection -> (Device|Host)' for readability; move calib SoAs to CondFormats/HGCalObjects/; move TestHGCalRecHitESProducers to test

rename 'HGCal.*(Device|Host)Collection -> HGCal.*(Device|Host)' for readability

add HGCalRecHit

add DIGI2RECO calibration algorithms (see https://gitlab.cern.ch/hgcal-dpg/hgcal-comm/-/merge_requests/3)[C; remove detId/elecId from SoAs (should be redundant since common indexer); add HGCROCChannelDataFrame<HGCalDetId> to classes_def.xml

Move HGCalRawDataDefinitions.h.

[WIP] Refactor unpacker.

parse arrays of gain-dependent calibration parameters in eRx-blocks

add missing Noise attribute

updating usage of Rcd

fixing cfg files

Convert ECON-D payload to 32b words.

[Unpacker] rebase the unpacker on denseIdx and SoA digis

Update unpacker code.

SoA format of HGCal ECON-D header

Unpack header information to HGCalECONDInfoSoA; Add in quality checks for ECON-D

remove unused

add typecode regexp; pass typecode to modIndexer_.processNewModule

update ESProducers

update record

fix ESInputTags

prefix filename with T_EventSetup_

add HGCalConfiguration & HGCalConfigurationESProducer

Update and rename HGCalRecHitProducer.cc to HGCalRecHitProducers.cc

fill HGCal config structs for unpacker (header marks, charMode)

update calibration kernels

read HGCal FED & ECON-D configuration from JSON

Update unpacker to use HGCalConfiguration structure

make manual overrides truly optional

fix compilation issue due to initialization of const gain

Change to LogDebug

adding dense index info producer

adding hgcalmapping customise

fix misunderstanding in max data size expected from indexer

Implement TH2Poly in DQM based on 14e856f

Update HGCalConfigurationESProducer.cc

update unpacker and HGCalConfiguration

Add definitions of flags in digi SOA

update the other part to most recent commit

Add flags for passthrough ECON-Ds

Debug: loop through every denseIdx

Fix the number of channel in eRx;Add default flag for digi, representing digi not read in raw data

fully read config parameters from JSONs for config SoA; add sanity checks; add HGCalMappingModuleIndexer::getNFED; rename getFedAndModuleIndex -> getIndexForFedAndModule; clean code; ...

rename macro

silence debugging message in ESProducers

silence cout

add HGCalMappingModuleIndexer::getMaxModuleSize(fedid)

typo; SystemTestEventFilters/data -> Configuration/data

bugfix in the cm Sum

fixing dense index issue

Update comment in HGCalMappingParameterSoA.h

Some additional fixes (#132)

- remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester
- do not use moduleLUT in DenseIndexInfoESProducer!! it has the full dimensions before trimming to the actual number of ECON-Ds in a FED

Add parsing function for econd flag; renaming HGCalECONDInfo to HGCalECONDPacketInfo

Add irot to module mapper (#134)

* - remove_if was missing a call to std::vector::erase to effectively remove unused entries
- typecodemap was missing a reset with the final indices
- usage of command line options was missing in module map tester

* do not use moduleLUT in DenseIndexInfoESProducer!!

* Adding optional irot to module mapper file
- If existing it will be put in the module info SoA
- othwerwise a default irot=0 (baseline is used)

Suppress "unused variable" warning: remove CellIndexer in HGCalDigiSoaFiller.cc

apply code formats (#135)

Apply suggestions from code review

Apply easy suggestions by Matti. (Looking into other comment & suggestions.)

Co-authored-by: Matti Kortelainen <matti.kortelainen@cern.ch>

'const json' to fix compilation issue for Matti's suggestion

gain index should be taken as is, not subtracted by 1

make use of valid at rechit level / set rechit valids

fill rec hitflags

update kernel after closure

Implement suggestions for CMSSW_14_1_X PR cms-sw#45665 (#137)

* Implement TH2Poly in DQM Services for HGCal DQM
* Apply changes from code quality checks
* Add a patch based on makortel's comments
* enum -> namespace + constexpr uint{8,16,32}_t
* Matti suggestion: remove '_t' postfix from HGCalFEDReadoutSequence_t, HGCalROCConfig_t, HGCalECONDConfig_t, HGCalFedConfig_t structs
* Matti suggestions & code checks: remove redundant comments; silence debugging cout or use MessageLogger; move includes; remove trailing '_' from function arguments
* Matti suggestion: reuse typecodeMap_.find() to avoid searching twice (TODO: throw exception if not found?)
* Matti suggestion: use SET_PORTABLEHOSTCOLLECTION_READ_RULES; add ClassVersion+checksum
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.
* Matti suggestion: iRecord.getRecord<...>. -> iRecord.; rm redundant includes & couts; pass reference to fill_SoA_column to avoid copy
* Matti suggestions: use SET_PORTABLEHOSTCOLLECTION_READ_RULES for HGCalRecHitHost; Remove deprecated HGCalFlaggedECONDInfo* & comments;
* Matti suggestion: throw exception if type code not found; replace for-loop with std::accumulate
* Matti suggestion: remove TestObjects
* Matti suggestions: remove config as member data and produce unique_ptr instead of shared_ptr; NULL → nullptr; initialize optional override; memcp -> std::copy
* Matti suggestions: use file in path
* removing soafiller and updating buildfile (#139)
* address comments on HGCalRecHitCalibrationAlgorithms.dev.cc
* irot as uint8_t instead of char
* restore DQM without TH2Poly
* apply code formats

---------

Co-authored-by: ywkao <ywkao@hep1.phys.ntu.edu.tw>
Co-authored-by: Pedro <psilva@cern.ch>

restore release base DQM classes (#140)

move to chrono calls instead of struct inside namespace (#141)

address comments (#142)

update include (#143)

remove ECON-D tests as they'll need to be re-written in a different form (#144)

Declare HGCalRecHitCalibrationAlgorithms::calibrate as const method (#145)

* remove ECON-D tests as they'll need to be re-written in a different form

* enable const

patching unpacker for padding of capture blocks within FED

move check of padding to start of capture block

Fix the throw, so the processing won't stop at the first event with error

Update HGCalRecHitCalibrationAlgorithms.dev.cc

CM sum -> 0.5*CM sum

Change the throws to specific returns

Optimize the producer

Debugging the padding of eRx subpackets

roll back (#149)

apply code checks and remove dqm
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants