ecalDevel workflow 28234.61 step3 fails in 12_2_0_pre3

[thomreis]

The ecalDevel matrix WF 28234.61 2026D60_ecalDevel+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal fails at step3 in CMSSW_12_2_0_pre3 with the exception message:

----- Begin Fatal Exception 08-Dec-2021 13:47:58 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module MuonIdProducer/'muons1stStep'
   [8] Prefetching for module HBHEPhase1Reconstructor/'hbhereco@cpu'
   [9] Calling method for module HcalRawToDigi/'hcalDigis'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: FEDRawDataCollection
Looking for module label: rawDataCollector
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Is this a known issue?

[cmsbuild]

A new Issue was created by @thomreis Thomas Reis.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign reconstruction, upgrade

[makortel]

FYI @cms-sw/ecal-dpg-l2 @cms-sw/hcal-dpg-l2

[cmsbuild]

New categories assigned: upgrade,reconstruction

@slava77,@jpata,@AdrianoDee,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

Is any ecalDevel workflow being run in IBs (i.e. part of full runTheMatrix)?

[thomreis]

A comparison with 28234.0 that runs fine shows only the phase2_ecal_devel modifier is given in addition on step3. That alone does not seem to be responsible for the exception. However, step2 of 28234.0 has also the digi2raw_step which might be needed to produce the seemingly missing FEDRawDataCollection.

[thomreis]

Is any ecalDevel workflow being run in IBs (i.e. part of full runTheMatrix)?

I do not know but I do not think so. In fact I wanted to set up a configuration that could become a WF to be run in IBs. 28234.61 seemed a good starting point.

[thomreis]

The WF was introduced by @kpedro88 a bit over a year ago as part of #31726. Has it ever been tested?

[thomreis]

Going through the comments in #31726 it seems that this was expected to fail at the time. #31726 (comment)
PR #32704, coming afterwards, states that the WF runs to completion though.

[kpedro88]

I introduced this workflow to facilitate the ongoing development by @argiro and @dariosol - they should comment further.

[thomreis]

Thanks. I will follow up with them about this and close the issue here.