Empty Trajectory exception in HLT step in many workflows

[makortel]

(At least) 5 workflows fail in the HLT step with exception like below

----- Begin Fatal Exception 30-Sep-2021 12:39:12 CEST-----------------------
An exception of category 'TrackingTools/PatternTools' occurred while
   [0] Processing  Event run: 1 lumi: 60 event: 2951 stream: 1
   [1] Running path 'HLT_DoubleTrkMu_16_6_NoFiltersNoVtx_v1'
   [2] Calling method for module CkfTrackCandidateMaker/'hltIterL3OITrackCandidatesNoVtx'
Exception Message:
Trajectory::check() - information requested from empty Trajectory 
----- End Fatal Exception -------------------------------------------------

• 11630.0: path HLT_DoubleTrkMu_16_6_NoFiltersNoVtx_v1 module HLT_DoubleTrkMu_16_6_NoFiltersNoVtx_v1 (log)
• 11650.911 path HLT_DoubleTrkMu_16_6_NoFiltersNoVtx_v1 module hltIterL3OITrackCandidatesNoVtx (log)
• 11723.17 path HLT_DoubleTrkMu_16_6_NoFiltersNoVtx_v1 module hltIterL3OITrackCandidatesNoVtx (log)
• 13034.0 path HLT_TrkMu6NoFiltersNoVtx_v1 module hltIterL3OITrackCandidatesNoVtx (log)
• 13034.99 path HLT_OldMu100_v3 module hltL3TrackCandidateFromL2OIState (log)

[makortel]

assign hlt, reconstruction

[cmsbuild]

New categories assigned: hlt,reconstruction

@slava77,@Martin-Grunewald,@jpata,@missirol you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

#35309 looks like a likely cause

[missirol]

FYI @swagata87 (in case she has a guess on what the right fix is)

[swagata87]

I am running runTheMatrix.py -l 11630.0 to see if I can reproduce the issue,

I think I should have put a check like if (!t.empty()) in
TrackingTools/TrajectoryFiltering/interface/MinHitsTrajectoryFilter.h before accessing t.lastMeasurement()

this is what I suspect.. checking..

[swagata87]

I could reproduce the issue in 11630.0,
applied a fix in TrackingTools/TrajectoryFiltering/interface/MinHitsTrajectoryFilter.h,
and reran 11630.0 w/o any problem this time

I'll check with other workflows reported here also (11650.911, 11723.17, 13034.0, 13034.99), and if all of them run fine then I will submit a bugfix PR with the fix.

The WFs are taking a long time to run, but I think I will manage to submit the PR by tonight..

Sorry about the problems caused by my earlier PR

[slava77]

@vmariani @mmusich
this may be of interest to you

IIUC, @swagata87 is available to check the fit. Thank you.

[missirol]

The WFs are taking a long time to run, but I think I will manage to submit the PR by tonight..

@swagata87, in the interest of time, since the fix seems well understood, you could open the PR already, have it reviewed, and have the tests run automatically (incl. the additional wfs), which will be done anyway later.

(But unless any of the experts agrees, please proceed with your current plan.)

[makortel]

(But unless any of the experts agrees, please proceed with your current plan.)

I could agree with that.

[VinInn]

How this could ever be happened?
HLT is essentially broken. This kind of issue should be caught before merging!

[jpata]

Is there a test we can add in the short matrix to catch this early? As I understand, some of the short matrix wfs run HLT, so it's not clear to me why the tests didn't fail.

[jpata]

Noting here that the culprit PR was signed by HLT. I'm happy to hear the suggestions of experts how we can improve test coverage to catch this early.

[Martin-Grunewald]

PR tests did not catch this bug, plain and simple. Some bugs are only uncovered by IB tests as they involve many more workflows/statistics.

[makortel]

The list of failing workflows over IBs also suggests that there is some randomness there, possibly caused by running the DIGI step multithreaded (the DIGI step random number sequence for a given SIM event can change according to the EDM stream the SIM event gets processed by in the DIGI step, and that "stream assignment" can be affected e.g. my machine load).

[makortel]

Anyway, the fix has been merged, so closing the issue.