Unit test failures in DQMServices/Demo and RecoTauTag/HLTProducers in the 12_1_X IBs

[perrotta]

We are going to uild CMSSW_12_1_0_pre5. After the last nightly IBs we are noticing two new unit test failures reported for the two systems in the subject.

The message generated by TestDQMServicesDemo is rather cryptic:

dropped waiting message count 0
++ dqmiolistmes.py alltypes.root -r 1
++ wc -l
+ '[' 66 = 1 ']'
status = 256

---> test TestDQMServicesDemo had ERRORS

Moreover, we tried to reproduce locally

cd DQMServices/Demo/test/
bash runtests.sh

and the tests ended without error: it could be a glitch then. We will wait for the next IB and see if it reproduces. In any case, if @cms-sw/dqm-l2 can have a look in the meanwhile it can help to speed up.

For the failure in TestRecoTauTagHLTProducers, this could have originated from the updates to the HLTScoutingPFProducer integrated with #35863. However (see the ongoing discussion in that PR thread) it looks like that the real culprit could be the way in which that unit test was originally designed in #35640.
Also for this we would like to reach quickly to a solution: @cms-sw/hlt-l2 and @azotz are informed and are currently looking at it. If no quick fix is found, we should decide to either remove the unit test, or revert #35863, the solution proposed by HLT being to remove the unit test.

[cmsbuild]

A new Issue was created by @perrotta Andrea Perrotta.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[perrotta]

assign dqm, hlt

[cmsbuild]

New categories assigned: dqm,hlt

@jfernan2,@ahmad3213,@rvenditti,@missirol,@emanueleusai,@pbo0,@Martin-Grunewald,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[perrotta]

For the HLT unit test failure see #35867 (thank you @missirol )

[azotz]

We were afraid this might happen in the future. A removal of the unit test in question seems appropriate. It requires a redesign to be more robust.

[perrotta]

We were afraid this might happen in the future. A removal of the unit test in question seems appropriate. It requires a redesign to be more robust.

Thank you @azotz
Please provide such more robust redesign of the unit test, then, which is definitely a better solution than letting the module untested in the release. For the meanwhile we'll stick on the solution proposed and remove the unit test from pre5.

[mmusich]

the TestDQMServicesDemo issue is seen also in #35869

[mmusich]

Is there any progress in understanding the TestDQMServicesDemo failure? It is showing up in all the "urgent" PRs targeting fixes for the online DQM during the beam test.
Thanks

[qliphy]

Looking into a previous IB log without this issue:
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_1_X_2021-10-27-1100/unitTestLogs/DQMServices/Demo#/

dropped waiting message count 0
++ dqmiolistmes.py alltypes.root -r 1
++ wc -l

• '[' 66 = 66 ']'
  ++ dqmiolistmes.py alltypes.root -r 1 -l 1
  ++ wc -l
• '[' 66 = 66 ']'
  ++ /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_12_1_X_2021-10-27-1100/src/DQMServices/Demo/test/dqmiodumpentries.py alltypes.root -r 1 --summary
• '[' '0: 1, 0.0: 1, 1: 9, 100: 27, 200: 9, 5: 14, 5.0: 5' = '0: 1, 0.0: 1, 1: 9, 100: 27, 200: 9, 5: 14, 5.0: 5' ']'
  ++ /data/cmsbld/jenkins/workspace/ib-run-qa/CMSSW_12_1_X_2021-10-27-1100/src/DQMServices/Demo/test/dqmiodumpentries.py alltypes.root -r 1 -l 1 --summary
• '[' '1: 24, 1.0: 6, 20: 36' = '1: 24, 1.0: 6, 20: 36' ']'

However, in recent IB test, it is stuck as below. Local test can reproduce similar behavior. (Although local test just exit and don't report error message)

dropped waiting message count 0
++ dqmiolistmes.py alltypes.root -r 1
++ wc -l

• '[' 66 = 1 ']'
  status = 256

@cms-sw/dqm-l2 Would you please have a check?

[qliphy]

The script exits at
https://github.com/cms-sw/cmssw/blob/master/DQMServices/Demo/test/runtests.sh#L14
and doesn't execute next lines

[missirol]

Looks like CMSSW_12_1_X_2021-10-27-2300 picked up version 4.1.5 of uproot (the previous IB had 3.13.0), and there were some API changes that broke the DQM unit test. A possible fix is in #35921.

What's unclear to me is why for some PR tests with recent IBs this unit-test error does not show up (see for example PR 35911).

[perrotta]

What's unclear to me is why for some PR tests with recent IBs this unit-test error does not show up (see for example PR 35911).

In the PR tests, unit tests are only run for the packages affected by the PR itself

[missirol]

Okay, that explains it. Thanks!

[missirol]

+hlt

the HLT side of this issue was addressed in #35867

[jfernan2]

+1
Fixed in #35921and #35390 for the DQM side

[cmsbuild]

This issue is fully signed and ready to be closed.

[perrotta]

Thank you all!
Closing this issue