-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comparison failed for tests of wf 39434.911
#37315
Comments
A new Issue was created by @francescobrivio . @Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks. cms-bot commands are listed here |
assign reconstruction |
New categories assigned: reconstruction @jpata,@slava77,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks |
.911 is dd4hep, we often had differences in it not related to reco. did something change, or is it still a geometry-related issue? |
I believe the message
refers only to the "validateJR" / "reco comparisons", and not to RelMon-based comparisons or DQM bin by bin. |
assign geometry |
New categories assigned: geometry @cvuosalo,@mdhildreth,@ianna,@Dr15Jones,@makortel,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks |
@jpata from the |
There are however also quite a lot of differences in the bin by bin DQM comparisons for that workflow in the test outputs, as you can verify by opening any of the links to the PR tests listed in the issue description above. |
Right, but the existence of the differences (whose cause should be identified) is a separate issue from one piece of comparisons infrastructure not recognizing this workflow. |
39434.911 is Phase 2 D88 DD4hep. It is quite new and may be still under development. This workflow runs DD4hep from XML files. That makes it very sensitive to any perturbations in the source files or test process. |
assign upgrade |
How can we test the stability of the workflow offline? I mean how do we know if some unexpected random behavior will happen somewhere without this kind of PR test. The test was done with ttbar 9k events, but that is to compare between DDD and DD4hep. I never try to compare DD4hep with DD4hep from 2 runs to see its stability. |
New categories assigned: upgrade @AdrianoDee,@srimanob you have been requested to review this Pull request/Issue and eventually sign? Thanks |
@francescobrivio |
@srimanob Could you elaborate what you mean with "test it offline"? Is it e.g. about comparing the DQM root files of different invocations of the workflow? |
Hi @makortel By the way, last fresh test of the following PR does not show perculiar on the workflow comparison, |
Right, run twice or many times. The comparison failure appears to be random, so it is hard to say beforehand how many times to run. In the past we've seen occurrence rates between O(1 %) and O(100 %) or so with the (Run 3) DD4Hep workflow. |
I think this problem should be addressed by this cms-sw/cms-bot@31e5f20 |
I've prepared the pull request to disable it in case we still face random issue of failure comparison. |
+reconstruction
|
Wf disabled from short matrix in #37337 |
In recent PRs test results there is a message saying:
But nonetheless the default comparison reports more than 50k differences for this wf.
Some examples:
The text was updated successfully, but these errors were encountered: