-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Storing Generator Filter Information in NanoAOD #34169
Conversation
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23376
Code check has found code style and quality issues which could be resolved by applying following patch(s)
|
0966e78
to
1eb6263
Compare
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23377
Code check has found code style and quality issues which could be resolved by applying following patch(s)
|
As discussed in yesterday's X-POG meeting, there might still be some open questions/comments:
I leave the above considerations to the experts. Cheers, |
@@ -271,3 +272,134 @@ class FirstObjectSimpleFlatTableProducer : public SimpleFlatTableProducerBase<T, | |||
return out; | |||
} | |||
}; | |||
|
|||
template <typename T, typename TProd> | |||
class SimpleFlatTableProducerBaseLumi : public edm::one::EDProducer<edm::EndLuminosityBlockProducer> { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why does this module (template) need to be edm::one
? Could it be edm::stream
?
If it needs to stay edm::one
for some reason, please use also edm::LuminosityBlockCache
extension to tell the framework the the module can process events from multiple LuminosityBlocks concurrently.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dear @makortel,
At the time of development I tried to use the edm::stream
(as in SimpleFlatTableProducerBase) guided by the FWMultithreadedFrameworkStreamModuleInterface Twiki however, this resulted in a series of compilation errors that I was not able to resolve. Using edm::one
solved the issues.
If you have any suggestions on how to make edm::stream
work, I would be grateful.
Best,
Mateusz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What kind of compilation errors did you get?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The first one is: /afs/cern.ch/work/m/mzarucki/nanoAOD/CMSSW_12_0_0_pre3/src/PhysicsTools/NanoAOD/interface/SimpleFlatTableProducer.h:315:8: error: 'void SimpleFlatTableProducerBaseLumi<T, TProd>::endLuminosityBlockProduce(edm::LuminosityBlock&, const edm::EventSetup&) [with T = GenFilterInfo; TProd = GenFilterInfo]' marked 'final', but is not virtual
which can be solved by just removing the final
specifier for the endLuminosityBlockProduce
method.
A following compilation error is:
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/interface/stream/callAbilities.h:455:43: error: 'globalEndLuminosityBlockProduce' is not a member of 'LumiSingletonSimpleFlatTableProducer<GenFilterInfo>' 455 | T::globalEndLuminosityBlockProduce(Lumi, iES, iRC);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My bad, after further thought stream module type actually wouldn't help. I was thinking we could use the "streams" to work around the thread-unsafety of StringObjectFunction
and StringCutObjectSelector
, but the LuminosityBlock transitions of stream modules are global (i.e. can occur concurrently).
So the best way to support "processing Events from multiple LuminosityBlocks concurrently" would be to use the edm::LuminosityBlockCache
extension. Since you don't really need the cache, you could declare it as e.g. edm::LuminosityBlockCache<int>
and in globalBeginLuminosityBlock()
return nullptr
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Dear @makortel,
Therefore, is edm:one
fine to use for this module?
As suggested, I have added the edm:LuminosityBlockCache
extension as follows:
class SimpleFlatTableProducerBaseLumi : public edm::one::EDProducer<edm::EndLuminosityBlockProducer, edm::LuminosityBlockCache<int>> {
I have also added the globalBeginLuminosityBlock
method which returns nullptr
.
Unfortunately, I get the following compilation errors:
from /afs/cern.ch/work/m/mzarucki/nanoAOD/CMSSW_12_0_0_pre3/src/PhysicsTools/NanoAOD/plugins/SimpleFlatTableProducerPlugins.cc:1:
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/include/c++/9.3.0/bits/unique_ptr.h: In instantiation of 'typename std::_MakeUniq<_Tp>::__single_object std::make_unique(_Args&& ...) [with _Tp = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>; _Args = {const edm::ParameterSet&}; typename std::_MakeUniq<_Tp>::__single_object = std::unique_ptr<LumiSingletonSimpleFlatTableProducer<GenFilterInfo>, std::default_delete<LumiSingletonSimpleFlatTableProducer<GenFilterInfo> > >]':
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/MakeModuleHelper.h:39:40: required from 'static std::unique_ptr<_Tp> edm::MakeModuleHelper<Base>::makeModule(const edm::ParameterSet&) [with T = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>; Base = edm::one::EDProducerBase]'
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/WorkerMaker.h:83:107: required from 'std::shared_ptr<edm::maker::ModuleHolder> edm::WorkerMaker<T>::makeModule(const edm::ParameterSet&) const [with T = LumiSingletonSimpleFlatTableProducer<GenFilterInfo>]'
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_0_0_pre3/src/FWCore/Framework/src/WorkerMaker.h:77:40: required from here
/cvmfs/cms.cern.ch/slc7_amd64_gcc900/external/gcc/9.3.0/include/c++/9.3.0/bits/unique_ptr.h:857:30: error: invalid new-expression of abstract class type 'LumiSingletonSimpleFlatTableProducer<GenFilterInfo>'
857 | { return unique_ptr<_Tp>(new _Tp(std::forward<_Args>(__args)...)); }
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Is there something obvious that I might be missing here?
Best regards,
Mateusz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The use of LuminosityBlockCache
requires overriding also void SimpleFlatTableProducerBaseLumi::globalEndLuminosityBlock(edm::LuminosityBlock const&, edm::EventSetup const&) override {}
.
(sorry for not mentioning it earlier)
https://twiki.cern.ch/twiki/bin/view/CMSPublic/FWMultithreadedFrameworkOneModuleInterface#edm_LuminosityBlockCacheT
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the clarification!
I have made the discussed updates to support processing events from multiple LuminosityBlocks concurrently in a4a7f44
I would be grateful if you could have a look whether this is what you were expecting.
Cheers,
Mateusz
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would be grateful if you could have a look whether this is what you were expecting.
Yes, it looks good now. Thanks!
in addition to makortel comments please run the following
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/23395
|
A new Pull Request was created by @mzarucki (Mateusz Zarucki) for master. It involves the following packages: PhysicsTools/NanoAOD @cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
please test |
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-84e32b/16108/summary.html Comparison SummarySummary:
|
Dear @kdlong, @mariadalfonso, I have added the I ran the workflows: Best regards, |
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34169/27267
|
Pull request #34169 was updated. @cmsbuild, @mariadalfonso, @gouskos, @fgolf can you please check and sign again. |
please test workflow 546,547,548 |
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-84e32b/21129/summary.html Comparison Summary@slava77 comparisons for the following workflows were not done due to missing matrix map:
Summary:
|
+xpog
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
+1
|
PR description:
SimpleFlatTableProducerBaseLumi
(edm::one::EDProducer
) that stores the generator filter information in a flat tableLuminosityBlocks
tree in miniAODnumEventsTotal
,numEventsPassed
,filterEfficiency
,filterEfficiencyError
genFilterTable
is added to theglobalTablesMC
sequence in globals_cff.pynanoaod::FlatTable
is written into theLuminosityBlocks
tree as new branches via the addedLumiOutputBranches
typePR validation:
The code has been tested in CMSSW_12_0_0_pre1, running a standard nanoAOD workflow on a miniAOD signal MC sample with a generator filter (SMS-T2tt_dM-10to80_genHT-160_genMET-80_mWMin-0p1), over several thousand events (several LS). The output root file correctly has the requested generator filter information stored.
Following rebasing on the latest master branch, the code has passes the nanoAOD unit tests that run over data (
runTheMatrix.py -l 136.8523
) in the latest IB CMSSW_12_0_X_2021-06-17-1100. The MC equivalent (runTheMatrix.py -l 1325.81
) andscram b runtests
fail due to a DAS error not being able to access the file. I believe it would still important to inspect the output of the nanoAOD MC unit tests.