Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mkFit external integration #36546

Merged
merged 1 commit into from
Feb 21, 2022
Merged

Conversation

osschar
Copy link
Contributor

@osschar osschar commented Dec 17, 2021

PR description

Integrate mkFit into CMSSW from external package.

This PR adds two subpackages to RecoTracker: MkFitCore and MkFitCMS – see below for their overall structure and content. Both of them include a subdirectory “standalone/” where code that is not needed for operation within CMSSW was collected. However, some of this code will need to be reactivated for CMSSW as it produces auto-generated code that describes Phase1 geometry and mkFit track-finding parameters.

RecoTracker
|-- MkFit		CMSSW mkFit producers, ESs; unchanged, just include directives are fixed
|   |-- interface
|   |-- plugins
|   |-- python
|   |-- src
|
|-- MkFitCMS		contains CMS specific steering code, algos & configuration:
|   |-- interface	  - geometry, layers, iterations
|   |-- src		  - scoring, seed cleaning, duplicate removal functions
|   `-- standalone	code to create: 
|       |-- CMS-2017	  . Phase1 geometry (auto-generated)
|       `-- tkNtuple	  . produce standalone mkfit::Event data files from tkNtuple
|
`-- MkFitCore		mkFit core algorithms and data structures
.   |-- interface
.   |-- src
.   |   `-- Matriplex	
.   `-- standalone		contains steering code, internal validation, build, docs
.       |-- Geoms               . trivial test geometry CylindricalCow
.       |-- attic               . older code that would need to be significantly modified
.       |-- code-mod-tools      . scripts for external-to-cmssw migration
.       |-- test		. various tests and demos for mkFit functionality
.       |-- val_scripts		. scripts for standalone validation & benchmarking
.       |-- plotting                 "
.       |-- web			     "
.       `-- xeon_scripts	     "

PR validation

MTV results in TTbar MC with PU (Run3, 2021 realistic conditions), comparing performance before and after mkFit integration in CMSSW:
http://uaf-10.t2.ucsd.edu/~mmasciov/MTV_TTbarPU_cmsswIntegration/

@cmsbuild
Copy link
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36546/27470

ERROR: Build errors found during clang-tidy run.

RecoTracker/MkFitCore/src/MkBuilder.cc:59:8: error: unused variable 'retfitr' [clang-diagnostic-unused-variable]
  auto retfitr = [](MkFitter *mkfttr) { g_exe_ctx.m_fitters.ReturnToPool(mkfttr); };
       ^
RecoTracker/MkFitCore/src/MkBuilder.cc:1748:17: warning: the 'empty' method should be used to check for emptiness instead of 'size' [readability-container-size-empty]
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:129: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

@jpata
Copy link
Contributor

jpata commented Dec 20, 2021

This is a preliminary/draft PR intended as a work-in-progress preview of the upcoming mkFit integration in CMSSW

Can you clarify? Should this be in the draft state (in which case, please change it), or should we consider it "ready for review"?

@slava77
Copy link
Contributor

slava77 commented Dec 20, 2021

This is a preliminary/draft PR intended as a work-in-progress preview of the upcoming mkFit integration in CMSSW

Can you clarify? Should this be in the draft state (in which case, please change it), or should we consider it "ready for review"?

[not with my reco hat]
this PR was meant to be ready for a review for the parts that get built and run in CMSSW.
The code in the /standalone subdirectories and the parts that come with CCCC comments are somewhat a work in the middle of migration from another repository, to be handled in January or so.

There was a push to have this code in a PR as soon as reasonably possible. Perhaps this is a better way to converge anyways.

@cmsbuild
Copy link
Contributor

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36546/27499

  • This PR adds an extra 756KB to repository

  • Found files with invalid states:

    • RecoTracker/MkFitCore/standalone/attic/mkFit.cc:
    • RecoTracker/MkFitCore/src/IceRevisitedRadix.cc:
    • RecoTracker/MkFitCore/standalone/test/CMS-2017.C:
    • RecoTracker/MkFitCore/standalone/attic/buildtestMPlex.h:
    • RecoTracker/MkFitCore/standalone/attic/buildtestMPlex.cc:

Code check has found code style and quality issues which could be resolved by applying following patch(s)

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36546/27530

  • This PR adds an extra 212KB to repository

  • Found files with invalid states:

    • RecoTracker/MkFitCore/standalone/attic/mkFit.cc:
    • RecoTracker/MkFitCore/src/IceRevisitedRadix.cc:
    • RecoTracker/MkFitCore/standalone/test/CMS-2017.C:
    • RecoTracker/MkFitCore/standalone/attic/buildtestMPlex.h:
    • RecoTracker/MkFitCore/standalone/attic/buildtestMPlex.cc:

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @osschar (Matevž Tadel) for master.

It involves the following packages:

  • RecoTracker/MkFit (reconstruction)
  • RecoTracker/MkFitCMS (****)
  • RecoTracker/MkFitCore (****)

The following packages do not have a category, yet:

RecoTracker/MkFitCMS
RecoTracker/MkFitCore
Please create a PR for https://github.com/cms-sw/cms-bot/blob/master/categories_map.py to assign category

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @ebrondol, @gpetruc, @mmusich, @mtosi, @dgulhan this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

@slava77
Copy link
Contributor

slava77 commented Dec 22, 2021

enable profiling

@slava77
Copy link
Contributor

slava77 commented Dec 22, 2021

@cmsbuild please test

@cmsbuild
Copy link
Contributor

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-79c073/21441/summary.html
COMMIT: d86f680
CMSSW: CMSSW_12_3_X_2021-12-21-2300/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36546/21441/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation warning when building: See details on the summary page.

Copy link
Contributor

@perrotta perrotta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a few suggestions for possible cleanup.
Not meant for this PR, but you can perhaps include them in #36966

Comment on lines +92 to +94
int lOffset = 0;
if (lo_ == TkLayout::phase1)
lOffset = 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can move into the scope that starts after L97 (even better: after L100)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I decided to refactor this part of the code in connection to other comments (it will come together with the update for the magic numbers in the layer index literals)

Comment on lines +1 to +2
#ifndef RecoTracker_MkFitCMS_interface_buildtestMPlex_h
#define RecoTracker_MkFitCMS_interface_buildtestMPlex_h
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#ifndef RecoTracker_MkFitCMS_interface_buildtestMPlex_h
#define RecoTracker_MkFitCMS_interface_buildtestMPlex_h
#ifndef RecoTracker_MkFitCMS_standalone_buildtestMPlex_h
#define RecoTracker_MkFitCMS_standalone_buildtestMPlex_h

#include "RecoTracker/MkFitCore/interface/Config.h"
#include "RecoTracker/MkFitCore/interface/Track.h"

#include "nlohmann/json.hpp"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#include "nlohmann/json.hpp"
#include <nlohmann/json.hpp>

@perrotta
Copy link
Contributor

perrotta commented Feb 17, 2022

I am a bit worried by the large number of hardcoded numerical parameters in several files included in these packages, and I wonder how can this code be easily maintainable and scalable...

@slava77
Copy link
Contributor

slava77 commented Feb 18, 2022

I am a bit worried by the large number of hardcoded numerical parameters in several files included in these packages, and I wonder how can this code be easily maintainable and scalable...

It would help to be a bit more specific if the comment is anticipating some action.

I'm not sure I understand the note about scalability. which dimensions are you considering? I suppose it should still be in the context of the CMS design.

Specific to the geometry(geometries), the current implementation is applicable to the phase-1 geometry and would work for 2017 just about as well as for the anticipated 2024 conditions.
The support for phase-2 is coming and will likely start with just one reference geometry.
This is also on the longer-term part of the todo list in #36966.

Please clarify.
Thank you.

@perrotta
Copy link
Contributor

I am a bit worried by the large number of hardcoded numerical parameters in several files included in these packages, and I wonder how can this code be easily maintainable and scalable...

It would help to be a bit more specific if the comment is anticipating some action.

I'm not sure I understand the note about scalability. which dimensions are you considering? I suppose it should still be in the context of the CMS design.

Specific to the geometry(geometries), the current implementation is applicable to the phase-1 geometry and would work for 2017 just about as well as for the anticipated 2024 conditions. The support for phase-2 is coming and will likely start with just one reference geometry. This is also on the longer-term part of the todo list in #36966.

Please clarify. Thank you.

@slava77 my comment did not require any action, sorry for not having specified it clearly.

Simply, when I (finally) went through the whole code I got the impression (apparently shared by other commenters that posted before me) that there are quite a lot of hardcoded parameters here and there in the code, whose meaning can only be clear to the authors of the code themselves or to someone who becomes expert with it. What happens if some special condition happens during the run, can it force us to tune them differently? You are claiming here that "the current implementation is applicable to the phase-1 geometry and would work for 2017 just about as well as for the anticipated 2024 conditions" which is already a good test of robustness, indeed.

On the other hand, it is quite common to the Tracking code to have parts which can only be fully understood by the real experts, and hard to maintain or intervene by other less experienced developers: we had plenty of examples as such also in the past.

If you want, my comment could be rephrased propositively as: when you will review and clean the code after this PR had been merged, please take care of the parts (mostly already commented specifically in this github thread) where there are "obscure" hardcoded parameters: perhaps giving in a few case some meaningful name, or adding some comment line, can help possible future maintenance of the code.

@slava77
Copy link
Contributor

slava77 commented Feb 18, 2022

@slava77 my comment did not require any action, sorry for not having specified it clearly.

Simply, when I (finally) went through the whole code I got the impression (apparently shared by other commenters that posted before me) that there are quite a lot of hardcoded parameters here and there in the code, whose meaning can only be clear to the authors of the code themselves or to someone who becomes expert with it.

@perrotta
Thank you for clarifying on the parameters that the comment is not implying an immediate action required.

I noted your comments in #36546 (review) and would prefer to address them in the follow up PR(s).

Based on your comment I conclude that you went through the code by now.
Please clarify if more comments on the code are expected or if this PR can be merged already.

@qliphy
Copy link
Contributor

qliphy commented Feb 20, 2022

please test
just to test again before final merge

@cmsbuild
Copy link
Contributor

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-79c073/22526/summary.html
COMMIT: a96a2c7
CMSSW: CMSSW_12_3_X_2022-02-19-1100/slc7_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36546/22526/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-79c073/22526/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-79c073/22526/git-merge-result

Comparison Summary

Summary:

  • No significant changes to the logs found
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 6 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3965135
  • DQMHistoTests: Total failures: 13
  • DQMHistoTests: Total nulls: 1
  • DQMHistoTests: Total successes: 3965099
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
  • DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
  • Checked 204 log files, 45 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@qliphy
Copy link
Contributor

qliphy commented Feb 21, 2022

+1
to do list (#36966) to be followed up shortly

@cmsbuild cmsbuild merged commit 40ff2aa into cms-sw:master Feb 21, 2022
@makortel
Copy link
Contributor

This PR broke cs8_aarch64_gcc11 build

>> Compiling  /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/MaterialEffects.cc
/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/gcc/11.2.1-f478fee2760dbd22aaabb4e3a8fe1640/bin/c++ -c -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=110201 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DCMSSW_GIT_HASH='CMSSW_12_3_X_2022-02-21-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_12_3_X_2022-02-21-1100' -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/pcre/8.43-e6b3d6f8e1424033990367f0ae94d8f3/include -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/bz2lib/1.0.6-a8795c5492b25410af2fdbfc34f86eab/include -isystem/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/lcg/root/6.24.07-04cc21bdb96105ce025f7ef7287a739e/include -isystem/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/tbb/v2021.4.0-a1ac0a4b6fb3ff931b4e97030094b2d8/include -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/xz/5.2.5-17d0b07ec73e157930bd27ad2eda52b4/include -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/zlib/1.2.11-31d2ceb65830b90a6cc7a5a90279f5d6/include -I/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/cs8_aarch64_gcc11/external/json/3.10.2-a6d86565b09ec3d0e02bf7b52c31bbfc/include -O2 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++1z -ftree-vectorize -Wstrict-overflow -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -fsigned-char -fsigned-bitfields -march=armv8-a -mno-outline-atomics -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-deprecated-copy -Wno-unused-parameter -Wunused -Wparentheses -Wno-deprecated -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -DBOOST_DISABLE_ASSERTS -fopenmp-simd -Wno-error=strict-aliasing  -fPIC  -MMD -MF tmp/cs8_aarch64_gcc11/src/RecoTracker/MkFitCore/src/RecoTrackerMkFitCore/MaterialEffects.cc.d /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/MaterialEffects.cc -o tmp/cs8_aarch64_gcc11/src/RecoTracker/MkFitCore/src/RecoTrackerMkFitCore/MaterialEffects.cc.o
In file included from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/Matriplex/MatriplexSym.h:4,
                 from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/Matrix.h:27,
                 from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/MkBase.h:4,
                 from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/MkFinder.h:4,
                 from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/CandCloner.h:4,
                 from /data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/CandCloner.cc:1:
/data/cmsbuild/jenkins_a/workspace/build-any-ib/w/tmp/BUILDROOT/6e100e8ffb746f5100679dfb0c62582b/opt/cmssw/cs8_aarch64_gcc11/cms/cmssw/CMSSW_12_3_X_2022-02-21-1100/src/RecoTracker/MkFitCore/src/Matriplex/MatriplexCommon.h:13:10: fatal error: immintrin.h: No such file or directory
   13 | #include "immintrin.h"
      |          ^~~~~~~~~~~~~
compilation terminated.

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/cs8_aarch64_gcc11/CMSSW_12_3_X_2022-02-21-1100/RecoTracker/MkFitCore

I'd guess other non-x86 IBs would fail to build in similar way.

@slava77
Copy link
Contributor

slava77 commented Feb 21, 2022

This PR broke cs8_aarch64_gcc11 build

I guess something like https://github.com/cms-sw/cmsdist/blob/f2fd150c7f36845f4f5a00a60b24dd33d265ec7b/mkfit-2.0.0-non-x86_64-fix.patch
is needed.
What is the right list of ifdef for these builds?

@smuzaffar
Copy link
Contributor

as patch was applied for non x86_64 so how about

#if defined(__x86_64__) // OR if !defined(__aarch64__) && !defined(__powerpc64__)
#include "immintrin.h"
#else
#include <stdlib.h>
#define _mm_malloc(a, b) aligned_alloc(b, a)
#define _mm_free(p) free(p)
#define _mm_prefetch(a,b)  __builtin_prefetch(a)
#endif

@slava77
Copy link
Contributor

slava77 commented Feb 21, 2022

as patch was applied for non x86_64 so how about

I'm curious if there is perhaps a feature-specific macro.

@slava77
Copy link
Contributor

slava77 commented Feb 21, 2022

as patch was applied for non x86_64 so how about

I'm curious if there is perhaps a feature-specific macro.

looking at https://github.com/gcc-mirror/gcc/blob/master/gcc/config/i386/immintrin.h#L24
#define _IMMINTRIN_H_INCLUDED

but this seems ugly, or is it OK?

@slava77
Copy link
Contributor

slava77 commented Feb 21, 2022

as patch was applied for non x86_64 so how about

I'm curious if there is perhaps a feature-specific macro.

looking at https://github.com/gcc-mirror/gcc/blob/master/gcc/config/i386/immintrin.h#L24 #define _IMMINTRIN_H_INCLUDED

but this seems ugly, or is it OK?

nevermind, that's a tautology

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.