Update Triton to v2.11 #7142

kpedro88 · 2021-07-15T21:19:59Z

This PR updates Triton to its latest release. The Triton repository has been refactored into separate components, so the external is renamed to triton-inference-client.

In the process of refactoring, access to some helper functions was lost, so these are installed by hand in the spec file. This should be fixed in an upcoming version of Triton. (Hopefully the various CMake issues currently solved in the spec file will also be addressed.)

The Triton updates require a newer version of CMake, so that is done here as well.

This PR requires an associated CMSSW PR (to be submitted simultaneously) to compile: cms-sw/cmssw#34508

…issues

cmsbuild · 2021-07-15T21:20:19Z

A new Pull Request was created by @kpedro88 (Kevin Pedro) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.
cms-bot commands are listed here

kpedro88 · 2021-07-15T21:31:01Z

please test

smuzaffar · 2021-07-15T21:34:21Z

triton-inference-client.spec

+fi
+
+# extracted from https://github.com/triton-inference-server/server/blob/v2.11.0/src/core/model_config.h
+cat << 'EOF' > ${PROJ_DIR}/library/model_config.h


@kpedro88 , instead of generating these files here, can you please add these generated files as cmsdist/triton-inference-client/model_config.[h,cc].file and then add following sources

Source1: triton-inference-client/model_config.h Source2: triton-inference-client/model_config.cc

and then use

cp %{_sourcedir}/model_config.h ${PROJ_DIR}/library/ cp %{_sourcedir}/model_config.cc ${PROJ_DIR}/library/

cmsbuild · 2021-07-15T21:35:49Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16869/summary.html
COMMIT: 196d001
CMSSW: CMSSW_12_0_X_2021-07-15-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7142/16869/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

kpedro88 · 2021-07-15T21:41:46Z

@smuzaffar what is the right way to deal with the renaming of the tool? The error reported was:

ERROR! File not found: /data/cmsbld/jenkins/workspace/ib-run-pr-tests/cmsdist/triton-inference-server-toolfile.spec

which is intentional...

smuzaffar · 2021-07-15T21:48:40Z

@kpedro88 , you need to cleanup cmssw-tool-conf.spec and replace triton-inference-server-toolfile with triton-inference-client-toolfile

cmsbuild · 2021-07-15T22:01:37Z

Pull request #7142 was updated.

kpedro88 · 2021-07-15T22:02:32Z

please test

cmsbuild · 2021-07-16T04:20:53Z

-1

Failed Tests: Build
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16872/summary.html
COMMIT: 43e05c3
CMSSW: CMSSW_12_0_X_2021-07-15-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/7142/16872/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

TritonData.cc:(.text+0x138): undefined reference to `nvidia::inferenceserver::client::InferInput::Reset()'
/cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/HeterogeneousCore/SonicTriton/src/HeterogeneousCoreSonicTriton/TritonData.cc.o: in function `void TritonData::toServer(std::shared_ptr >, std::allocator > > > >)':
TritonData.cc:(.text._ZN10TritonDataIN6nvidia15inferenceserver6client10InferInputEE8toServerIfEEvSt10shared_ptrISt6vectorIS7_IT_SaIS8_EESaISA_EEE[_ZN10TritonDataIN6nvidia15inferenceserver6client10InferInputEE8toServerIfEEvSt10shared_ptrISt6vectorIS7_IT_SaIS8_EESaISA_EEE]+0x50): undefined reference to `nvidia::inferenceserver::client::InferInput::SetShape(std::vector > const&)'
/cvmfs/cms-ib.cern.ch/nweek-02689/slc7_amd64_gcc900/external/gcc/9.3.0/bin/../lib/gcc/x86_64-unknown-linux-gnu/9.3.0/../../../../x86_64-unknown-linux-gnu/bin/ld: tmp/slc7_amd64_gcc900/src/HeterogeneousCore/SonicTriton/src/HeterogeneousCoreSonicTriton/TritonData.cc.o: in function `void TritonData::toServer(std::shared_ptr >, std::allocator > > > >)':
TritonData.cc:(.text._ZN10TritonDataIN6nvidia15inferenceserver6client10InferInputEE8toServerIlEEvSt10shared_ptrISt6vectorIS7_IT_SaIS8_EESaISA_EEE[_ZN10TritonDataIN6nvidia15inferenceserver6client10InferInputEE8toServerIlEEvSt10shared_ptrISt6vectorIS7_IT_SaIS8_EESaISA_EEE]+0x50): undefined reference to `nvidia::inferenceserver::client::InferInput::SetShape(std::vector > const&)'
collect2: error: ld returned 1 exit status
gmake: *** [tmp/slc7_amd64_gcc900/src/HeterogeneousCore/SonicTriton/src/HeterogeneousCoreSonicTriton/libHeterogeneousCoreSonicTriton.so] Error 1
Leaving library rule at HeterogeneousCore/SonicTriton
------- copying files from src/HeterogeneousCore/SonicTriton/scripts -------
>> copied cmsTriton
Entering library rule at src/HeterogeneousCore/SonicTriton/test

kpedro88 · 2021-07-16T14:49:22Z

@smuzaffar this PR is included in the tests for cms-sw/cmssw#34508, which are running now. Does it need to be tested here also?

smuzaffar · 2021-07-16T14:55:10Z

@kpedro88 , no need to test it here.

kpedro88 · 2021-07-17T16:59:07Z

@smuzaffar cms-sw/cmssw#34508 tests passed

smuzaffar · 2021-07-19T07:29:31Z

test parameters:

pull_request = SonicTriton feature updates, improvements, bug fixes cmssw#34508

smuzaffar · 2021-07-19T07:29:41Z

please test for slc7_amd64_gcc10

smuzaffar · 2021-07-19T07:30:06Z

please test for cc8_amd64_gcc9

smuzaffar · 2021-07-19T07:30:21Z

please test for slc7_aarch64_gcc9

smuzaffar · 2021-07-19T07:30:43Z

please test for CMSSW_12_0_X/slc7_ppc64le_gcc9

smuzaffar · 2021-07-19T15:58:59Z

+externals
new external builds fine for all archs

cmsbuild · 2021-07-19T15:59:20Z

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_0_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2021-07-19T16:25:22Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16972/summary.html
COMMIT: 43e05c3
CMSSW: CMSSW_12_0_X_2021-07-18-2300/slc7_ppc64le_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7142/16972/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [BuildRules] cleanup PYTHON27; use python3 as default #7146

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16972/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16972/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test test_PrepareInputDb had ERRORS
---> test test_MpsWorkFlow had ERRORS
---> test TestDQMServicesDemo had ERRORS
---> test TestHeterogeneousCoreSonicTritonProducerGPU had ERRORS
and more ...

cmsbuild · 2021-07-19T18:45:43Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16976/summary.html
COMMIT: 43e05c3
CMSSW: CMSSW_12_0_X_2021-07-18-2300/slc7_aarch64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7142/16976/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [BuildRules] cleanup PYTHON27; use python3 as default #7146

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16976/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16976/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS
---> test testFWCoreUtilities had ERRORS
---> test TestFWCoreServicesDriver had ERRORS
---> test testUploadConditions had ERRORS
and more ...

cmsbuild · 2021-07-19T21:04:03Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16970/summary.html
COMMIT: 43e05c3
CMSSW: CMSSW_12_0_X_2021-07-18-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7142/16970/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [BuildRules] cleanup PYTHON27; use python3 as default #7146

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16970/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16970/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-c3860b/11634.912_TTbar_14TeV+2021_DD4hepDB+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA

Summary:

No significant changes to the logs found
Reco comparison results: 48194 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2996268
DQMHistoTests: Total failures: 205806
DQMHistoTests: Total nulls: 13
DQMHistoTests: Total successes: 2790427
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.446 KiB( 38 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.915 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 250202.181 ): -0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.352 KiB SiStrip/MechanicalView
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: found differences in 12 / 38 workflows

cmsbuild · 2021-07-19T22:22:51Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16971/summary.html
COMMIT: 43e05c3
CMSSW: CMSSW_12_0_X_2021-07-18-2300/cc8_amd64_gcc9
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7142/16971/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

@cms-sw [BuildRules] cleanup PYTHON27; use python3 as default #7146

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16971/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c3860b/16971/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test TestDQMServicesDemo had ERRORS

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-c3860b/11634.912_TTbar_14TeV+2021_DD4hepDB+TTbar_14TeV_TuneCP5_GenSim+Digi+Reco+HARVEST+ALCA

Summary:

No significant changes to the logs found
Reco comparison results: 53316 differences found in the comparisons
DQMHistoTests: Total files compared: 39
DQMHistoTests: Total histograms compared: 2996268
DQMHistoTests: Total failures: 285032
DQMHistoTests: Total nulls: 253
DQMHistoTests: Total successes: 2710961
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -3.634 KiB( 38 files compared)
DQMHistoSizes: changed ( 10224.0 ): 0.786 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 136.874 ): -0.023 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 250202.181 ): -0.240 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): -0.777 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
DQMHistoSizes: changed ( 4.53 ): 0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 7.3 ): -3.380 KiB SiStrip/MechanicalView
Checked 165 log files, 37 edm output root files, 39 DQM output files
TriggerResults: found differences in 14 / 38 workflows

kpedro88 added 6 commits July 2, 2021 15:52

first attempt to install separate client repo; still ExternalProject …

2f757ee

…issues

bump cmake version

2fb1145

install common repo separately; some issues with TARGET_OBJECTS

a52fab3

modify common repo when importing in client

e0f94d2

fix 'relocation cannot be used' error: compiles successfully

7feb0e0

add missing functions to build

196d001

kpedro88 mentioned this pull request Jul 15, 2021

SonicTriton feature updates, improvements, bug fixes cms-sw/cmssw#34508

Merged

cmsbuild added externals-pending orp-pending pending-signatures tests-pending labels Jul 15, 2021

cmsbuild added tests-started and removed tests-pending labels Jul 15, 2021

smuzaffar reviewed Jul 15, 2021

View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Jul 15, 2021

kpedro88 added 2 commits July 15, 2021 17:00

model_config files as sources

7ac09dc

update tool conf

43e05c3

cmsbuild added tests-pending and removed tests-rejected labels Jul 15, 2021

cmsbuild added tests-started and removed tests-pending tests-started labels Jul 15, 2021

cmsbuild added the tests-rejected label Jul 16, 2021

cmsbuild added the requires-external label Jul 19, 2021

cmsbuild added externals-approved fully-signed and removed externals-pending pending-signatures labels Jul 19, 2021

smuzaffar merged commit 61af3a1 into cms-sw:IB/CMSSW_12_0_X/master Jul 19, 2021

iarspider mentioned this pull request Jul 20, 2021

Use python3 to generate geometry cms-sw/cmssw#34554

Closed

qliphy mentioned this pull request Jul 20, 2021

Introduce conddb script unit tests cms-sw/cmssw#34392

Merged

This was referenced Jul 20, 2021

Update tag for RecoTracker-MkFit to V00-02-00 #7149

Merged

scram-tools repo to provide scram toolfile definitions #7147

Merged

[TF] patch it to use correct pybind11 paths #7151

Merged

kpedro88 mentioned this pull request Jul 27, 2021

remove unneeded triton toolfile #7175

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Triton to v2.11 #7142

Update Triton to v2.11 #7142

kpedro88 commented Jul 15, 2021 •

edited

Loading

cmsbuild commented Jul 15, 2021

kpedro88 commented Jul 15, 2021

smuzaffar Jul 15, 2021

cmsbuild commented Jul 15, 2021

kpedro88 commented Jul 15, 2021

smuzaffar commented Jul 15, 2021 •

edited

Loading

cmsbuild commented Jul 15, 2021

kpedro88 commented Jul 15, 2021

cmsbuild commented Jul 16, 2021

kpedro88 commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

kpedro88 commented Jul 17, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

Update Triton to v2.11 #7142

Update Triton to v2.11 #7142

Conversation

kpedro88 commented Jul 15, 2021 • edited Loading

cmsbuild commented Jul 15, 2021

kpedro88 commented Jul 15, 2021

smuzaffar Jul 15, 2021

Choose a reason for hiding this comment

cmsbuild commented Jul 15, 2021

External Build

kpedro88 commented Jul 15, 2021

smuzaffar commented Jul 15, 2021 • edited Loading

cmsbuild commented Jul 15, 2021

kpedro88 commented Jul 15, 2021

cmsbuild commented Jul 16, 2021

Build

kpedro88 commented Jul 16, 2021

smuzaffar commented Jul 16, 2021

kpedro88 commented Jul 17, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

smuzaffar commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

cmsbuild commented Jul 19, 2021

Unit Tests

cmsbuild commented Jul 19, 2021

Unit Tests

cmsbuild commented Jul 19, 2021

Unit Tests

Comparison Summary

cmsbuild commented Jul 19, 2021

Unit Tests

Comparison Summary

kpedro88 commented Jul 15, 2021 •

edited

Loading

smuzaffar commented Jul 15, 2021 •

edited

Loading