cmsTriton bug fixes and improvements

[kpedro88]

PR description:

Rollup of several pending bug fixes and improvements, primarily aimed at the cmsTriton server management script:

1. Some relval jobs have been seeing timeouts when starting the local fallback server if the server image (hosted on /cvms/unpacked.cern.ch) is not already loaded in the worker node's cache. It does not appear to be possible for a non-privileged user to check if something is in the cvmfs cache or not, so the default server start timeout is just extended to minimize these failures.
2. Thread control settings for the local fallback server are only used in the CPU case, not the GPU case.
3. The OSG apptainer version is used by default if available. This avoids the need to propagate updated apptainer versions through a slew of different containers.
4. The SONIC special workflows are extended to 14 TeV processes for Run 3.
5. The TritonService will print the local fallback server log if the process is terminated because of an exception that could be related to the local fallback server. (This is denoted by a third argument in the TRITON_THROW_IF_ERROR macro.) During inference requests, which can be retried, an exception does not necessarily lead to process termination, so the notification to the TritonService can be undone in that case.
6. The unit of the inference request timeout can now be selected. (This was useful for testing the above feature.)
7. More granular verbosity control in the unit test script is now available; clients, the fallback server, and the TritonService itself can all be targeted separately.
8. The unit test script is migrated from VarParsing to argparse, now that the latter works properly with cmsRun.
9. The GPU driver check is skipped if the local fallback server is not using GPU.
10. Fix handling of symbolic links in checksum and versioncheck modes of cmsTritonConfigTool
11. Avoid throwing exceptions in TritonMemResource destructors (addresses TritonService throwing multiple exceptions #38260)

PR validation:

Unit tests pass.

The following command was used to test item 5:

cmsRun test/tritonTest_cfg.py --modules TritonGraphProducer --maxEvents 10 --timeout 1 --timeoutUnit microseconds --verboseService --device CPU --noShm

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not intended to be backported.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43814/38622

• This PR adds an extra 124KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Pixel Alpaka Migration: Configs and Fixes [VII] #43294, Introduce matrix workflows for Alpaka-based Particle Flow #43701, add Era_Run3_2024, Era_Run3_2025, Tier0 reco scenario for 2024, and change default era in PyReleaseValidation #43761, Geometry: add Phase2 scenarios D107 = T32+C17+M11+I17+O9+F8 and D108 = T35+C19+M11+I17+O9+F8 #43805

Code check has found code style and quality issues which could be resolved by applying following patch(s)

• code-format:
  https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43814/38622/code-format.patch
  e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43814/38622/code-format.patch | patch -p1
  You can also run scram build code-format to apply code format directly

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-43814/38624

• This PR adds an extra 120KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add run3 tracking low pu era #33532, Pixel Alpaka Migration: Configs and Fixes [VII] #43294, Introduce matrix workflows for Alpaka-based Particle Flow #43701, add Era_Run3_2024, Era_Run3_2025, Tier0 reco scenario for 2024, and change default era in PyReleaseValidation #43761, Geometry: add Phase2 scenarios D107 = T32+C17+M11+I17+O9+F8 and D108 = T35+C19+M11+I17+O9+F8 #43805

[kpedro88]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4df92d/37258/summary.html
COMMIT: ec8dda1
CMSSW: CMSSW_14_1_X_2024-02-06-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/43814/37258/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially added 6 lines to the logs
• Reco comparison results: 57 differences found in the comparisons
• DQMHistoTests: Total files compared: 48
• DQMHistoTests: Total histograms compared: 3248614
• DQMHistoTests: Total failures: 1189
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3247403
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
• Checked 200 log files, 161 edm output root files, 48 DQM output files
• TriggerResults: no differences found

[makortel]

The usual

• spurious differences in outputs of Run-3 wfs *.7 #39803
• Non reproducibility in wf 136.793 #43293
• Non-reproducibility in TriggerResults in 24834.0, 24834.911, and 25034.999 #43790

[makortel]

+heterogeneous

[srimanob]

+Upgrade

[kpedro88]

@cms-sw/pdmv-l2 please sign (from your side, just expanding a special workflow to another fragment)

[AdrianoDee]

+pdmv

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @antoniovilela, @sextonkennedy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

[antoniovilela]

+1