Tier0Handler: add maxTime, improve cert warning

[JanChyczynski]

PR description:

Before this PR Tier0Handler was running a curl command using only the --connect-timeout flag which sets the timeout for connecting. If the connection is established but the server takes forever to respond, the command waits indefinitely. To avoid hung-up processes maxTime argument was added to Tier0Handler setting the --max-time flag. This flag sets the maximum time for the whole curl command.

Tier0Handler is used in the conddb command and the maxTime was set to 60 s, the connection timeout was left at 1 s.

Also, extends the warning information when the certificates are not provided with information about the X509_USER_CERT and X509_USER_KEY env variables.

This is a follow-up to #45779

PR validation:

Tested by running python3 tier0.py which is running the test for it and by running conddb showFSCR (for the latter scram b is needed beforehand).

The proper functionality of --connect-timeout and --max-time in Tier0Handler was tested by changing the T0 API URL using the TIER0_API_URL env variable to:

• http://localhost:8099/ and running nc -kl 8099 to test being able to connect but never receiving a response
• https://example.com:81/ to test the connection timeout

Backport

Backports PRs to 14_1_X and 14_0_X:
#45944 (14_1_X )
#45945 (14_0_X)

FYI @perrotta @francescobrivio @PonIlya

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-45943/41701

• There are other open Pull requests which might conflict with changes you have proposed:
  • File CondCore/Utilities/scripts/conddb modified in PR(s): [PY312] Fix SyntaxWarning in conddb #45760

[cmsbuild]

A new Pull Request was created by @JanChyczynski for master.

It involves the following packages:

• CondCore/Utilities (db)

@cmsbuild, @consuegs, @francescobrivio, @perrotta, @saumyaphor4252 can you please review it and eventually sign? Thanks.
@JanChyczynski, @PonIlya, @mmusich, @rsreds, @yuanchao this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

[perrotta]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Size: This PR adds an extra 48KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c39738/41379/summary.html
COMMIT: 5239086
CMSSW: CMSSW_14_2_X_2024-09-05-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/45943/41379/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @mmusich Move RecoTracker/PixelVertexFinding to RecoVertex/PixelVertexFinding #45842
• @alaperto Improve QTest Pixel DQM #45840
• @smuzaffar [ALCA-DB] Changes suggested by new llvm18 clang-format #45922
• @kyungminparkdrums Add ECAL DCC desync DQM trend plots and update the DQM threshold for Integrity summary #45766
• @bsunanda Phase2-hgx359Z Update the scripts in Validation/HGCalValidation for the changes in Configuration/Geometry #45849
• @smuzaffar [L1-UPGRADE] Changes suggested by new llvm18 clang-format #45907
• @smuzaffar [DAQ-RECONSTRUCTION] Changes suggested by new llvm18 clang-format #45906
• @smuzaffar [RECONSTRUCTION-XPOG] Changes suggested by new llvm18 clang-format #45905
• @smuzaffar [GEOMETRY] Changes suggested by new llvm18 clang-format #45904
• @smuzaffar [DAQ] Changes suggested by new llvm18 clang-format #45903
• @smuzaffar [FASTSIM] Changes suggested by new llvm18 clang-format #45902
• @smuzaffar [ALCA] Changes suggested by new llvm18 clang-format #45901
• @smuzaffar [GEOMETRY-UPGRADE] Changes suggested by new llvm18 clang-format #45900
• @smuzaffar [L1] Changes suggested by new llvm18 clang-format #45908
• @smuzaffar [ANALYSIS] Changes suggested by new llvm18 clang-format #45929
• @smuzaffar [HETERO-RECO-UPGRADE] Changes suggested by new llvm18 clang-format #45928
• @mmusich Add plotting utilities to generate Tracker maps given user input #45795
• @smuzaffar [DB-HLT] Changes suggested by new llvm18 clang-format #45921
• @smuzaffar [DQM-RECONSTRUCTION] Changes suggested by new llvm18 clang-format #45923
• @AdrianoDee Update to relval_gpu.pyfor Alpaka and Pixel Triplets Alpaka Workflows #45694
• @smuzaffar [RECONSTRUCTION] Changes suggested by new llvm18 clang-format #45925
• @smuzaffar [DB] Changes suggested by new llvm18 clang-format #45924
• @smuzaffar [DQM-GEOMETRY] Changes suggested by new llvm18 clang-format #45889
• @sihyunjeon change tune names for auto-bkg production #45755
• @perrotta Update run3_data GT in autoCond from 140X_dataRun3_v4 to 140X_dataRun3_v9 #45879
• @rbhattacharya04 Updates and Bug Fixes for NanoAOD related to muon #45635
• @smuzaffar [CORE] Changes suggested by new llvm18 clang-format #45910
• @smuzaffar [DQM-L1] Changes suggested by new llvm18 clang-format #45913
• @smuzaffar [DQM-HLT] Changes suggested by new llvm18 clang-format #45914
• @smuzaffar [XPOG] Changes suggested by new llvm18 clang-format #45917
• @smuzaffar [DB-RECONSTRUCTION] Changes suggested by new llvm18 clang-format #45918
• @AdrianoDee Adding Golden Workflows for 2022 and 2023 Data #45836
• @smuzaffar [ML] Changes suggested by new llvm18 clang-format #45895
• @smuzaffar [RECONSTRUCTION-UPGRADE] Changes suggested by new llvm18 clang-format #45894
• @smuzaffar [HLT] Changes suggested by new llvm18 clang-format #45897
• @smuzaffar [HETEROGENEOUS] Changes suggested by new llvm18 clang-format #45896
• @makortel Another workaround for -O3 ASAN compilation issue in CTPPSProtonReconstructionPlotter #45786
• @smuzaffar [ALCA-RECONSTRUCTION] Changes suggested by new llvm18 clang-format #45892
• @smuzaffar [VISUALIZATION] Changes suggested by new llvm18 clang-format #45898

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c39738/41379/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-c39738/41379/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test testSiStripPayloadInspector had ERRORS

Comparison Summary

Summary:

• You potentially added 2 lines to the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 6841 differences found in the comparisons
• DQMHistoTests: Total files compared: 44
• DQMHistoTests: Total histograms compared: 3328579
• DQMHistoTests: Total failures: 2536
• DQMHistoTests: Total nulls: 11
• DQMHistoTests: Total successes: 3326012
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 237.2239999999998 KiB( 43 files compared)
• DQMHistoSizes: changed ( 1000.0,... ): 6.777 KiB Ecal/Trends
• DQMHistoSizes: changed ( 140.023 ): -0.008 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 140.043 ): 0.020 KiB JetMET/SUSYDQM
• DQMHistoSizes: changed ( 140.063 ): 0.016 KiB JetMET/SUSYDQM
• Checked 193 log files, 163 edm output root files, 44 DQM output files
• TriggerResults: found differences in 1 / 42 workflows

[mmusich]

-1

the failure is caused by #45795, my bad

[perrotta]

+db

• Error in the unit test is unrelated, see Tier0Handler: add maxTime, improve cert warning #45943 (comment)

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @sextonkennedy, @rappoccio, @antoniovilela (and backports should be raised in the release meeting by the corresponding L2)

[mandrenguyen]

ignore tests-rejected with ib-failure

[mandrenguyen]

+1