Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAOS-16330 cart: Reduce error logs during timeouts #14905

Merged
merged 9 commits into from
Aug 12, 2024
Merged

DAOS-16330 cart: Reduce error logs during timeouts #14905

merged 9 commits into from
Aug 12, 2024

Conversation

frostedcmos
Copy link
Contributor

  • Reduce burst timeout error prints when multiple rpcs fail at the same time
  • Replace error prints with info during rpc state processing, as error is already logged above
  • Few cleanups, added crt_gdata_dump() at the end of init.

Required-githooks: true

Before requesting gatekeeper:

  • Two review approvals and any prior change requests have been resolved.
  • Testing is complete and all tests passed or there is a reason documented in the PR why it should be force landed and forced-landing tag is set.
  • Features: (or Test-tag*) commit pragma was used or there is a reason documented that there are no appropriate tags for this PR.
  • Commit messages follows the guidelines outlined here.
  • Any tests skipped by the ticket being addressed have been run and passed in the PR.

Gatekeeper:

  • You are the appropriate gatekeeper to be landing the patch.
  • The PR has 2 reviews by people familiar with the code, including appropriate owners.
  • Githooks were used. If not, request that user install them and check copyright dates.
  • Checkpatch issues are resolved. Pay particular attention to ones that will show up on future PRs.
  • All builds have passed. Check non-required builds for any new compiler warnings.
  • Sufficient testing is done. Check feature pragmas and test tags and that tests skipped for the ticket are run and now pass with the changes.
  • If applicable, the PR has addressed any potential version compatibility issues.
  • Check the target branch. If it is master branch, should the PR go to a feature branch? If it is a release branch, does it have merge approval in the JIRA ticket.
  • Extra checks if forced landing is requested
    • Review comments are sufficiently resolved, particularly by prior reviewers that requested changes.
    • No new NLT or valgrind warnings. Check the classic view.
    • Quick-build or Quick-functional is not used.
  • Fix the commit message upon landing. Check the standard here. Edit it to create a single commit. If necessary, ask submitter for a new summary.

@frostedcmos
- Reduce burst timeout error prints when multiple rpcs fail at the sa…
fc7fa9c 
…me time

- Replace error prints with info during state processing, as error is already
logged above

Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Aug 9, 2024

Ticket title is 'cart: reduce flood of error prints when rpcs time out'
Status is 'Open'
https://daosio.atlassian.net/browse/DAOS-16330

@frostedcmos frostedcmos requested a review from jolivier23 August 9, 2024 06:12
jolivier23
jolivier23 previously approved these changes Aug 9, 2024
View reviewed changes
src/cart/crt_context.c
RPC_ERROR(rpc_priv,
"d_binheap_insert failed, rc: %d\n",
rc);
RPC_ERROR(rpc_priv, "d_binheap_insert failed, rc: %d\n", rc);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May I suggest we start using DL_ERROR internally here (or add another version of RPC_ERROR that takes rc as a parameter). It will print string version of error

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dont think we ever hit this particular error; this would be something fatal. I agree though we should use DL_ERROR in places where it makes sense

@frostedcmos
- retrigger
bccb10a 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@frostedcmos
- remove []'s from print to not confuse log parser
6309fd2 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Unit Test on EL 8.8 completed with status FAILURE. https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-14905/2/display/redirect

@frostedcmos
- nuke 'No more free indices left' debug message
d4fd0db 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
soumagne
soumagne reviewed Aug 9, 2024
View reviewed changes
src/cart/crt_internal_types.h Outdated
/** whether metrics are used */
cg_use_sensors : 1,
/** whether we are on a primary provider */
cg_provider_is_primary : 1;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great if you were using unsigned int on each field to fix the format or just start using bool instead :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it would be great if you were using unsigned int on each field to fix the format or just start using bool instead :)

we use bitfields here as we have to be careful about the size of this rpc_priv struct. its annoying for formatting though

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe I was not clear, what I'm saying is that you should not separate each field with , but instead repeat unsigned int for each field so that they remain aligned.

	    /** whether metrics are used */
	    unsigned int cg_use_sensors         : 1;
	    /** whether we are on a primary provider */
	    unsigned int cg_provider_is_primary : 1;

does that make sense ?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also fwiw this is the struct crt_gdata not the rpc_priv struct...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe I was not clear, what I'm saying is that you should not separate each field with , but instead repeat unsigned int for each field so that they remain aligned.

	    /** whether metrics are used */
	    unsigned int cg_use_sensors         : 1;
	    /** whether we are on a primary provider */
	    unsigned int cg_provider_is_primary : 1;

does that make sense ?

Neat, i didnt know it allowed such style

@frostedcmos
Merge branch 'master' into aaoganez/reduce-timeout-errors-and-cleanup
8aa63ac
@frostedcmos
codespell
15378bb 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@frostedcmos
- remove empty line
6c65123 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@frostedcmos
- style
34d483e 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Build on EL 8 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14905/8/execution/node/384/log

@daosbuild1
Copy link
Collaborator

Test stage Build on Leap 15.5 with Intel-C and TARGET_PREFIX completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14905/8/execution/node/388/log

@frostedcmos
- style
6a31898 
Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@daosbuild1
Copy link
Collaborator

Test stage Build RPM on EL 9 completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-14905/8/execution/node/367/log

@frostedcmos frostedcmos marked this pull request as ready for review August 11, 2024 05:41
@frostedcmos frostedcmos requested review from a team as code owners August 11, 2024 05:41
@frostedcmos frostedcmos requested a review from jgmoore-or August 11, 2024 21:08
@frostedcmos frostedcmos merged commit f6afdfe into master Aug 12, 2024
55 checks passed
@frostedcmos frostedcmos deleted the aaoganez/reduce-timeout-errors-and-cleanup branch August 12, 2024 17:52
frostedcmos added a commit that referenced this pull request Aug 12, 2024
@frostedcmos
DAOS-16330 cart: Reduce error logs during timeouts (#14905)
7893c38 
- Reduce burst timeout error prints when multiple rpcs fail at the same time
- Replace error prints with info during state processing, as error is already logged above
- nuke 'No more free indices left' debug message
- cleanup of startup logs

Required-githooks: true

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
mchaarawi pushed a commit that referenced this pull request Aug 15, 2024
@frostedcmos
DAOS-16330 cart: Reduce error logs during timeouts (#14905) (#14915)
ab25284 
- Reduce burst timeout error prints when multiple rpcs fail at the same time
- Replace error prints with info during state processing, as error is already logged above
- nuke 'No more free indices left' debug message
- cleanup of startup logs

Signed-off-by: Alexander A Oganezov <alexander.a.oganezov@intel.com>
@jolivier23 jolivier23 mentioned this pull request Aug 19, 2024
Merged
@mjmac mjmac mentioned this pull request Nov 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgmoore-or jgmoore-or jgmoore-or approved these changes

@jolivier23 jolivier23 jolivier23 approved these changes

@soumagne soumagne Awaiting requested review from soumagne

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@frostedcmos @daosbuild1 @soumagne @jolivier23 @jgmoore-or