Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Metrics emission from pd is broken #3780

Closed
conorsch opened this issue Feb 8, 2024 · 1 comment
Closed

Metrics emission from pd is broken #3780

conorsch opened this issue Feb 8, 2024 · 1 comment
Assignees
Labels
A-telemetry Area: Metrics, logging, and other observability-related features _P-low Priority: low

Comments

@conorsch
Copy link
Contributor

conorsch commented Feb 8, 2024

We recently lost pd metrics emission. As of v0.65.0, polling a running pd instance for metrics shows nothing:

❯ curl -s http://localhost:9000/metrics

whereas cometbft still works fine:

❯ curl -s http://localhost:26660/metrics | head -n 3
# HELP cometbft_abci_connection_method_timing_seconds Timing for each ABCI method.
# TYPE cometbft_abci_connection_method_timing_seconds histogram
cometbft_abci_connection_method_timing_seconds_bucket{chain_id="penumbra-testnet-deimos",method="begin_block",type="sync",le="0.0001"} 0

We can see this drop-off on preview, starting ~2d ago:

metrics-loss

which corresponds with the merge of #3745. Notably we also touched metrics in #3747, but those changes were cometbft-specific, and do not appear to have broken emission.

@conorsch conorsch added the A-telemetry Area: Metrics, logging, and other observability-related features label Feb 8, 2024
@conorsch conorsch self-assigned this Feb 8, 2024
@github-project-automation github-project-automation bot moved this to Future in Testnets Feb 8, 2024
@github-project-automation github-project-automation bot moved this to 🗄️ Backlog in Penumbra Feb 8, 2024
@hdevalence hdevalence added the _P-low Priority: low label Feb 9, 2024
@conorsch
Copy link
Contributor Author

Fixed via #3801, confirmed by checking preview metrics post-deploy.

@github-project-automation github-project-automation bot moved this from 🗄️ Backlog to Done in Penumbra Feb 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-telemetry Area: Metrics, logging, and other observability-related features _P-low Priority: low
Projects
Archived in project
Status: Future
Development

No branches or pull requests

2 participants