Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

autopilot stats opens extra yamux stream from server to itself #25032

Open
tgross opened this issue Feb 5, 2025 · 0 comments
Open

autopilot stats opens extra yamux stream from server to itself #25032

tgross opened this issue Feb 5, 2025 · 0 comments

Comments

@tgross
Copy link
Member

tgross commented Feb 5, 2025

While debugging the issue described in hashicorp/yamux#143 I discovered that a server will open yamux connections to itself in order to collect autopilot stats. The stack trace when we check out the stream from the pool is as follows:

github.com/hashicorp/nomad/helper/pool.(*ConnPool).RPC(0xc000e02500, {0x369e892?, 0x0?}, {0x3d354c0?, 0xc0025d3860?}, {0x36d8eb2, 0x10}, {0x3466960, 0xc0022ce8c0}, {0x2e06bc0, ...})
	github.com/hashicorp/nomad/helper/pool/pool.go:502 +0x16f
github.com/hashicorp/nomad/nomad.(*StatsFetcher).fetch(0xc000984d80, 0xc001a6efc0, 0xc000779490)
	github.com/hashicorp/nomad/nomad/stats_fetcher.go:68 +0x26b
created by github.com/hashicorp/nomad/nomad.(*StatsFetcher).Fetch in goroutine 181
	github.com/hashicorp/nomad/nomad/stats_fetcher.go:97 +0x39d

This is fairly minor but it's unnecessary overhead by opening the stream every 2 seconds, potentially using one of the 64 idle streams in the pool that could be used for real work, requiring an extra TCP connection to self, and making noise when debugging yamux issues. Ideally we'd use the in-mem codec when making this call to the agent it's originating from.

@tgross tgross changed the title autopilot stats open extra yamux stream to the server checking the stats autopilot stats opens extra yamux stream from server to itself Feb 5, 2025
@tgross tgross added the hcc/jira label Feb 5, 2025
@jrasell jrasell moved this from Needs Triage to Needs Roadmapping in Nomad - Community Issues Triage Feb 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Needs Roadmapping
Development

No branches or pull requests

1 participant