Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DeterministicSamplerTest flakey #269

Closed
vreynolds opened this issue Aug 25, 2021 · 1 comment · Fixed by #289
Closed

DeterministicSamplerTest flakey #269

vreynolds opened this issue Aug 25, 2021 · 1 comment · Fixed by #289
Assignees
Labels
type: maintenance The necessary chores to keep the dust off.

Comments

@vreynolds
Copy link
Contributor

I noticed that the DeterministicSamplerTest fails occasionally on dependabot PRs (for example: https://app.circleci.com/pipelines/github/honeycombio/beeline-go/958/workflows/37be948a-6bb5-4606-a629-e5c6f785e07e/jobs/2992)

Re-running passes

@vreynolds vreynolds added the type: maintenance The necessary chores to keep the dust off. label Aug 25, 2021
@vreynolds vreynolds self-assigned this Nov 10, 2021
@vreynolds
Copy link
Contributor Author

Looks like the failure is always just a little off the margin of error (~10 or less off), and always with a sample rate of 100. This makes sense, because with 200 000 test cases and sample rate of 100, the test expects to see results in the 2000 (200 000/100) +/- 100 (2000 * 0.05) range. For comparison, with a sample rate of 50, the test expects to see results in the 4000 (200 000/50) +/- 200 (4000 * 0.05) range. (Smaller sample rate => larger window of error)

I think the options are:

  • Increase the margin of error for the 100 sample rate
  • Increase the total number of test cases (200k currently)
  • Don't test the 100 sample rate (I'm leaning towards this, because I noticed that the Go beeline is an outlier, and other ones don't test sample rates higher than 50)

I see we already ran into this in the Ruby beeline in the past: honeycombio/beeline-ruby#107 and decided to increase the margin of error for a higher sample rate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: maintenance The necessary chores to keep the dust off.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant