You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Looks like the failure is always just a little off the margin of error (~10 or less off), and always with a sample rate of 100. This makes sense, because with 200 000 test cases and sample rate of 100, the test expects to see results in the 2000 (200 000/100) +/- 100 (2000 * 0.05) range. For comparison, with a sample rate of 50, the test expects to see results in the 4000 (200 000/50) +/- 200 (4000 * 0.05) range. (Smaller sample rate => larger window of error)
I think the options are:
Increase the margin of error for the 100 sample rate
Increase the total number of test cases (200k currently)
Don't test the 100 sample rate (I'm leaning towards this, because I noticed that the Go beeline is an outlier, and other ones don't test sample rates higher than 50)
I see we already ran into this in the Ruby beeline in the past: honeycombio/beeline-ruby#107 and decided to increase the margin of error for a higher sample rate.
I noticed that the DeterministicSamplerTest fails occasionally on dependabot PRs (for example: https://app.circleci.com/pipelines/github/honeycombio/beeline-go/958/workflows/37be948a-6bb5-4606-a629-e5c6f785e07e/jobs/2992)
Re-running passes
The text was updated successfully, but these errors were encountered: