Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[crashtracker] Implement telemetry for RFC5 #741

Merged
merged 5 commits into from
Nov 22, 2024

Conversation

danielsn
Copy link
Contributor

@danielsn danielsn commented Nov 14, 2024

What does this PR do?

What it says on the tin

Motivation

A format we can't actually upload isn't that useful.

Additional Notes

Some fields are duplicated from the CrashInfo into telemetry headers, but I think there is value in posting precisely the RFC5 struct, and in having searchable telemetry headers.

How to test the change?

Added a version of the additional telemetry test, with additional checks.

@danielsn danielsn requested a review from a team as a code owner November 14, 2024 21:53
@pr-commenter
Copy link

pr-commenter bot commented Nov 14, 2024

Benchmarks

Comparison

Benchmark execution time: 2024-11-22 15:47:20

Comparing candidate commit f4b295e in PR branch dsn/crashtracker-rfc5-telemetry with baseline commit 580c2f4 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 51 metrics, 2 unstable metrics.

Candidate

Candidate benchmark details

Group 1

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
tags/replace_trace_tags execution_time 2.652µs 2.700µs ± 0.018µs 2.700µs ± 0.007µs 2.706µs 2.740µs 2.749µs 2.752µs 1.90% 0.196 2.360 0.65% 0.001µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
tags/replace_trace_tags execution_time [2.698µs; 2.702µs] or [-0.090%; +0.090%] None None None

Group 2

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
concentrator/add_spans_to_concentrator execution_time 8.996ms 9.041ms ± 0.017ms 9.040ms ± 0.010ms 9.050ms 9.065ms 9.073ms 9.143ms 1.14% 1.148 6.328 0.18% 0.001ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
concentrator/add_spans_to_concentrator execution_time [9.038ms; 9.043ms] or [-0.025%; +0.025%] None None None

Group 3

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_trace/test_trace execution_time 298.393ns 309.821ns ± 13.343ns 303.341ns ± 3.095ns 315.466ns 341.555ns 345.185ns 347.364ns 14.51% 1.354 0.525 4.30% 0.943ns 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_trace/test_trace execution_time [307.971ns; 311.670ns] or [-0.597%; +0.597%] None None None

Group 4

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time 301.724µs 307.588µs ± 3.126µs 308.096µs ± 3.054µs 310.252µs 311.984µs 312.508µs 313.019µs 1.60% -0.109 -1.389 1.01% 0.221µs 1 200
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput 3194695.037op/s 3251441.543op/s ± 33081.615op/s 3245738.960op/s ± 32260.534op/s 3283876.886op/s 3299816.297op/s 3309650.098op/s 3314283.427op/s 2.11% 0.127 -1.386 1.01% 2339.223op/s 1 200
normalization/normalize_name/normalize_name/bad-name execution_time 27.859µs 28.096µs ± 0.108µs 28.079µs ± 0.065µs 28.163µs 28.284µs 28.404µs 28.465µs 1.38% 0.816 0.747 0.38% 0.008µs 1 200
normalization/normalize_name/normalize_name/bad-name throughput 35130435.253op/s 35592589.292op/s ± 135840.705op/s 35613980.782op/s ± 82481.218op/s 35692305.897op/s 35773712.214op/s 35837690.323op/s 35895625.186op/s 0.79% -0.793 0.693 0.38% 9605.388op/s 1 200
normalization/normalize_name/normalize_name/good execution_time 16.515µs 16.631µs ± 0.051µs 16.636µs ± 0.037µs 16.662µs 16.706µs 16.734µs 16.827µs 1.15% 0.170 0.326 0.30% 0.004µs 1 200
normalization/normalize_name/normalize_name/good throughput 59426820.696op/s 60130156.062op/s ± 183075.643op/s 60109862.333op/s ± 134012.130op/s 60266579.176op/s 60416988.762op/s 60516950.072op/s 60551108.241op/s 0.73% -0.149 0.280 0.30% 12945.403op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... execution_time [307.154µs; 308.021µs] or [-0.141%; +0.141%] None None None
normalization/normalize_name/normalize_name/Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Long-.Too-Lo... throughput [3246856.750op/s; 3256026.337op/s] or [-0.141%; +0.141%] None None None
normalization/normalize_name/normalize_name/bad-name execution_time [28.081µs; 28.111µs] or [-0.053%; +0.053%] None None None
normalization/normalize_name/normalize_name/bad-name throughput [35573763.077op/s; 35611415.507op/s] or [-0.053%; +0.053%] None None None
normalization/normalize_name/normalize_name/good execution_time [16.624µs; 16.638µs] or [-0.042%; +0.042%] None None None
normalization/normalize_name/normalize_name/good throughput [60104783.538op/s; 60155528.585op/s] or [-0.042%; +0.042%] None None None

Group 5

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
credit_card/is_card_number/ execution_time 4.615µs 4.632µs ± 0.006µs 4.632µs ± 0.002µs 4.634µs 4.639µs 4.642µs 4.700µs 1.47% 5.761 61.351 0.14% 0.000µs 1 200
credit_card/is_card_number/ throughput 212756732.041op/s 215888500.390op/s ± 297666.155op/s 215877591.906op/s ± 109839.818op/s 215997560.769op/s 216313054.104op/s 216370112.240op/s 216683875.588op/s 0.37% -5.637 59.723 0.14% 21048.176op/s 1 200
credit_card/is_card_number/ 3782-8224-6310-005 execution_time 90.842µs 91.732µs ± 0.536µs 91.686µs ± 0.286µs 91.974µs 92.527µs 92.812µs 96.482µs 5.23% 3.589 29.067 0.58% 0.038µs 1 200
credit_card/is_card_number/ 3782-8224-6310-005 throughput 10364580.723op/s 10901678.155op/s ± 62481.773op/s 10906769.497op/s ± 33964.123op/s 10940166.056op/s 10977502.928op/s 10999377.036op/s 11008162.553op/s 0.93% -3.286 25.619 0.57% 4418.129op/s 1 200
credit_card/is_card_number/ 378282246310005 execution_time 83.809µs 84.055µs ± 0.367µs 83.987µs ± 0.046µs 84.075µs 84.295µs 84.552µs 88.868µs 5.81% 11.439 146.693 0.44% 0.026µs 1 200
credit_card/is_card_number/ 378282246310005 throughput 11252705.023op/s 11897129.984op/s ± 49480.501op/s 11906601.615op/s ± 6573.762op/s 11910983.714op/s 11918627.981op/s 11925032.873op/s 11931855.668op/s 0.21% -11.205 142.443 0.41% 3498.800op/s 1 200
credit_card/is_card_number/37828224631 execution_time 4.613µs 4.631µs ± 0.007µs 4.631µs ± 0.003µs 4.634µs 4.638µs 4.643µs 4.695µs 1.37% 4.308 41.537 0.14% 0.000µs 1 200
credit_card/is_card_number/37828224631 throughput 212994520.447op/s 215913834.041op/s ± 305211.543op/s 215912965.575op/s ± 145146.450op/s 216070020.196op/s 216290880.900op/s 216418434.212op/s 216772074.054op/s 0.40% -4.203 40.288 0.14% 21581.715op/s 1 200
credit_card/is_card_number/378282246310005 execution_time 81.049µs 81.196µs ± 0.118µs 81.161µs ± 0.029µs 81.196µs 81.424µs 81.643µs 81.837µs 0.83% 2.570 7.572 0.15% 0.008µs 1 200
credit_card/is_card_number/378282246310005 throughput 12219461.128op/s 12315954.250op/s ± 17873.336op/s 12321184.464op/s ± 4349.581op/s 12324627.202op/s 12331433.654op/s 12335309.605op/s 12338221.292op/s 0.14% -2.557 7.488 0.14% 1263.836op/s 1 200
credit_card/is_card_number/37828224631000521389798 execution_time 58.996µs 59.238µs ± 0.149µs 59.241µs ± 0.123µs 59.321µs 59.518µs 59.664µs 59.735µs 0.83% 0.835 0.374 0.25% 0.011µs 1 200
credit_card/is_card_number/37828224631000521389798 throughput 16740571.170op/s 16881216.465op/s ± 42485.865op/s 16880190.253op/s ± 34996.204op/s 16918461.849op/s 16932461.730op/s 16938487.996op/s 16950376.519op/s 0.42% -0.822 0.341 0.25% 3004.204op/s 1 200
credit_card/is_card_number/x371413321323331 execution_time 6.830µs 6.842µs ± 0.004µs 6.841µs ± 0.002µs 6.843µs 6.849µs 6.853µs 6.855µs 0.20% 0.597 0.794 0.06% 0.000µs 1 200
credit_card/is_card_number/x371413321323331 throughput 145887334.265op/s 146165577.734op/s ± 87971.019op/s 146175150.225op/s ± 50363.371op/s 146224664.445op/s 146284427.328op/s 146361837.422op/s 146405465.430op/s 0.16% -0.593 0.790 0.06% 6220.490op/s 1 200
credit_card/is_card_number_no_luhn/ execution_time 4.617µs 4.633µs ± 0.004µs 4.633µs ± 0.002µs 4.635µs 4.639µs 4.641µs 4.644µs 0.24% -0.607 1.632 0.08% 0.000µs 1 200
credit_card/is_card_number_no_luhn/ throughput 215347046.221op/s 215852520.871op/s ± 183825.400op/s 215860985.525op/s ± 113512.186op/s 215957612.920op/s 216143888.252op/s 216315590.843op/s 216589491.619op/s 0.34% 0.615 1.654 0.08% 12998.419op/s 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time 72.888µs 73.703µs ± 0.151µs 73.703µs ± 0.058µs 73.761µs 73.840µs 74.309µs 74.433µs 0.99% 0.759 10.095 0.20% 0.011µs 1 200
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput 13434913.131op/s 13567953.519op/s ± 27680.086op/s 13568034.389op/s ± 10667.563op/s 13578504.891op/s 13604681.789op/s 13621470.965op/s 13719636.603op/s 1.12% -0.688 10.111 0.20% 1957.278op/s 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time 64.710µs 65.073µs ± 0.194µs 65.035µs ± 0.104µs 65.181µs 65.435µs 65.620µs 65.724µs 1.06% 0.685 0.308 0.30% 0.014µs 1 200
credit_card/is_card_number_no_luhn/ 378282246310005 throughput 15215209.756op/s 15367439.631op/s ± 45781.972op/s 15376255.219op/s ± 24548.709op/s 15396668.546op/s 15434976.664op/s 15443574.812op/s 15453555.843op/s 0.50% -0.669 0.275 0.30% 3237.274op/s 1 200
credit_card/is_card_number_no_luhn/37828224631 execution_time 4.614µs 4.632µs ± 0.005µs 4.633µs ± 0.003µs 4.635µs 4.639µs 4.641µs 4.646µs 0.28% -0.447 0.713 0.10% 0.000µs 1 200
credit_card/is_card_number_no_luhn/37828224631 throughput 215256546.938op/s 215877426.506op/s ± 221161.187op/s 215864307.701op/s ± 129006.118op/s 215993018.026op/s 216269072.248op/s 216456191.613op/s 216721813.673op/s 0.40% 0.455 0.726 0.10% 15638.458op/s 1 200
credit_card/is_card_number_no_luhn/378282246310005 execution_time 63.131µs 63.532µs ± 0.134µs 63.563µs ± 0.071µs 63.619µs 63.685µs 63.758µs 64.162µs 0.94% -0.270 2.239 0.21% 0.009µs 1 200
credit_card/is_card_number_no_luhn/378282246310005 throughput 15585564.453op/s 15740207.672op/s ± 33181.643op/s 15732460.548op/s ± 17466.157op/s 15759074.089op/s 15808373.221op/s 15827983.260op/s 15840042.042op/s 0.68% 0.296 2.172 0.21% 2346.296op/s 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time 58.991µs 59.299µs ± 0.231µs 59.244µs ± 0.158µs 59.456µs 59.752µs 59.875µs 59.972µs 1.23% 0.790 -0.272 0.39% 0.016µs 1 200
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput 16674576.402op/s 16864007.287op/s ± 65443.815op/s 16879251.185op/s ± 45160.586op/s 16921389.906op/s 16938944.412op/s 16949642.962op/s 16951641.481op/s 0.43% -0.777 -0.300 0.39% 4627.577op/s 1 200
credit_card/is_card_number_no_luhn/x371413321323331 execution_time 6.829µs 6.840µs ± 0.004µs 6.841µs ± 0.002µs 6.842µs 6.845µs 6.849µs 6.851µs 0.15% -0.438 1.024 0.05% 0.000µs 1 200
credit_card/is_card_number_no_luhn/x371413321323331 throughput 145967867.301op/s 146194367.655op/s ± 77502.357op/s 146186031.927op/s ± 41381.145op/s 146227794.257op/s 146335691.970op/s 146403407.779op/s 146433062.863op/s 0.17% 0.442 1.026 0.05% 5480.244op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
credit_card/is_card_number/ execution_time [4.631µs; 4.633µs] or [-0.019%; +0.019%] None None None
credit_card/is_card_number/ throughput [215847246.723op/s; 215929754.056op/s] or [-0.019%; +0.019%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 execution_time [91.658µs; 91.806µs] or [-0.081%; +0.081%] None None None
credit_card/is_card_number/ 3782-8224-6310-005 throughput [10893018.782op/s; 10910337.528op/s] or [-0.079%; +0.079%] None None None
credit_card/is_card_number/ 378282246310005 execution_time [84.005µs; 84.106µs] or [-0.060%; +0.060%] None None None
credit_card/is_card_number/ 378282246310005 throughput [11890272.462op/s; 11903987.505op/s] or [-0.058%; +0.058%] None None None
credit_card/is_card_number/37828224631 execution_time [4.631µs; 4.632µs] or [-0.020%; +0.020%] None None None
credit_card/is_card_number/37828224631 throughput [215871534.656op/s; 215956133.425op/s] or [-0.020%; +0.020%] None None None
credit_card/is_card_number/378282246310005 execution_time [81.179µs; 81.212µs] or [-0.020%; +0.020%] None None None
credit_card/is_card_number/378282246310005 throughput [12313477.177op/s; 12318431.322op/s] or [-0.020%; +0.020%] None None None
credit_card/is_card_number/37828224631000521389798 execution_time [59.217µs; 59.259µs] or [-0.035%; +0.035%] None None None
credit_card/is_card_number/37828224631000521389798 throughput [16875328.333op/s; 16887104.598op/s] or [-0.035%; +0.035%] None None None
credit_card/is_card_number/x371413321323331 execution_time [6.841µs; 6.842µs] or [-0.008%; +0.008%] None None None
credit_card/is_card_number/x371413321323331 throughput [146153385.797op/s; 146177769.671op/s] or [-0.008%; +0.008%] None None None
credit_card/is_card_number_no_luhn/ execution_time [4.632µs; 4.633µs] or [-0.012%; +0.012%] None None None
credit_card/is_card_number_no_luhn/ throughput [215827044.439op/s; 215877997.304op/s] or [-0.012%; +0.012%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 execution_time [73.683µs; 73.724µs] or [-0.028%; +0.028%] None None None
credit_card/is_card_number_no_luhn/ 3782-8224-6310-005 throughput [13564117.325op/s; 13571789.713op/s] or [-0.028%; +0.028%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 execution_time [65.046µs; 65.100µs] or [-0.041%; +0.041%] None None None
credit_card/is_card_number_no_luhn/ 378282246310005 throughput [15361094.690op/s; 15373784.572op/s] or [-0.041%; +0.041%] None None None
credit_card/is_card_number_no_luhn/37828224631 execution_time [4.632µs; 4.633µs] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/37828224631 throughput [215846775.692op/s; 215908077.319op/s] or [-0.014%; +0.014%] None None None
credit_card/is_card_number_no_luhn/378282246310005 execution_time [63.513µs; 63.550µs] or [-0.029%; +0.029%] None None None
credit_card/is_card_number_no_luhn/378282246310005 throughput [15735609.015op/s; 15744806.328op/s] or [-0.029%; +0.029%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 execution_time [59.267µs; 59.331µs] or [-0.054%; +0.054%] None None None
credit_card/is_card_number_no_luhn/37828224631000521389798 throughput [16854937.404op/s; 16873077.170op/s] or [-0.054%; +0.054%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 execution_time [6.840µs; 6.841µs] or [-0.007%; +0.007%] None None None
credit_card/is_card_number_no_luhn/x371413321323331 throughput [146183626.573op/s; 146205108.736op/s] or [-0.007%; +0.007%] None None None

Group 6

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching deserializing traces from msgpack to their internal representation execution_time 60.322ms 60.505ms ± 0.073ms 60.498ms ± 0.034ms 60.537ms 60.611ms 60.848ms 60.903ms 0.67% 1.840 8.502 0.12% 0.005ms 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching deserializing traces from msgpack to their internal representation execution_time [60.495ms; 60.515ms] or [-0.017%; +0.017%] None None None

Group 7

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
redis/obfuscate_redis_string execution_time 37.558µs 38.228µs ± 1.090µs 37.746µs ± 0.074µs 37.830µs 40.596µs 40.650µs 40.718µs 7.87% 1.696 0.905 2.85% 0.077µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
redis/obfuscate_redis_string execution_time [38.077µs; 38.380µs] or [-0.395%; +0.395%] None None None

Group 8

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
write only interface execution_time 1.415µs 3.225µs ± 1.423µs 3.074µs ± 0.021µs 3.090µs 3.190µs 14.096µs 15.150µs 392.88% 7.631 58.203 43.99% 0.101µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
write only interface execution_time [3.028µs; 3.423µs] or [-6.112%; +6.112%] None None None

Group 9

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time 619.765µs 621.231µs ± 0.557µs 621.282µs ± 0.251µs 621.506µs 621.794µs 622.068µs 626.002µs 0.76% 2.676 25.805 0.09% 0.039µs 1 200
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput 1597438.305op/s 1609709.509op/s ± 1439.342op/s 1609574.349op/s ± 649.499op/s 1610336.989op/s 1612101.154op/s 1612946.248op/s 1613514.974op/s 0.24% -2.621 25.268 0.09% 101.777op/s 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time 466.104µs 467.096µs ± 0.539µs 467.032µs ± 0.221µs 467.281µs 467.698µs 467.981µs 472.988µs 1.28% 6.601 69.562 0.12% 0.038µs 1 200
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput 2114218.307op/s 2140889.018op/s ± 2452.242op/s 2141181.266op/s ± 1012.851op/s 2142102.273op/s 2143298.780op/s 2144433.700op/s 2145442.724op/s 0.20% -6.505 68.173 0.11% 173.400op/s 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time 190.897µs 191.403µs ± 0.218µs 191.369µs ± 0.136µs 191.528µs 191.814µs 191.988µs 192.264µs 0.47% 0.718 1.017 0.11% 0.015µs 1 200
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput 5201183.789op/s 5224585.794op/s ± 5942.254op/s 5225493.441op/s ± 3712.348op/s 5228530.523op/s 5232481.677op/s 5236981.421op/s 5238418.562op/s 0.25% -0.709 0.998 0.11% 420.181op/s 1 200
normalization/normalize_service/normalize_service/[empty string] execution_time 47.085µs 47.418µs ± 0.123µs 47.415µs ± 0.086µs 47.502µs 47.604µs 47.666µs 47.809µs 0.83% -0.003 0.102 0.26% 0.009µs 1 200
normalization/normalize_service/normalize_service/[empty string] throughput 20916529.936op/s 21088994.987op/s ± 54758.290op/s 21090538.265op/s ± 38330.914op/s 21127531.740op/s 21175320.507op/s 21231921.705op/s 21238170.376op/s 0.70% 0.020 0.099 0.26% 3871.996op/s 1 200
normalization/normalize_service/normalize_service/test_ASCII execution_time 51.457µs 51.632µs ± 0.128µs 51.626µs ± 0.066µs 51.689µs 51.782µs 51.850µs 52.919µs 2.51% 5.092 48.450 0.25% 0.009µs 1 200
normalization/normalize_service/normalize_service/test_ASCII throughput 18896691.123op/s 19367869.916op/s ± 47608.161op/s 19370248.166op/s ± 24636.770op/s 19396666.699op/s 19419515.748op/s 19431063.579op/s 19433573.075op/s 0.33% -4.913 46.087 0.25% 3366.405op/s 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... execution_time [621.153µs; 621.308µs] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/A0000000000000000000000000000000000000000000000000... throughput [1609510.030op/s; 1609908.988op/s] or [-0.012%; +0.012%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて execution_time [467.022µs; 467.171µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Data🐨dog🐶 繋がっ⛰てて throughput [2140549.161op/s; 2141228.875op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters execution_time [191.373µs; 191.433µs] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/Test Conversion 0f Weird !@#$%^&**() Characters throughput [5223762.254op/s; 5225409.333op/s] or [-0.016%; +0.016%] None None None
normalization/normalize_service/normalize_service/[empty string] execution_time [47.401µs; 47.435µs] or [-0.036%; +0.036%] None None None
normalization/normalize_service/normalize_service/[empty string] throughput [21081406.015op/s; 21096583.960op/s] or [-0.036%; +0.036%] None None None
normalization/normalize_service/normalize_service/test_ASCII execution_time [51.614µs; 51.650µs] or [-0.034%; +0.034%] None None None
normalization/normalize_service/normalize_service/test_ASCII throughput [19361271.883op/s; 19374467.950op/s] or [-0.034%; +0.034%] None None None

Group 10

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
benching string interning on wordpress profile execution_time 137.139µs 137.924µs ± 0.284µs 137.903µs ± 0.139µs 138.041µs 138.403µs 138.830µs 139.467µs 1.13% 1.292 5.194 0.21% 0.020µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
benching string interning on wordpress profile execution_time [137.885µs; 137.964µs] or [-0.029%; +0.029%] None None None

Group 11

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
two way interface execution_time 18.460µs 25.668µs ± 13.720µs 21.541µs ± 2.962µs 23.011µs 47.896µs 50.964µs 139.161µs 546.04% 4.501 29.586 53.32% 0.970µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
two way interface execution_time [23.767µs; 27.570µs] or [-7.408%; +7.408%] None None None

Group 12

cpu_model git_commit_sha git_commit_date git_branch
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz f4b295e 1732289778 dsn/crashtracker-rfc5-telemetry
scenario metric min mean ± sd median ± mad p75 p95 p99 max peak_to_median_ratio skewness kurtosis cv sem runs sample_size
sql/obfuscate_sql_string execution_time 68.458µs 68.781µs ± 0.167µs 68.766µs ± 0.037µs 68.806µs 68.870µs 69.154µs 70.396µs 2.37% 7.446 65.938 0.24% 0.012µs 1 200
scenario metric 95% CI mean Shapiro-Wilk pvalue Ljung-Box pvalue (lag=1) Dip test pvalue
sql/obfuscate_sql_string execution_time [68.758µs; 68.804µs] or [-0.034%; +0.034%] None None None

Baseline

Omitted due to size.

@codecov-commenter
Copy link

codecov-commenter commented Nov 14, 2024

Codecov Report

Attention: Patch coverage is 95.97523% with 13 lines in your changes missing coverage. Please review.

Project coverage is 70.84%. Comparing base (580c2f4) to head (f4b295e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #741      +/-   ##
==========================================
+ Coverage   70.57%   70.84%   +0.27%     
==========================================
  Files         296      297       +1     
  Lines       43246    43568     +322     
==========================================
+ Hits        30519    30865     +346     
+ Misses      12727    12703      -24     
Components Coverage Δ
crashtracker 43.83% <95.97%> (+6.68%) ⬆️
crashtracker-ffi 8.67% <ø> (ø)
datadog-alloc 98.73% <ø> (ø)
data-pipeline 91.98% <ø> (ø)
data-pipeline-ffi 0.00% <ø> (ø)
ddcommon 83.46% <ø> (ø)
ddcommon-ffi 69.12% <ø> (ø)
ddtelemetry 59.05% <ø> (ø)
ddtelemetry-ffi 22.13% <ø> (ø)
dogstatsd 89.45% <ø> (ø)
dogstatsd-client 79.77% <ø> (ø)
ipc 82.76% <ø> (-0.11%) ⬇️
profiling 84.30% <ø> (ø)
profiling-ffi 77.46% <ø> (ø)
serverless 0.00% <ø> (ø)
sidecar 38.01% <ø> (ø)
sidecar-ffi 0.00% <ø> (ø)
spawn-worker 50.36% <ø> (ø)
tinybytes 94.77% <ø> (ø)
trace-mini-agent 72.36% <ø> (ø)
trace-normalization 98.23% <ø> (ø)
trace-obfuscation 95.77% <ø> (ø)
trace-protobuf 77.67% <ø> (ø)
trace-utils 93.54% <ø> (ø)
---- 🚨 Try these New Features:

@danielsn danielsn force-pushed the dsn/crashtracker-rfc5-telemetry branch from 745232c to 5d60fdb Compare November 18, 2024 23:11
@danielsn danielsn force-pushed the dsn/crashtracker-rfc5-telemetry branch from 5d60fdb to 83ed9ae Compare November 18, 2024 23:12
@danielsn
Copy link
Contributor Author

Sample output looks like this:

{
    "api_version": "v2",
    "tracer_time": 1568898000,
    "runtime_id": "xyz",
    "seq_id": 1,
    "application": {
        "service_name": "foo",
        "service_version": "bar",
        "language_name": "native",
        "language_version": "unknown",
        "tracer_version": "unknown"
    },
    "host": {
        "hostname": "COMP-M63PJ2N3HN",
        "os": "macos",
        "os_version": "23.6.0"
    },
    "request_type": "logs",
    "payload": [
        {
            "message": "{\"counters\":{\"not_profiling\":0,\"collecting_sample\":1},\"data_schema_version\":\"1.0\",\"error\":{\"is_crash\":true,\"kind\":\"UnixSignal\",\"source_type\":\"Crashtracking\",\"stack\":{\"format\":\"Datadog Crashtracker 1.0\",\"frames\":[{\"ip\":\"0x0\",\"build_id\":\"abcde0x0\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo0\",\"column\":0,\"file\":\"banana0.rs\",\"function\":\"Bar::baz0\",\"line\":1},{\"ip\":\"0x1\",\"build_id\":\"abcde0x1\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo1\",\"column\":2,\"file\":\"banana1.rs\",\"function\":\"Bar::baz1\",\"line\":3},{\"ip\":\"0x2\",\"build_id\":\"abcde0x2\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo2\",\"column\":4,\"file\":\"banana2.rs\",\"function\":\"Bar::baz2\",\"line\":5},{\"ip\":\"0x3\",\"build_id\":\"abcde0x3\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo3\",\"column\":6,\"file\":\"banana3.rs\",\"function\":\"Bar::baz3\",\"line\":7},{\"ip\":\"0x4\",\"build_id\":\"abcde0x4\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo4\",\"column\":8,\"file\":\"banana4.rs\",\"function\":\"Bar::baz4\",\"line\":9},{\"ip\":\"0x5\",\"build_id\":\"abcde0x5\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo5\",\"column\":10,\"file\":\"banana5.rs\",\"function\":\"Bar::baz5\",\"line\":11},{\"ip\":\"0x6\",\"build_id\":\"abcde0x6\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo6\",\"column\":12,\"file\":\"banana6.rs\",\"function\":\"Bar::baz6\",\"line\":13},{\"ip\":\"0x7\",\"build_id\":\"abcde0x7\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo7\",\"column\":14,\"file\":\"banana7.rs\",\"function\":\"Bar::baz7\",\"line\":15},{\"ip\":\"0x8\",\"build_id\":\"abcde0x8\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo8\",\"column\":16,\"file\":\"banana8.rs\",\"function\":\"Bar::baz8\",\"line\":17},{\"ip\":\"0x9\",\"build_id\":\"abcde0x9\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo9\",\"column\":18,\"file\":\"banana9.rs\",\"function\":\"Bar::baz9\",\"line\":19}]}},\"incomplete\":true,\"metadata\":{\"library_name\":\"libdatadog\",\"library_version\":\"1.2.3\",\"family\":\"native\",\"tags\":[\"service:foo\",\"service_version:bar\",\"runtime-id:xyz\",\"language:native\"]},\"os_info\":{\"architecture\":\"unknown\",\"bitness\":\"unknown bitness\",\"os_type\":\"Unknown\",\"version\":\"Unknown\"},\"proc_info\":{\"pid\":1},\"sig_info\":{\"si_addr\":\"0x0000000000001234\",\"si_code\":1,\"si_code_human_readable\":\"SEGV_BNDERR\",\"si_signo\":11,\"si_signo_human_readable\":\"SIGSEGV\"},\"span_ids\":[{\"id\":\"42\",\"thread_name\":\"thread1\"},{\"id\":\"24\",\"thread_name\":\"thread2\"}],\"timestamp\":\"2019-09-19 13:00:00 UTC\",\"trace_ids\":[{\"id\":\"345\",\"thread_name\":\"thread111\"},{\"id\":\"666\",\"thread_name\":\"thread222\"}],\"uuid\":\"1d6b97cb-968c-40c9-af6e-e4b4d71e8781\"}",
            "level": "ERROR",
            "count": 1,
            "stack_trace": "{\"format\":\"Datadog Crashtracker 1.0\",\"frames\":[{\"ip\":\"0x0\",\"build_id\":\"abcde0x0\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo0\",\"column\":0,\"file\":\"banana0.rs\",\"function\":\"Bar::baz0\",\"line\":1},{\"ip\":\"0x1\",\"build_id\":\"abcde0x1\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo1\",\"column\":2,\"file\":\"banana1.rs\",\"function\":\"Bar::baz1\",\"line\":3},{\"ip\":\"0x2\",\"build_id\":\"abcde0x2\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo2\",\"column\":4,\"file\":\"banana2.rs\",\"function\":\"Bar::baz2\",\"line\":5},{\"ip\":\"0x3\",\"build_id\":\"abcde0x3\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo3\",\"column\":6,\"file\":\"banana3.rs\",\"function\":\"Bar::baz3\",\"line\":7},{\"ip\":\"0x4\",\"build_id\":\"abcde0x4\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo4\",\"column\":8,\"file\":\"banana4.rs\",\"function\":\"Bar::baz4\",\"line\":9},{\"ip\":\"0x5\",\"build_id\":\"abcde0x5\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo5\",\"column\":10,\"file\":\"banana5.rs\",\"function\":\"Bar::baz5\",\"line\":11},{\"ip\":\"0x6\",\"build_id\":\"abcde0x6\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo6\",\"column\":12,\"file\":\"banana6.rs\",\"function\":\"Bar::baz6\",\"line\":13},{\"ip\":\"0x7\",\"build_id\":\"abcde0x7\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo7\",\"column\":14,\"file\":\"banana7.rs\",\"function\":\"Bar::baz7\",\"line\":15},{\"ip\":\"0x8\",\"build_id\":\"abcde0x8\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo8\",\"column\":16,\"file\":\"banana8.rs\",\"function\":\"Bar::baz8\",\"line\":17},{\"ip\":\"0x9\",\"build_id\":\"abcde0x9\",\"build_id_type\":\"GNU\",\"file_type\":\"ELF\",\"path\":\"/usr/bin/foo9\",\"column\":18,\"file\":\"banana9.rs\",\"function\":\"Bar::baz9\",\"line\":19}]}",
            "tags": "data_schema_version:1.0,incomplete:true,is_crash:true,uuid:1d6b97cb-968c-40c9-af6e-e4b4d71e8781,not_profiling:0,collecting_sample:1,si_addr:0x0000000000001234,si_code:1,si_code_human_readable:SEGV_BNDERR,si_signo:1,si_signo_human_readable:SEGV_BNDERR",
            "is_sensitive": true
        }
    ]
}

@danielsn danielsn force-pushed the dsn/crashtracker-rfc5-telemetry branch from ccfae94 to 206fe5b Compare November 19, 2024 14:57
@danielsn danielsn enabled auto-merge (squash) November 22, 2024 15:36
@danielsn danielsn merged commit 191a68a into main Nov 22, 2024
32 checks passed
@danielsn danielsn deleted the dsn/crashtracker-rfc5-telemetry branch November 22, 2024 16:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants