Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WAF telemetry #2735

Open
wants to merge 16 commits into
base: master
Choose a base branch
from
Open

WAF telemetry #2735

wants to merge 16 commits into from

Conversation

cataphract
Copy link
Contributor

Description

See individual commits for descriptions of the changes.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@cataphract cataphract requested review from a team as code owners June 27, 2024 10:26
@codecov-commenter
Copy link

codecov-commenter commented Jun 27, 2024

Codecov Report

Attention: Patch coverage is 36.88525% with 77 lines in your changes missing coverage. Please review.

Project coverage is 77.77%. Comparing base (4c3832d) to head (a225a90).

Additional details and impacted files

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #2735      +/-   ##
============================================
- Coverage     77.96%   77.77%   -0.19%     
  Complexity     2216     2216              
============================================
  Files           227      227              
  Lines         26643    26737      +94     
  Branches        989     1017      +28     
============================================
+ Hits          20771    20796      +25     
- Misses         5346     5408      +62     
- Partials        526      533       +7     
Flag Coverage Δ
appsec-extension 68.43% <36.45%> (-0.77%) ⬇️
tracer-extension 78.72% <38.46%> (-0.09%) ⬇️
tracer-php 80.55% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
appsec/src/extension/request_abort.c 72.28% <100.00%> (-0.10%) ⬇️
ext/configuration.h 100.00% <ø> (ø)
ext/sidecar.c 87.50% <100.00%> (+0.09%) ⬆️
ext/otel_config.c 79.50% <0.00%> (ø)
appsec/src/extension/commands/client_init.c 82.35% <0.00%> (-1.86%) ⬇️
appsec/src/extension/ddtrace.c 57.77% <34.78%> (-2.62%) ⬇️
ext/telemetry.c 84.41% <37.50%> (-8.92%) ⬇️
appsec/src/extension/commands_helpers.c 62.56% <35.82%> (-7.07%) ⬇️

... and 1 file with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c3832d...a225a90. Read the comment docs.

@pr-commenter
Copy link

pr-commenter bot commented Jun 27, 2024

Benchmarks

Benchmark execution time: 2024-07-13 00:47:21

Comparing candidate commit a225a90 in PR branch glopes/waf-telemetry with baseline commit 4c3832d in branch master.

Found 17 performance improvements and 0 performance regressions! Performance is the same for 161 metrics, 0 unstable metrics.

scenario:EmptyFileBench/benchEmptyFileBaseline

  • 🟩 execution_time [-398.670µs; -235.850µs] or [-14.264%; -8.439%]

scenario:EmptyFileBench/benchEmptyFileBaseline-opcache

  • 🟩 execution_time [-430.977µs; -266.503µs] or [-14.768%; -9.132%]

scenario:EmptyFileBench/benchEmptyFileOverhead

  • 🟩 execution_time [-500.727µs; -340.853µs] or [-15.926%; -10.841%]

scenario:EmptyFileBench/benchEmptyFileOverhead-opcache

  • 🟩 execution_time [-541.737µs; -402.743µs] or [-16.620%; -12.356%]

scenario:LaravelBench/benchLaravelBaseline

  • 🟩 execution_time [-419.144µs; -246.976µs] or [-14.387%; -8.478%]

scenario:LaravelBench/benchLaravelBaseline-opcache

  • 🟩 execution_time [-407.735µs; -267.945µs] or [-13.502%; -8.873%]

scenario:LaravelBench/benchLaravelOverhead

  • 🟩 execution_time [-606.339µs; -426.181µs] or [-18.143%; -12.753%]

scenario:LaravelBench/benchLaravelOverhead-opcache

  • 🟩 execution_time [-565.431µs; -362.429µs] or [-16.594%; -10.636%]

scenario:MessagePackSerializationBench/benchMessagePackSerialization-opcache

  • 🟩 execution_time [-7.998µs; -5.782µs] or [-3.347%; -2.419%]

scenario:SymfonyBench/benchSymfonyBaseline

  • 🟩 execution_time [-300.154µs; -218.546µs] or [-5.000%; -3.641%]

scenario:SymfonyBench/benchSymfonyBaseline-opcache

  • 🟩 execution_time [-314.024µs; -269.696µs] or [-5.139%; -4.413%]

scenario:SymfonyBench/benchSymfonyOverhead

  • 🟩 execution_time [-734.478µs; -678.602µs] or [-10.406%; -9.615%]

scenario:SymfonyBench/benchSymfonyOverhead-opcache

  • 🟩 execution_time [-720.026µs; -639.954µs] or [-10.068%; -8.948%]

scenario:TraceSerializationBench/benchSerializeTrace

  • 🟩 execution_time [-8.473µs; -5.227µs] or [-4.195%; -2.588%]

scenario:TraceSerializationBench/benchSerializeTrace-opcache

  • 🟩 mem_peak [-167.139KB; -62.645KB] or [-7.538%; -2.825%]

scenario:WordPressBench/benchWordPressOverhead

  • 🟩 execution_time [-11.984ms; -9.469ms] or [-5.473%; -4.325%]

scenario:WordPressBench/benchWordPressOverhead-opcache

  • 🟩 execution_time [-10.767ms; -8.338ms] or [-4.933%; -3.821%]

@cataphract cataphract force-pushed the glopes/waf-telemetry branch 2 times, most recently from 8464095 to d91cd3f Compare June 27, 2024 17:35
ext/telemetry.c Outdated
@@ -240,3 +247,142 @@ void ddtrace_telemetry_send_trace_api_metrics(trace_api_metrics metrics) {

ddog_sidecar_telemetry_buffer_flush(&ddtrace_sidecar, ddtrace_sidecar_instance_id, &dd_bgs_queued_id, buffer);
}

ZEND_TLS HashTable metric_buffers;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't it possible to use a buffer created by ddog_sidecar_telemetry_buffer_alloc and add the metrics directly into it instead of having 2 levels of buffers?
Then in dd_commit_metrics we could just flush it.

@cataphract cataphract force-pushed the glopes/waf-telemetry branch 5 times, most recently from e777ab6 to 8239978 Compare July 4, 2024 16:53
@cataphract cataphract closed this Jul 5, 2024
@cataphract cataphract reopened this Jul 5, 2024
@cataphract cataphract force-pushed the glopes/waf-telemetry branch 9 times, most recently from 6b37ec2 to ad72485 Compare July 11, 2024 17:19
cataphract and others added 5 commits July 12, 2024 13:54
* Support aarch64
* Show the log of the sidecar
* Update dependencies
* Compile rust code in the tracer in debug mode
* Fix build failing when gradle finds broken symlinks
* Add PHP debug images (for troubleshooting)

The images are now multiarch. See https://registry.hub.docker.com/r/datadog/dd-appsec-php-ci/tags
@cataphract cataphract force-pushed the glopes/waf-telemetry branch 2 times, most recently from 583b4db to 1b22395 Compare July 12, 2024 23:50
Introduces the interface metrics::TelemetrySubmitter and uses two
implementations to submit metrics 1) RequestMetricsSubmitter: generated
during a request, which are forwarded to the extension in that
requests's rshutdown message and
2) MetricsImpl: generated outside a request, inside the service. These
are forwarded to the extension in either client_init or rshutdown of any
client of that service.

This service is also used to submit the legacy meta and span metrics.
asio synchronous network APIs don't support timeouts. Switch to async
APIs with stackless C++20 coroutine executor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants