Add an option to log multihttp responses #550

rdubrock · 2023-10-30T21:16:08Z

One of the main points of feedback we've gotten is that it would be nice to have more visibility into what's happening during the request. Logging response bodies is tricky because the can be 1.) gigantic or 2.) contain sensitive information. However, it would be super useful for debugging

I feel that by making logging response bodies an opt in option on a per request basis, and limiting to 1000 characters would give reasonable protection from both of these problems.

mem · 2023-10-30T21:22:37Z

pkg/pb/synthetic_monitoring/checks.proto

+  MultiHttpEntryRequest            request     = 1 [(gogoproto.jsontag) = "request,omitempty"];    // The request parameters.
+  repeated MultiHttpEntryAssertion assertions  = 2 [(gogoproto.jsontag) = "checks,omitempty"];     // Zero or more assertions to be made on the response.
+  repeated MultiHttpEntryVariable  variables   = 3 [(gogoproto.jsontag) = "variables,omitempty"];  // Zero or more variables to be used in the request.
+  bool                             logResponse = 4 [(gogoproto.jsontag) = "logResponse,omitempty"];  // Whether to log the response body in the output.


Since there might be multiple requests, wouldn't it be better to call the field LogResponseBodies or LogResponses. I'm slightly in favor of the former, because it specifies what exactly will get added.

I was thinking this would be an option on a per request basis instead of all or nothing

Could be kind of a pain to specify for every request, though

I thought about that.

Why would anyone want to have the body to be logged? Mostly for debugging. In that case, I imagine they want all the responses.

Why would anyone not want a specific response logged? Because there's something sensitive in there that they don't want uploaded to Loki. In that case, why would they want the other responses logged?

If this is for debugging (the test button), the frontend can rework the request so that it includes the option in each response.

Also, the frontend could have a "log all responses" button that causes all the individual options to be set.

Ok yeah I'm convinced. I'll rework it to be a global option

…into log-response-option

* Add an option to log multihttp responses (#550) * Update to grafana-build-tools v0.6.0 * Chore(deps): Bump golang.org/x/net from 0.20.0 to 0.21.0 * Chore(deps): Bump github.com/rs/zerolog from 1.31.0 to 1.32.0 * Chore(deps): Bump github.com/miekg/dns from 1.1.57 to 1.1.58 * Fix: add test for HTTP check with a long URL * Add tenant label limits (#581) Signed-off-by: ka3de <danijs12@hotmail.com>

* Add an option to log multihttp responses (#550) * Update to grafana-build-tools v0.6.0 * Chore(deps): Bump golang.org/x/net from 0.20.0 to 0.21.0 * Chore(deps): Bump github.com/rs/zerolog from 1.31.0 to 1.32.0 * Chore(deps): Bump github.com/miekg/dns from 1.1.57 to 1.1.58 * Fix: add test for HTTP check with a long URL * Add tenant label limits (#581) * Release v0.20.0 (#613) * Fix global tenant id usage for label limits query Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>

* Add an option to log multihttp responses (#550) * Update to grafana-build-tools v0.6.0 * Chore(deps): Bump golang.org/x/net from 0.20.0 to 0.21.0 * Chore(deps): Bump github.com/rs/zerolog from 1.31.0 to 1.32.0 * Chore(deps): Bump github.com/miekg/dns from 1.1.57 to 1.1.58 * Fix: add test for HTTP check with a long URL * Add tenant label limits (#581) * Release v0.20.0 (#613) * Fix global tenant id usage for label limits query * Release v0.20.1 * Feature: promote adhoc to permanent feature (#615) * Update release script (#617) * Chore(deps): Bump github.com/mccutchen/go-httpbin/v2 (#618) * Fix: missing http check regex validations (#612) Signed-off-by: Marcelo E. Magallon <marcelo.magallon@grafana.com>

* Add an option to log multihttp responses (#550) * Update to grafana-build-tools v0.6.0 * Chore(deps): Bump golang.org/x/net from 0.20.0 to 0.21.0 * Chore(deps): Bump github.com/rs/zerolog from 1.31.0 to 1.32.0 * Chore(deps): Bump github.com/miekg/dns from 1.1.57 to 1.1.58 * Fix: add test for HTTP check with a long URL * Add tenant label limits (#581) * Release v0.20.0 (#613) * Fix global tenant id usage for label limits query * Release v0.20.1 * Feature: promote adhoc to permanent feature (#615) * Update release script (#617) * Chore(deps): Bump github.com/mccutchen/go-httpbin/v2 (#618) * Fix: missing http check regex validations (#612) * Release v0.20.2 (#620) * Chore(deps): Bump github.com/mccutchen/go-httpbin/v2 (#626) * change the name of the k6 check type to scripted (#622) * Chore(deps): Bump github.com/prometheus/client_model from 0.5.0 to 0.6.0 * Chore(deps): Bump google.golang.org/grpc from 1.61.0 to 1.61.1 * Chore(deps): Bump github.com/prometheus/common from 0.46.0 to 0.47.0 * Chore(deps): Bump github.com/prometheus/prometheus from 0.49.1 to 0.50.0 * Add header to ease tracking HTTP requests (#624) * Add telemetry protobuf definitions (#627) Signed-off-by: ka3de <danijs12@hotmail.com>

rdubrock added 2 commits October 30, 2023 12:30

add the option to include response bodies in the log output

93d3a91

add response body log to template

3c0da9f

rdubrock requested a review from a team as a code owner October 30, 2023 21:16

mem reviewed Oct 30, 2023

View reviewed changes

rdubrock added 2 commits October 30, 2023 14:03

make response logging a global option

7af7448

include LogResponseBodies in script test

4f50612

rdubrock mentioned this pull request Oct 31, 2023

feat: add the option to log multihttp request responses grafana/synthetic-monitoring-app#635

Closed

rdubrock added 2 commits November 2, 2023 11:30

Merge branch 'main' of github.com:grafana/synthetic-monitoring-agent …

9d59636

…into log-response-option

regenerate protobuf definitions

ab86278

rdubrock force-pushed the log-response-option branch from 6316d44 to ab86278 Compare February 7, 2024 20:51

regenerate again

5aafcef

mem approved these changes Feb 7, 2024

View reviewed changes

rdubrock merged commit 2584f5d into main Feb 7, 2024
4 checks passed

rdubrock deleted the log-response-option branch February 7, 2024 22:45

ka3de mentioned this pull request Feb 12, 2024

Release v0.20.0 #613

Merged

mem mentioned this pull request Feb 12, 2024

Release v0.20.1 #616

Merged

mem mentioned this pull request Feb 14, 2024

Release v0.20.2 #620

Merged

ka3de mentioned this pull request Feb 26, 2024

Release v0.21.0 #632

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an option to log multihttp responses #550

Add an option to log multihttp responses #550

rdubrock commented Oct 30, 2023

mem Oct 30, 2023

rdubrock Oct 30, 2023

rdubrock Oct 30, 2023

mem Oct 30, 2023

rdubrock Oct 30, 2023

Add an option to log multihttp responses #550

Add an option to log multihttp responses #550

Conversation

rdubrock commented Oct 30, 2023

mem Oct 30, 2023

Choose a reason for hiding this comment

rdubrock Oct 30, 2023

Choose a reason for hiding this comment

rdubrock Oct 30, 2023

Choose a reason for hiding this comment

mem Oct 30, 2023

Choose a reason for hiding this comment

rdubrock Oct 30, 2023

Choose a reason for hiding this comment