Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

profiler: v3 upload and metrics #781

Merged
merged 6 commits into from
Jan 7, 2021
Merged

profiler: v3 upload and metrics #781

merged 6 commits into from
Jan 7, 2021

Conversation

pmbauer
Copy link
Member

@pmbauer pmbauer commented Nov 25, 2020

The new format replaces ordinal parallel arrays for the attachments,
substituting canonical names for the attachment types.

  • heap
  • cpu
  • block
  • mutex
  • goroutines

Example upload

 curl -i \
  -H "DD-API-KEY:redacted" \
  -F recording-start=$(date -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F recording-end=$(date -d '+1 min' -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F tags\[\]=host:test-host \
  -F tags\[\]=service:test-service \
  -F tags\[\]=language:go \
  -F data\[heap.pprof\]=@heap.pprof \
  -F data\[cpu.pprof\]=@cpu.pprof \
  -F data\[goroutines.pprof\]=@goroutines.pprof \
  -F data\[block.pprof\]=@block.pprof \
  -F data\[mutex.pprof\]=@mutex.pprof \
  -F version=3 \
  -F family=go \
  https://intake.profile.datadoghq.com/v1/input

@pmbauer pmbauer added this to the 1.28.0 milestone Nov 25, 2020
@pmbauer
Copy link
Member Author

pmbauer commented Nov 25, 2020

NB: this pull is not ready until intake backend code to support the change is deployed.

@gbbr gbbr changed the title profile V3 upload format profiler: v3 upload format Nov 26, 2020
Copy link
Contributor

@gbbr gbbr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This seems like an improvement :)

profiler/profiler.go Show resolved Hide resolved
@knusbaum knusbaum modified the milestones: 1.28.0, 1.29.0 Dec 10, 2020
@knusbaum knusbaum modified the milestones: 1.29.0, Triage Dec 22, 2020
The new format replaces ordinal parallel arrays for the attachments,
substituting canonical names for the attachment types.

- heap
- cpu
- block
- mutex
- goroutines

Example upload
```bash
 curl -i \
  -H "DD-API-KEY:redacted" \
  -F recording-start=$(date -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F recording-end=$(date -d '+1 min' -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F tags\[\]=host:test-host \
  -F tags\[\]=service:test-service \
  -F tags\[\]=language:go \
  -F data\[heap.pprof\]=@heap.pprof \
  -F data\[cpu.pprof\]=@cpu.pprof \
  -F data\[goroutines.pprof\]=@goroutines.pprof \
  -F data\[block.pprof\]=@block.pprof \
  -F data\[mutex.pprof\]=@mutex.pprof \
  -F version=3 \
  -F family=go \
  https://intake.profile.datadoghq.com/v1/input
```
* normalize upload filenames and string ids

* add metric profile and heap metrics

* add gc metrics
@pmbauer pmbauer changed the title profiler: v3 upload format profiler: v3 upload and metrics Jan 5, 2021
@pmbauer pmbauer marked this pull request as ready for review January 7, 2021 18:58
@pmbauer
Copy link
Member Author

pmbauer commented Jan 7, 2021

Gabriel initially screened this pull (profile v3 uploads) pending the server side changes being deployed to support the new uploads

Since then, #801 was approved and merged into the profiling-v3-branch since it depends on 781. The requisite server-side changes to support this client change are in production and we’ve also tested the changes with one of our own production services in prod

@pmbauer pmbauer requested a review from knusbaum January 7, 2021 19:14
@knusbaum knusbaum modified the milestones: Triage, 1.29.0 Jan 7, 2021
Copy link
Contributor

@knusbaum knusbaum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks fine to me.

profiler/metrics.go Show resolved Hide resolved
@pmbauer pmbauer merged commit 634b231 into v1 Jan 7, 2021
@pmbauer pmbauer deleted the pmbauer/profiling-v3-input branch January 8, 2021 21:01
dianashevchenko pushed a commit that referenced this pull request Feb 8, 2021
* profile V3 upload format

The new format replaces ordinal parallel arrays for the attachments,
substituting canonical names for the attachment types.

- heap
- cpu
- block
- mutex
- goroutines

Example upload
```bash
 curl -i \
  -H "DD-API-KEY:redacted" \
  -F recording-start=$(date -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F recording-end=$(date -d '+1 min' -u --iso-8601=seconds | sed 's/\+.*/Z/') \
  -F tags\[\]=host:test-host \
  -F tags\[\]=service:test-service \
  -F tags\[\]=language:go \
  -F data\[heap.pprof\]=@heap.pprof \
  -F data\[cpu.pprof\]=@cpu.pprof \
  -F data\[goroutines.pprof\]=@goroutines.pprof \
  -F data\[block.pprof\]=@block.pprof \
  -F data\[mutex.pprof\]=@mutex.pprof \
  -F version=3 \
  -F family=go \
  https://intake.profile.datadoghq.com/v1/input
```

* drive-by clarifying comment on batch end field

* gofmt fixes

* profiles: add custom metrics payload (#801)

* normalize upload filenames and string ids

* add metric profile and heap metrics

* add gc metrics

* fixup copyright date
This was referenced Mar 13, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants