Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[coreinternal/pdatautil] Reduce allocations in Map/ValueHash functions #17827

Merged

Conversation

dmitryax
Copy link
Member

@dmitryax dmitryax commented Jan 16, 2023

Use a buffer pool instead of making allocations each time. The remaining allocations are caused by sort.Strings and pdata.BytesSlice.AsRaw.

Before:

goos: darwin
goarch: arm64
pkg: github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/pdatautil
BenchmarkMapHashFourItems
BenchmarkMapHashFourItems-10                  	 3136358	       383.7 ns/op	     288 B/op	      11 allocs/op
BenchmarkMapHashEightItems
BenchmarkMapHashEightItems-10                 	 1588498	       777.5 ns/op	     456 B/op	      19 allocs/op
BenchmarkMapHashWithEmbeddedSliceAndMap
BenchmarkMapHashWithEmbeddedSliceAndMap-10    	  933703	      1295 ns/op	     784 B/op	      32 allocs/op
PASS

After:

pkg: github.com/open-telemetry/opentelemetry-collector-contrib/internal/coreinternal/pdatautil
BenchmarkMapHashFourItems
BenchmarkMapHashFourItems-10                  	 5163958	       226.0 ns/op	      24 B/op	       1 allocs/op
BenchmarkMapHashEightItems
BenchmarkMapHashEightItems-10                 	 2389960	       502.4 ns/op	      32 B/op	       2 allocs/op
BenchmarkMapHashWithEmbeddedSliceAndMap
BenchmarkMapHashWithEmbeddedSliceAndMap-10    	 1497609	       799.8 ns/op	      56 B/op	       3 allocs/op
PASS

@dmitryax dmitryax requested review from a team and bogdandrutu January 16, 2023 22:17
@dmitryax dmitryax added the Skip Changelog PRs that do not require a CHANGELOG.md entry label Jan 16, 2023
@dmitryax dmitryax changed the title [coreinternal/pdatautil] Reduce number of allocations in Map/ValueHash functions [coreinternal/pdatautil] Reduce allocations made by Map/ValueHash functions Jan 16, 2023
@dmitryax dmitryax changed the title [coreinternal/pdatautil] Reduce allocations made by Map/ValueHash functions [coreinternal/pdatautil] Reduce allocations in Map/ValueHash functions Jan 16, 2023
@runforesight
Copy link

runforesight bot commented Jan 16, 2023

Foresight Summary

    
Major Impacts

TestValueHash ❌ failed 1 times in 4 runs (25% fail rate).
TestValueHash/empty_bytes ❌ failed 1 times in 4 runs (25% fail rate).
build-and-test-windows duration(5 seconds) has decreased 42 minutes 27 seconds compared to main branch avg(42 minutes 32 seconds).
View More Details

⭕  changelog workflow has finished in 3 seconds (2 minutes 1 second less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
changelog -     🔗  N/A See Details

⭕  build-and-test-windows workflow has finished in 5 seconds (42 minutes 27 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
windows-unittest-matrix -     🔗  N/A See Details
windows-unittest -     🔗  N/A See Details

✅  check-links workflow has finished in 39 seconds (57 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
changed files -     🔗  N/A See Details
check-links -     🔗  N/A See Details

✅  telemetrygen workflow has finished in 55 seconds (1 minute 19 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
publish-latest -     🔗  N/A See Details
build-dev -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  tracegen workflow has finished in 1 minute (1 minute 11 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
build-dev -     🔗  N/A See Details
publish-latest -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  prometheus-compliance-tests workflow has finished in 3 minutes 14 seconds (4 minutes 15 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
prometheus-compliance-tests -     🔗  ✅ 21  ❌ 0  ⏭ 0    🔗 See Details

✅  build-and-test workflow has finished in 37 minutes 8 seconds (10 minutes 4 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
unittest-matrix (1.19, internal) -     🔗  ✅ 651  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, internal) -     🔗  ✅ 651  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, processor) -     🔗  ✅ 1481  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, processor) -     🔗  ✅ 1481  ❌ 0  ⏭ 0    🔗 See Details
correctness-metrics -     🔗  ✅ 2  ❌ 0  ⏭ 0    🔗 See Details
correctness-traces -     🔗  ✅ 17  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, extension) -     🔗  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, extension) -     🔗  ✅ 528  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-0) -     🔗  ✅ 2565  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-0) -     🔗  ✅ 2565  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, exporter) -     🔗  ✅ 2468  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, exporter) -     🔗  ✅ 2468  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, other) -     🔗  ✅ 4454  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, other) -     🔗  ✅ 4454  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.19, receiver-1) -     🔗  ✅ 1894  ❌ 0  ⏭ 0    🔗 See Details
unittest-matrix (1.18, receiver-1) -     🔗  ✅ 1894  ❌ 0  ⏭ 0    🔗 See Details
integration-tests -     🔗  ✅ 55  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details
checks -     🔗  N/A See Details
check-codeowners -     🔗  N/A See Details
check-collector-module-version -     🔗  N/A See Details
lint-matrix (receiver-0) -     🔗  N/A See Details
lint-matrix (receiver-1) -     🔗  N/A See Details
lint-matrix (processor) -     🔗  N/A See Details
lint-matrix (exporter) -     🔗  N/A See Details
lint-matrix (extension) -     🔗  N/A See Details
lint-matrix (internal) -     🔗  N/A See Details
lint-matrix (other) -     🔗  N/A See Details
build-examples -     🔗  N/A See Details
lint -     🔗  N/A See Details
unittest (1.19) -     🔗  N/A See Details
unittest (1.18) -     🔗  N/A See Details
cross-compile (darwin, amd64) -     🔗  N/A See Details
cross-compile (darwin, arm64) -     🔗  N/A See Details
cross-compile (linux, 386) -     🔗  N/A See Details
cross-compile (linux, amd64) -     🔗  N/A See Details
cross-compile (linux, arm) -     🔗  N/A See Details
cross-compile (linux, arm64) -     🔗  N/A See Details
cross-compile (windows, 386) -     🔗  N/A See Details
cross-compile (linux, ppc64le) -     🔗  N/A See Details
cross-compile (windows, amd64) -     🔗  N/A See Details
build-package (deb) -     🔗  N/A See Details
build-package (rpm) -     🔗  N/A See Details
windows-msi -     🔗  N/A See Details
publish-check -     🔗  N/A See Details
publish-dev -     🔗  N/A See Details
publish-stable -     🔗  N/A See Details

✅  load-tests workflow has finished in 7 minutes 54 seconds (6 minutes 25 seconds less than main branch avg.) and finished at 16th Jan, 2023.


Job Failed Steps Tests
loadtest (TestTraceAttributesProcessor) -     🔗  ✅ 3  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestIdleMode) -     🔗  ✅ 1  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetric10kDPS|TestMetricsFromFile) -     🔗  ✅ 6  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs) -     🔗  ✅ 8  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestMetricResourceProcessor|TestTrace10kSPS) -     🔗  ✅ 12  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestTraceBallast1kSPSWithAttrs|TestTraceBallast1kSPSAddAttrs) -     🔗  ✅ 10  ❌ 0  ⏭ 0    🔗 See Details
loadtest (TestBallastMemory|TestLog10kDPS) -     🔗  ✅ 19  ❌ 0  ⏭ 0    🔗 See Details
setup-environment -     🔗  N/A See Details

🔎 See details on Foresight

*You can configure Foresight comments in your organization settings page.

@dmitryax dmitryax force-pushed the improve-pdata-hash-performance branch from b315935 to 6e68e08 Compare January 16, 2023 22:22
@MovieStoreGuy
Copy link
Contributor

This looks fine, the only concern I had which I checked isn't the case is trying to borrow the underlying slice which can lead to some weirdness on Reset.

@dmitryax dmitryax merged commit 956d956 into open-telemetry:main Jan 17, 2023
@dmitryax dmitryax deleted the improve-pdata-hash-performance branch January 21, 2023 21:17
@plantfansam plantfansam mentioned this pull request Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internal/core Skip Changelog PRs that do not require a CHANGELOG.md entry
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants