Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize marshalTags() #2131

Merged
merged 1 commit into from
Mar 31, 2015
Merged

Optimize marshalTags() #2131

merged 1 commit into from
Mar 31, 2015

Conversation

benbjohnson
Copy link
Contributor

Overview

This pull request optimizes the marshaling of tag sets. Previously the strings.Join() function was used with added a lot of memory allocations. The new implementation manually allocates and appends to a byte slice.

The number of allocations has dropped to 3:

  • keys := make([]string, 0, len(tags))
  • sort.Strings(keys)
  • b := make([]byte, sz)

The new implementation manually handles the append instead of using the built-in append() function because appending a string to a []byte requires a memory allocation whereas a copy() does not.

A table test suite was also added to verify correctness.

Benchmark

Previous implementation

$ go test -bench=BenchmarkMarshalTags -v
PASS
BenchmarkMarshalTags_KeyN1   2000000           824 ns/op         176 B/op          6 allocs/op
BenchmarkMarshalTags_KeyN3   1000000          1319 ns/op         416 B/op          6 allocs/op
BenchmarkMarshalTags_KeyN5   1000000          2173 ns/op         656 B/op          6 allocs/op
BenchmarkMarshalTags_KeyN10   500000          3929 ns/op        1280 B/op          6 allocs/op

New implementation

$ go test -bench=BenchmarkMarshalTags -v
PASS
BenchmarkMarshalTags_KeyN1   3000000           458 ns/op          80 B/op          3 allocs/op
BenchmarkMarshalTags_KeyN3   2000000           753 ns/op         160 B/op          3 allocs/op
BenchmarkMarshalTags_KeyN5   1000000          1193 ns/op         240 B/op          3 allocs/op
BenchmarkMarshalTags_KeyN10   500000          2477 ns/op         448 B/op          3 allocs/op

This commit optimizes the marshaling of tag sets. Previously the
strings.Join() function was used with added a lot of memory allocations.
The new implementation manually allocates and appends to a byte slice.

Previous implementation:

$ go test -run=^\$ -bench=BenchmarkMarshalTags -v
PASS
BenchmarkMarshalTags_KeyN1	 2000000	       824 ns/op	     176 B/op	       6 allocs/op
BenchmarkMarshalTags_KeyN3	 1000000	      1319 ns/op	     416 B/op	       6 allocs/op
BenchmarkMarshalTags_KeyN5	 1000000	      2173 ns/op	     656 B/op	       6 allocs/op
BenchmarkMarshalTags_KeyN10	  500000	      3929 ns/op	    1280 B/op	       6 allocs/op

New implementation:

$ go test -run=^\$ -bench=BenchmarkMarshalTags -v
PASS
BenchmarkMarshalTags_KeyN1	 3000000	       458 ns/op	      80 B/op	       3 allocs/op
BenchmarkMarshalTags_KeyN3	 2000000	       753 ns/op	     160 B/op	       3 allocs/op
BenchmarkMarshalTags_KeyN5	 1000000	      1193 ns/op	     240 B/op	       3 allocs/op
BenchmarkMarshalTags_KeyN10	  500000	      2477 ns/op	     448 B/op	       3 allocs/op
@toddboom
Copy link
Contributor

Nice. :shipit:

benbjohnson added a commit that referenced this pull request Mar 31, 2015
@benbjohnson benbjohnson merged commit 58cd43d into master Mar 31, 2015
@benbjohnson benbjohnson deleted the marshalTags branch March 31, 2015 23:07
mark-rushakoff pushed a commit that referenced this pull request Jan 11, 2019
…#2131)

* refactor(query/functions/outputs): refactor to.go  to use arrow data structures

* update flux to latest master
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants