Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor dynamic config Client and Collection #3271

Merged
merged 19 commits into from
Aug 31, 2022
Merged

Refactor dynamic config Client and Collection #3271

merged 19 commits into from
Aug 31, 2022

Conversation

dnr
Copy link
Member

@dnr dnr commented Aug 26, 2022

What and why?

This is a large refactor of dynamic config. Conceptually think about like this:

At the top, the Collection interface is what's used by the server. That doesn't change at all.

Below that, Collection was implemented in terms of Client. Client mixed two concerns: getting values from an external source, and filtering constrained value sets. In practice, this had a few problems:

  • The filtering was too flexible: it accepted arbitrary sets of filters, when Collection only uses a few types of sets.
  • The interface was in terms of maps and slices, which made usage very allocation-heavy.
  • The interface was too high-level: alternative implementations had to duplicate filtering logic and type conversion.

The fix is to change Client to a lower-level interface, which is only responsible for getting values from a source. Collection is reimplemented on top of the new Client, with the filtering and type conversion logic consolidated, which greatly simplifies it.

The new Client interface returns constrained values, but with a more type-friendly and allocation-friendly style: constraints are represented as a plain struct, and can be compared easily. (Note that we can still add new constraint types without breaking source-compatibility, though alternative implementations will need to be changed to support those new constraints.)

The file based client is compatible with existing dynamic config yaml files: more logic to convert the yaml structure to the new constrained value structure was added to fileBasedClient.

Benchmark results:

before:

goos: linux
goarch: amd64
pkg: go.temporal.io/server/common/dynamicconfig
cpu: Intel(R) Xeon(R) W-10855M CPU @ 2.80GHz

BenchmarkCollection/global_int-12                2670097               495.8 ns/op            80 B/op          2 allocs/op
BenchmarkCollection/namespace_int-12             1098027              1017 ns/op             360 B/op          6 allocs/op
BenchmarkCollection/taskqueue_int-12              358624              3113 ns/op            1248 B/op         18 allocs/op
BenchmarkCollection/single_default-12             296676              3481 ns/op            1248 B/op         18 allocs/op
BenchmarkCollection/structured_default-12         271657              3807 ns/op            1415 B/op         20 allocs/op

after:

BenchmarkCollection/global_int-12               11011748               215.0 ns/op            48 B/op          1 allocs/op
BenchmarkCollection/namespace_int-12             5473850               206.1 ns/op            48 B/op          1 allocs/op
BenchmarkCollection/taskqueue_int-12             4934986               253.7 ns/op            48 B/op          1 allocs/op
BenchmarkCollection/single_default-12            3778916               295.2 ns/op            48 B/op          1 allocs/op
BenchmarkCollection/structured_default-12        2488480               475.8 ns/op            72 B/op          2 allocs/op

How did you test it?
converted unit tests to use new interfaces, existing integration tests

Potential risks
it's a big rewrite, there could be new bugs

@dnr dnr requested a review from a team as a code owner August 26, 2022 05:32
common/dynamicconfig/collection.go Show resolved Hide resolved
common/dynamicconfig/collection.go Show resolved Hide resolved
common/dynamicconfig/collection.go Outdated Show resolved Hide resolved
Comment on lines -41 to -56
// dynamic config for tests
testGetPropertyKey = "testGetPropertyKey"
testCaseInsensitivePropertyKey = "testCaseInsensitivePropertyKey"
testGetIntPropertyKey = "testGetIntPropertyKey"
testGetFloat64PropertyKey = "testGetFloat64PropertyKey"
testGetDurationPropertyKey = "testGetDurationPropertyKey"
testGetBoolPropertyKey = "testGetBoolPropertyKey"
testGetStringPropertyKey = "testGetStringPropertyKey"
testGetMapPropertyKey = "testGetMapPropertyKey"
testGetIntPropertyFilteredByNamespaceKey = "testGetIntPropertyFilteredByNamespaceKey"
testGetDurationPropertyFilteredByNamespaceKey = "testGetDurationPropertyFilteredByNamespaceKey"
testGetIntPropertyFilteredByTaskQueueInfoKey = "testGetIntPropertyFilteredByTaskQueueInfoKey"
testGetDurationPropertyFilteredByTaskQueueInfoKey = "testGetDurationPropertyFilteredByTaskQueueInfoKey"
testGetDurationPropertyStructuredDefaults = "testGetDurationPropertyStructuredDefaults"
testGetBoolPropertyFilteredByNamespaceIDKey = "testGetBoolPropertyFilteredByNamespaceIDKey"
testGetBoolPropertyFilteredByTaskQueueInfoKey = "testGetBoolPropertyFilteredByTaskQueueInfoKey"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

common/dynamicconfig/source.go Outdated Show resolved Hide resolved
@dnr
Copy link
Member Author

dnr commented Aug 30, 2022

After seeing what the changes look like in other repos, and more discussion, changing the interface name back to Client to avoid unnecessary source level churn

@dnr dnr changed the title Change dynamic config Client to Source Refactor dynamic config Client and Collection Aug 30, 2022
@dnr dnr merged commit bd4e420 into temporalio:master Aug 31, 2022
@dnr dnr deleted the dc1 branch August 31, 2022 01:28
yycptt added a commit that referenced this pull request Sep 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants