fix(enginenetx): periodically trim statistics #1317

bassosimone · 2023-09-27T09:36:36Z

If we're running the enginenetx code inside an environment that sprays ~random IP addresses during DNS lookup, we're going to end up using quite a large amount of statistics. While this amount of statistics could potentially be great in general, we need to be mindful of memory and disk occupation as well as of the overhead of serializing and deserializing.

To address these concerns, add the arbitrary rule that we don't want more than ten domain-endpoint entries and implement code to zap old and/or always-failing entries. The code will run when loading the entries, when serializing them, and periodically while the *Network is up and running.

See ooni/probe#2531

If we're running the enginenetx code inside an environment that sprays ~random IP addresses during DNS lookup, we're going to end up using quite a large amount of statistics. While this amount of statistics could potentially be great in general, we need to be mindful of memory and disk occupation as well as of the overhead of serializing and deserializing. To address these concerns, add the arbitrary rule that we don't want more than ten domain-endpoint entries and implement code to zap old and/or always-failing entries. The code will run when loading the entries, when serializing them, and periodically while the `*Network` is up and running. See ooni/probe#2531

We still need to see if we need to add new tests or modify existing tests because they're incomplete

internal/enginenetx/statsmanager_test.go

internal/enginenetx/statsmanager.go

internal/enginenetx/statsmanager_test.go

If we're running the enginenetx code inside an environment that sprays ~random IP addresses during DNS lookup, we're going to end up using quite a large amount of statistics. While this amount of statistics could potentially be great in general, we need to be mindful of memory and disk occupation as well as of the overhead of serializing and deserializing. To address these concerns, add the arbitrary rule that we don't want more than ten domain-endpoint entries and implement code to zap old and/or always-failing entries. The code will run when loading the entries, when serializing them, and periodically while the `*Network` is up and running. See ooni/probe#2531

bassosimone requested a review from hellais as a code owner September 27, 2023 09:36

bassosimone added 14 commits September 27, 2023 11:40

x

7026e7e

x

0dcec35

x

3a03062

x

07e76f8

fix: repair existing tests

4cebfde

We still need to see if we need to add new tests or modify existing tests because they're incomplete

x

af6cd27

x

2ecd1d3

fix: make trim interval testable

67f9502

x

59e7979

x

fbf5c79

add more unit tests

3062e05

more unit tests

b77a7e6

more unit tests

bbdc4d2

test that we prune in the background

b83f2e3