Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix with labels bug reported in issue #225 #227

Merged
merged 3 commits into from
Jan 9, 2022
Merged

Fix with labels bug reported in issue #225 #227

merged 3 commits into from
Jan 9, 2022

Commits on Jun 20, 2021

  1. Add tests to with_labels 

    This reproduces the error reported in issue #225, where `with_labels`
doesn't work at all the way it is intended, and in addition to that,
if you're using DirectFileStore, it can very easily corrupt your files.

The problem is the new metric received after calling `with_labels` is a
copy of the original, with the labels pre-set, but it has
**its own internal store**, so it doesn't actually impact the values of
the original metric, which is what gets exported by the registry.

Signed-off-by: Daniel Magliola <dmagliola@crystalgears.com>
    @dmagliola
    dmagliola committed Jun 20, 2021
    Configuration menu
    Copy the full SHA
    c8ad029 View commit details
    Browse the repository at this point in the history

  2. Use the original metric's store in with_labels clone 

    When calling `with_labels` on a metric object (let's call it "the original"),
we instantiate a new metric (the "clone") that is identical except that
it has some more pre-set labels, that allow the caller to observe it
without having to specify the labels every time.

"currying", if you will.

The problem with the existing code (as exemplified by issue #225, and by
the tests introduced in the previous commit), is that as part of making
this new metric, we end up instantiating a new store for this metric.

With in-memory stores, the new one will be empty. With file stores, it'll
bring over the data from the original metric until the point the clone
gets observed once, at which point they fork, while pointing at the same
file and keeping separate internal state. An almost sure recipe for file
corruption.

And when exporting, only the data in the "original" metric's store will
be exported, the clone's will be ignored, assuming files didn't get
corrupted.

The fix is not particularly elegant, but I don't see any way around it:
we replace the internal store of the "clone" metric with the one from
the "original", through the use of a protected method.

The only real alternative is getting rid of `with_labels`, which is a
nice-to-have for convenience and performance, but not a necessity.

Signed-off-by: Daniel Magliola <dmagliola@crystalgears.com>
    @dmagliola
    dmagliola committed Jun 20, 2021
    Configuration menu
    Copy the full SHA
    adf63e8 View commit details
    Browse the repository at this point in the history

  3. Explain the test that checks for file corruption 

    The test that reproduces our file corruption issue follows a very specific
set of steps to actually corrupt the files, and it's probably impossible
to figure out why.

Explaining those specific steps, however, requires explaining a lot about
the internals of `FileMappedDict`, of the files we store, and how corruption
happens in the first place, before we can understand why we need those
specific steps to reproduce it.

I hope this explanation makes sense.

Signed-off-by: Daniel Magliola <dmagliola@crystalgears.com>
    @dmagliola
    dmagliola committed Jun 20, 2021
    Configuration menu
    Copy the full SHA
    097cb04 View commit details
    Browse the repository at this point in the history