Avoiding high frequency logging

[oschaaf]

Sometimes there's error flow control in the hot path that deserves logging, but plain logging there
risks generating a large amount of log spam.
It would be great to have a low-barrier and easy-to-consume way to emit a single or bounded number of log lines in these cases.

[oschaaf]

/cc @htuch do you know if Envoy already has something in place for this?

[oschaaf]

Maybe something like ENVOY_LOG_ONCE(...) in this minimal PoC would work?:

#include <atomic>
#include <iostream>

#define ENVOY_LOG(...) std::cerr << "logline" << std::endl;

#define ENVOY_LOG_ONCE(...) \
do { \
    static std::atomic<bool>* logged = new std::atomic<bool>(false); \
    bool kExpected = false; \
    if (logged->compare_exchange_strong(kExpected /*expected value*/, true /*new value*/)) { \
        ENVOY_LOG(__VA_ARGS_); \
    } \
} while (0)

int main() {
    int i =0;
    while (i++ < 100) {
        ENVOY_LOG_ONCE("1");
        ENVOY_LOG_ONCE("2");
    }
}

// clang  -lstdc++ test.cc  && ./a.out
// Only two log lines will be emitted from the while loop above.

[oschaaf]

afterthought -- the PoC above attempts to be precise and logs exactly once in multi-threaded scenarios. But it might be possible to trade in precision for performance by looking into eventual consistency (or even just accepting logging once / thread as a reasonable bound). It may also be reasonable to emit another log message indicating that log messages of this type are being truncated.

[htuch]

@oschaaf please add this to upstream Envoy, this is an often requested feature. I think there are multiple variations that could make sence, ENVOY_LOG_ONCE, ENVOY_LOG_EVERY_N, ENVOY_LOG_EVERY_N_SECONDS.

[oschaaf]

@htuch proposed envoyproxy/envoy#12830