[runtime] Add an in-memory cache for Benchmark protos.

[ChrisCummins]

This will be used by the CompilationSession runtime to keep track of
the Benchmark protobufs that have been sent by the user to the
service, so that CompilationSession::init() can be passed a benchmark
proto.

This is a generalization of the BenchmarkFactory class that is used by
the LLVM service to keep a bunch of llvm::Modules loaded in
memory. The same class is implemented twice in C++ and Python using
the same semantics and with the same tests.

The cache has a target maximum size based on the number of bytes of
its elements. When this size is reached, benchamrks are evicted using
a random policy. The idea behind random cache eviction is that this
cache will be large enough by default to store a good number of
benchmarks, so exceeding the max cache size implies a training loop in
which random programs are selected from a very large pool, rather than
smaller pool where an LRU policy would be better.

Issue #254.

[codecov-commenter]

Codecov Report

Merging #263 (d8715f6) into development (05e56cb) will increase coverage by 0.13%.
The diff coverage is 91.78%.

@@               Coverage Diff               @@
##           development     #263      +/-   ##
===============================================
+ Coverage        83.32%   83.46%   +0.13%     
===============================================
  Files               76       78       +2     
  Lines             4353     4426      +73     
===============================================
+ Hits              3627     3694      +67     
- Misses             726      732       +6     

Impacted Files	Coverage Δ
compiler_gym/service/compilation_session.py	64.70% <64.70%> (ø)
compiler_gym/service/__init__.py	100.00% <100.00%> (ø)
compiler_gym/service/runtime/benchmark_cache.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0aa1051...d8715f6. Read the comment docs.

[hughleat]

LGTM

[hughleat]

Consider (but not for long, you know what I'm like):

class BenchmarkCache {
public:
  BenchmarkCache(std::function<Benchmark(string url)> benchmarkCreatorThingie, ...)...
  ...
  Benchmark& operator[] (string url) {
    if (url in cache) return cache[url];
    auto b = benchmarkCreatorThingie(url); // May throw std::out_of_range like std::vector.at
    if (doesn't have capacity) {
      evictUntilCapacity(b.size());
    }
    cache[url] = b;
    return b;
  }
}```
    

[ChrisCummins]

I don't know if that makes sense for this use case because there is no std::function<Benchmark(string url)> benchmarkCreatorThingie callback that would make sense. If the benchmark can't be found, we return an error to the frontend. Pseudo-code:

class CompilerGymRuntime:
    def session_rpc_endpoint(self, benchmark_uri):
        if benchmark_uri not in self.benchmark_cache:
            return ErrorCode.NOT_FOUND
        benchmark = self.benchmark_cache[benchmark_uri]
        ...

[hughleat]

Ignorable: I always thought the advantages of random over lru were that random doesn't need to keep the recently used list. If the data items are small that makes sense. Otherwise, isn't lru pretty much always better?

[ChrisCummins]

I figured that there are two use common cases for this cache: (1) in a tight loop over a couple hundred benchmarks that will all fit in cache (2) over a massive set of training programs where the chance of a cache hit is negligible.

Given that, it didn't seem to me like LRU would be much of an advantage. I also found this interesting: "A random eviction policy degrades gracefully as the loop gets too big." from https://danluu.com/2choices-eviction/

Disclaimer: I know nothing about cache eviction policies and have utterly no idea what I'm talking about :)