Transitioning Rules/Alerts to the objstore client

[jtlisi]

As discussed in past community meetings there are two Object Storage client implementations used by the Cortex project. The chunks package implementation and the Thanos objstore implementation.

Currently, we use the Chunks client for:

• Chunks Backend
• Rules Storage
• Alertmanager Storage
• Purger Storage

We use the blocks client for the TSDB blocks backend.

I want to take up the work of migrating the Alertmanager and Rules Storage backends to use the same client as the TSDB blocks backend. The goal of this is to:

• Make the object storage clients in the configuration file
• Reduce the surface area for required changes to a client to support different features of a backend.
• Simplify instrumentation (Unify metrics for object stores #2478)

I wanted to use this discussion to figure out what the best course of attack would be for tackling #2262. Since this will be a breaking change I'm wondering how we could keep compatibility for two versions before deprecating the old config. For the GCS client this is trivial since the configs are very similar. For S3 there are some features in the chunk client that do not exist in the TSDB objstore client. How would we go about deprecating those features without having a breaking config change for two versions?

Some ideas I have include:

• Separate config block for new clients, remove old blocks after two versions
• Compatibility config structs with fields for both, use the correct client based on the provided implementation. Remove deprecated fields until only the TSDB client config struct remains

[pstibrany]

Separate config block for new clients, remove old blocks after two versions

I vote for this option. Seems like cleanest to me, at the cost of having two different object-store clients for two releases.

Note that Thanos objstore client currently doesn't support recursive listing used by our own clients, so that is something we may want to implement there first.

[jtlisi]

Note that Thanos objstore client currently doesn't support recursive listing used by our own clients, so that is something we may want to implement there first.

Makes sense, I was able to make things work here: #2489. However, there have been some recent changes to the rules backend and I'm not sure if that PR is compatible anymore. In general though, it involves nesting lists which is not ideal. A better approach is needed

[pracucci]

Separate config block for new clients, remove old blocks after two versions

I also vote for this.