Remove support for ephemeral storage

CHANGELOG.md

@@ -8,6 +8,22 @@
 * [CHANGE] Store-gateway: When a query hits `max_fetched_chunks_per_query` and `max_fetched_series_per_query` limits, an error with the status code `422` is created and returned. #4056
 * [CHANGE] Packaging: Migrate FPM packaging solution to NFPM. Rationalize packages dependencies and add package for all binaries. #3911
 * [CHANGE] Store-gateway: Deprecate flag `-blocks-storage.bucket-store.chunks-cache.subrange-size` since there's no benefit to changing the default of `16000`. #4135
+* [CHANGE] Experimental support for ephemeral storage introduced in Mimir 2.6.0 has been removed. Following options are no longer available: #4252
+  * `-blocks-storage.ephemeral-tsdb.*`
+  * `-distributor.ephemeral-series-enabled`
+  * `-distributor.ephemeral-series-matchers`
+  * `-ingester.max-ephemeral-series-per-user`
+  * `-ingester.instance-limits.max-ephemeral-series`
+Querying with using `{__mimir_storage__="ephemeral"}` selector no longer works. All label values with `ephemeral-` prefix in `reason` label of `cortex_discarded_samples_total` metric are no longer available. Following metrics have been removed:
+  * `cortex_ingester_ephemeral_series`
+  * `cortex_ingester_ephemeral_series_created_total`
+  * `cortex_ingester_ephemeral_series_removed_total`
+  * `cortex_ingester_ingested_ephemeral_samples_total`
+  * `cortex_ingester_ingested_ephemeral_samples_failures_total`
+  * `cortex_ingester_memory_ephemeral_users`
+  * `cortex_ingester_queries_ephemeral_total`
+  * `cortex_ingester_queried_ephemeral_samples`
+  * `cortex_ingester_queried_ephemeral_series`
 * [FEATURE] Ruler: added `keep_firing_for` support to alerting rules. #4099
 * [FEATURE] Distributor, ingester: ingestion of native histograms. The new per-tenant limit `-ingester.native-histograms-ingestion-enabled` controls whether native histograms are stored or ignored. #4159
 * [FEATURE] Query-frontend: Introduce experimental `-query-frontend.query-sharding-target-series-per-shard` to allow query sharding to take into account cardinality of similar requests executed previously. #4121 #4177 #4188

cmd/mimir/config-descriptor.json

@@ -1695,17 +1695,6 @@
           ],
           "fieldValue": null,
           "fieldDefaultValue": null
-        },
-        {
-          "kind": "field",
-          "name": "ephemeral_series_enabled",
-          "required": false,
-          "desc": "Enable marking series as ephemeral based on the given matchers in the runtime config.",
-          "fieldValue": null,
-          "fieldDefaultValue": false,
-          "fieldFlag": "distributor.ephemeral-series-enabled",
-          "fieldType": "boolean",
-          "fieldCategory": "experimental"
         }
       ],
       "fieldValue": null,
@@ -2780,17 +2769,6 @@
               "fieldType": "int",
               "fieldCategory": "advanced"
             },
-            {
-              "kind": "field",
-              "name": "max_ephemeral_series",
-              "required": false,
-              "desc": "Max ephemeral series that this ingester can hold (across all tenants). Requests to create additional ephemeral series will be rejected. 0 = unlimited.",
-              "fieldValue": null,
-              "fieldDefaultValue": 0,
-              "fieldFlag": "ingester.instance-limits.max-ephemeral-series",
-              "fieldType": "int",
-              "fieldCategory": "experimental"
-            },
             {
               "kind": "field",
               "name": "max_inflight_push_requests",
@@ -3043,17 +3021,6 @@
           "fieldFlag": "ingester.max-global-series-per-metric",
           "fieldType": "int"
         },
-        {
-          "kind": "field",
-          "name": "max_ephemeral_series_per_user",
-          "required": false,
-          "desc": "The maximum number of in-memory ephemeral series per tenant, across the cluster before replication. 0 to disable ephemeral storage.",
-          "fieldValue": null,
-          "fieldDefaultValue": 0,
-          "fieldFlag": "ingester.max-ephemeral-series-per-user",
-          "fieldType": "int",
-          "fieldCategory": "experimental"
-        },
         {
           "kind": "field",
           "name": "max_global_metadata_per_user",
@@ -3587,17 +3554,6 @@
           "fieldValue": null,
           "fieldDefaultValue": {},
           "fieldType": "map of string to validation.ForwardingRule"
-        },
-        {
-          "kind": "field",
-          "name": "ephemeral_series_matchers",
-          "required": false,
-          "desc": "Lists of series matchers prefixed by the source. The source must be one of any, api, rule. If an incoming sample matches at least one of the matchers with its source it gets marked as ephemeral. The format of the value looks like: api:{namespace=\"dev\"};rule:{host=\"server1\",namespace=\"prod\"}",
-          "fieldValue": null,
-          "fieldDefaultValue": {},
-          "fieldFlag": "distributor.ephemeral-series-matchers",
-          "fieldType": "map of source name (string) to series matchers ([]string)",
-          "fieldCategory": "experimental"
         }
       ],
       "fieldValue": null,
@@ -6078,104 +6034,6 @@
           ],
           "fieldValue": null,
           "fieldDefaultValue": null
-        },
-        {
-          "kind": "block",
-          "name": "ephemeral_tsdb",
-          "required": false,
-          "desc": "",
-          "blockEntries": [
-            {
-              "kind": "field",
-              "name": "retention_period",
-              "required": false,
-              "desc": "Retention of ephemeral series.",
-              "fieldValue": null,
-              "fieldDefaultValue": 600000000000,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.retention-period",
-              "fieldType": "duration",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_chunks_write_buffer_size_bytes",
-              "required": false,
-              "desc": "The write buffer size used by the head chunks mapper. Lower values reduce memory utilisation on clusters with a large number of tenants at the cost of increased disk I/O operations.",
-              "fieldValue": null,
-              "fieldDefaultValue": 4194304,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-chunks-write-buffer-size-bytes",
-              "fieldType": "int",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_chunks_end_time_variance",
-              "required": false,
-              "desc": "How much variance (as percentage between 0 and 1) should be applied to the chunk end time, to spread chunks writing across time. Doesn't apply to the last chunk of the chunk range. 0 means no variance.",
-              "fieldValue": null,
-              "fieldDefaultValue": 0,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-chunks-end-time-variance",
-              "fieldType": "float",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "stripe_size",
-              "required": false,
-              "desc": "The number of shards of series to use in TSDB (must be a power of 2). Reducing this will decrease memory footprint, but can negatively impact performance.",
-              "fieldValue": null,
-              "fieldDefaultValue": 16384,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.stripe-size",
-              "fieldType": "int",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_chunks_write_queue_size",
-              "required": false,
-              "desc": "The size of the write queue used by the head chunks mapper. Lower values reduce memory utilisation at the cost of potentially higher ingest latency. Value of 0 switches chunks mapper to implementation without a queue.",
-              "fieldValue": null,
-              "fieldDefaultValue": 1000000,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-chunks-write-queue-size",
-              "fieldType": "int",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_postings_for_matchers_cache_ttl",
-              "required": false,
-              "desc": "How long to cache postings for matchers in the Head and OOOHead. 0 disables the cache and just deduplicates the in-flight calls.",
-              "fieldValue": null,
-              "fieldDefaultValue": 10000000000,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-ttl",
-              "fieldType": "duration",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_postings_for_matchers_cache_size",
-              "required": false,
-              "desc": "Maximum number of entries in the cache for postings for matchers in the Head and OOOHead when ttl \u003e 0.",
-              "fieldValue": null,
-              "fieldDefaultValue": 100,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-size",
-              "fieldType": "int",
-              "fieldCategory": "experimental"
-            },
-            {
-              "kind": "field",
-              "name": "head_postings_for_matchers_cache_force",
-              "required": false,
-              "desc": "Force the cache to be used for postings for matchers in the Head and OOOHead, even if it's not a concurrent (query-sharding) call.",
-              "fieldValue": null,
-              "fieldDefaultValue": false,
-              "fieldFlag": "blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-force",
-              "fieldType": "boolean",
-              "fieldCategory": "experimental"
-            }
-          ],
-          "fieldValue": null,
-          "fieldDefaultValue": null
         }
       ],
       "fieldValue": null,

cmd/mimir/help-all.txt.tmpl

@@ -403,22 +403,6 @@ Usage of ./cmd/mimir/mimir:
     	How frequently to scan the bucket, or to refresh the bucket index (if enabled), in order to look for changes (new blocks shipped by ingesters and blocks deleted by retention or compaction). (default 15m0s)
   -blocks-storage.bucket-store.tenant-sync-concurrency int
     	Maximum number of concurrent tenants synching blocks. (default 10)
-  -blocks-storage.ephemeral-tsdb.head-chunks-end-time-variance float
-    	[experimental] How much variance (as percentage between 0 and 1) should be applied to the chunk end time, to spread chunks writing across time. Doesn't apply to the last chunk of the chunk range. 0 means no variance.
-  -blocks-storage.ephemeral-tsdb.head-chunks-write-buffer-size-bytes int
-    	[experimental] The write buffer size used by the head chunks mapper. Lower values reduce memory utilisation on clusters with a large number of tenants at the cost of increased disk I/O operations. (default 4194304)
-  -blocks-storage.ephemeral-tsdb.head-chunks-write-queue-size int
-    	[experimental] The size of the write queue used by the head chunks mapper. Lower values reduce memory utilisation at the cost of potentially higher ingest latency. Value of 0 switches chunks mapper to implementation without a queue. (default 1000000)
-  -blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-force
-    	[experimental] Force the cache to be used for postings for matchers in the Head and OOOHead, even if it's not a concurrent (query-sharding) call.
-  -blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-size int
-    	[experimental] Maximum number of entries in the cache for postings for matchers in the Head and OOOHead when ttl > 0. (default 100)
-  -blocks-storage.ephemeral-tsdb.head-postings-for-matchers-cache-ttl duration
-    	[experimental] How long to cache postings for matchers in the Head and OOOHead. 0 disables the cache and just deduplicates the in-flight calls. (default 10s)
-  -blocks-storage.ephemeral-tsdb.retention-period duration
-    	[experimental] Retention of ephemeral series. (default 10m0s)
-  -blocks-storage.ephemeral-tsdb.stripe-size int
-    	[experimental] The number of shards of series to use in TSDB (must be a power of 2). Reducing this will decrease memory footprint, but can negatively impact performance. (default 16384)
   -blocks-storage.filesystem.dir string
     	Local filesystem storage directory. (default "blocks")
   -blocks-storage.gcs.bucket-name string
@@ -769,10 +753,6 @@ Usage of ./cmd/mimir/mimir:
     	How frequently to clean up clients for ingesters that have gone away. (default 15s)
   -distributor.drop-label string
     	This flag can be used to specify label names that to drop during sample ingestion within the distributor and can be repeated in order to drop multiple labels.
-  -distributor.ephemeral-series-enabled
-    	[experimental] Enable marking series as ephemeral based on the given matchers in the runtime config.
-  -distributor.ephemeral-series-matchers value
-    	Lists of series matchers prefixed by the source. The source must be one of any, api, rule. If an incoming sample matches at least one of the matchers with its source it gets marked as ephemeral. The format of the value looks like: api:{namespace="dev"};rule:{host="server1",namespace="prod"}
   -distributor.forwarding.enabled
     	[experimental] Enables the feature to forward certain metrics in remote_write requests, depending on defined rules.
   -distributor.forwarding.grpc-client.backoff-max-period duration
@@ -1025,8 +1005,6 @@ Usage of ./cmd/mimir/mimir:
     	Override the expected name on the server certificate.
   -ingester.ignore-series-limit-for-metric-names string
     	Comma-separated list of metric names, for which the -ingester.max-global-series-per-metric limit will be ignored. Does not affect the -ingester.max-global-series-per-user limit.
-  -ingester.instance-limits.max-ephemeral-series int
-    	[experimental] Max ephemeral series that this ingester can hold (across all tenants). Requests to create additional ephemeral series will be rejected. 0 = unlimited.
   -ingester.instance-limits.max-inflight-push-requests int
     	Max inflight push requests that this ingester can handle (across all tenants). Additional requests will be rejected. 0 = unlimited. (default 30000)
   -ingester.instance-limits.max-ingestion-rate float
@@ -1035,8 +1013,6 @@ Usage of ./cmd/mimir/mimir:
     	Max series that this ingester can hold (across all tenants). Requests to create additional series will be rejected. 0 = unlimited.
   -ingester.instance-limits.max-tenants int
     	Max tenants that this ingester can hold. Requests from additional tenants will be rejected. 0 = unlimited.
-  -ingester.max-ephemeral-series-per-user int
-    	[experimental] The maximum number of in-memory ephemeral series per tenant, across the cluster before replication. 0 to disable ephemeral storage.
   -ingester.max-global-exemplars-per-user int
     	[experimental] The maximum number of exemplars in memory, across the cluster. 0 to disable exemplars ingestion.
   -ingester.max-global-metadata-per-metric int

cmd/mimir/help.txt.tmpl

@@ -287,8 +287,6 @@ Usage of ./cmd/mimir/mimir:
     	Expands ${var} or $var in config according to the values of the environment variables.
   -config.file value
     	Configuration file to load.
-  -distributor.ephemeral-series-matchers value
-    	Lists of series matchers prefixed by the source. The source must be one of any, api, rule. If an incoming sample matches at least one of the matchers with its source it gets marked as ephemeral. The format of the value looks like: api:{namespace="dev"};rule:{host="server1",namespace="prod"}
   -distributor.ha-tracker.cluster string
     	Prometheus label to look for in samples to identify a Prometheus HA cluster. (default "cluster")
   -distributor.ha-tracker.consul.hostname string

docs/sources/mimir/operators-guide/configure/about-versioning.md

@@ -55,9 +55,6 @@ The following features are currently experimental:
 - Distributor
   - Metrics relabeling
   - OTLP ingestion path
-  - Marking of series for ephemeral storage
-    - `-distributor.ephemeral-series-enabled`
-    - `-distributor.ephemeral-series-matchers`
 - Hash ring
   - Disabling ring heartbeat timeouts
     - `-distributor.ring.heartbeat-timeout=0`
@@ -85,11 +82,6 @@ The following features are currently experimental:
     - `-blocks-storage.tsdb.head-postings-for-matchers-cache-ttl`
     - `-blocks-storage.tsdb.head-postings-for-matchers-cache-size`
     - `-blocks-storage.tsdb.head-postings-for-matchers-cache-force`
-  - Support for ephemeral storage:
-    - `-ingester.max-ephemeral-series-per-user`
-    - `-ingester.instance-limits.max-ephemeral-series`
-    - Use of `__mimir_storage__` label matcher.
-    - All `-blocks-storage.ephemeral-tsdb.*` options.
 - Query-frontend
   - `-query-frontend.querier-forget-delay`
   - Instant query splitting (`-query-frontend.split-instant-queries-by-interval`)