Removes a now-bogus assertion, and instead asserts that the assigned
roles correctly influence the engine created for each shard copy.

Today we say `manually call [/_cluster/reroute?retry_failed=true]` if a
shard exceeds the max number of retries. We should give a verb here, and
also since elastic#90399 we should recommend saying `metric=none` too.

…n listeners (elastic#93185)

…ingTasks (elastic#93174)

This should help debugging elastic#92910 and rule out possible environmental
issues, such us slowness creating threads due to an overloaded CI
worker.

If this test runs right before midnight, it's possible when getting
the settings for the index, it doesn't exist, due to a mismatch between
the names (datastream names have the date in their name, "uuuu.MM.dd")

Closes elastic#92368

Alternative approach to elastic#93143, avoiding adding overhead in production.

The vast majority of this commit is doing a simple s/Version/TransportVersion/, however there are a couple of areas of complexity.

Handshakes proved to be a particularly tricky issue, as they have complex usage of Version.minimumCompatibilityVersion. In lieu of solving this problem now, I've replaced uses of minimumCompatibilityVersion with TransportVersion.minimumCompatibilityVersion and TransportVersion.calculateMinimumCompatVersion.

As a later piece of work, code using both those methods will need to be refactored to do something else, and the methods removed.

Co-authored-by: Przemyslaw Gomulka <przemyslaw.gomulka@elastic.co>

…93188)

Lucene 9.5 changed the allowed range and the default value for the
allowed percentage of deletes in an index. It used to allow values in
[20, 50] with a default of 33. It now allows values in [5, 50] with a
default of 20.

See apache/lucene#11831.

…ic#93191)

Update tsdb docs to include a warning that the format of the `_tsid` field shouldn't be relied upon and added additional limitations about dimension fields.

`ListenableFuture` throws an `IllegalStateException` if completed more
than once, which (at best) is just silently swallowed and (at worst) may
prevent other cleanup from happening. In particular, we do not expect
`onFailure()` to throw an exception as happens here. This commit
elevates this check to an assertion, and fixes the places that violate
it using `ActionListener#notifyOnce`.

…elastic#93194)

This commit unmutes the test and adds additional logging in order
to debug the test failures in the CI environment.

Relates elastic#91967

This is a workaround that provides support for the 
`IMMEDIATE` and `WAIT_UNTIL` Refresh policies.

Relates ES-5292

…lthMetadata (elastic#93157)

Closes elastic#93155

…c#93195)

This commit adds a mechanism so geohex aggregations with precision <= 3 are created with unbounded bounds.

Bootstrap check failures were unintentionally moved to the info level
in the log output. This commit restores logging them as errors.

closes elastic#93074

The cluster reroute API (optionally) returns the cluster state in its
response, which can therefore be rather large. elastic#92285 enables a chunked
encoding of the cluster state, and this commit adjusts the reroute API
to make use of this encoding too.

* Documentation for geohex_grid over geo_shape

The feature to add support for geohex_grid aggregations over geo_shape
fields was added in elastic#91956.
This is the associated documentation for that.

* Update docs/reference/aggregations/bucket/geohexgrid-aggregation.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Fix explanation for geo_point vs geo_shape proj

When aggregating geohex over geoshape we use requirectangular because
underlying lucene index indexes and searches the polygons in that way.

* Correct spelling

According to grammarly, "therefor" is not an alternative spelling
of "therefore". We should use the conjunctive form here.

See https://www.grammarly.com/blog/therefore-vs-therefor/

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Alternative algorithm for inflated bounds

The BoundedGeoHexGridTiler inflates the bounding box for parent
cell searches since the parent cells bounds do not match the
descendents.

This algorithm does a slightly more refined calculation of the
inflation factor by considering all four corners and inflating in
the four directions independently.

* Remove alternative algorithm for bbox inflation

The new algorithm was faster than the old for the same inflation factor
but had more failures, and a longer tail of failures (which needed
higher inflation factors). So it would take a lot more work (probably)
to find and fix these edge cases. An easier option is to keep the
original algorithm.

* Reduce bbox inflation factor to 0.25

This increases performance by about 10% (a factor of 0.22 increased
performance by about 13%).

We found the error rate for `testGeoShapeGeoHex` was about 0.5% for
inflation factor 0.19 and 0.2% for inflation factor 0.22.
These last 2/1000 failed tests required inflation factors of 0.27
0.30 respectively, so we'll investigate them individually.

* Increase bbox inflation factor to 0.35

Since there were three failures discovered for 0.25, all of which passed
at 0.30, we decided to set the inflation factor to 0.35 initially for
the sake of caution, and consider revising it down again later with
more careful testing.

* Fixed comment about bounded tiler inflation

* Fixed comment about bounded tiler inflation

We introduced a bug in elastic#79885 by not accounting for cases where there's
a nonzero offset in the underlying byte array. Unfortunately
`ESIndexInputTestCase` didn't cover these code paths. This commit
extends the test coverage and fixes the bug.

We introduced the ability to resolve the cluster alias in elastic#91724, given a
connection towards a remote cluster. This works for connections
obtained via RemoteConnectionManager::getConnection. However,
connections can also be initiated via openConnection (e.g., in
SniffConnectionStrategy). This PR adds support to correctly resolve the
cluster alias for such connections, by wrapping them in an
InternalRemoteConnection instance. This is necessary to support the new
remote cluster security model.

…93209)

When numDataNodes is 0 the test framework doesn't create nodes
automatically, for this test we needed to manage the node roles
manually in order to exercise different topology scenarios, by
a misunderstanding autoManageMasterNodes was set to false, while
that was not necessary. This commit sets autoManageMasterNodes to
its default value simplifying slightly the test suite.

Previously UserExceptions could be thrown from the main Elasticsearch
process during startup. With recent refactorings all remaining
validation of arguments was moved to the server cli. Since there are no
longer any uses of the user exception in server, the marker code is no
longer needed. This commit removes that marker and the associated code
in the server cli error pump.

This commit adds a listener that will be called when new operations are written to the translog writer.

…c#93211)

The systemd unit file is part of the Elasticsearch package and should
not be edited. Instead, we recommend creating a service override file.
This commit tweaks the docs for setting tmp dir with systemd to use the
override file instead of editing the unit file.

relates elastic#93121

…c#93213)

…lastic#93181)

Similar to existing dedicated IP filter settings for transport and http,
the IP filter settings for remote access are now prefixed with
xpack.security.

Relates: elastic#93126

…lastic#93179)

This PR makes JsonProcessor's JSON parsing a little bit stricter so that
we are not silently dropping data when given bad inputs. Previously if
the input string began with something that could be parsed as a valid
json field, then the processor would grab that and ignore the rest. For
example, `123 "foo"` would be parsed as `123`, dropping the `"foo"`. Now
by default it will throw an IllegalArgumentException on a string like
this. A user can now set the `strict_json_parsing` parameter to false to
get the old behavior. For example:

```
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "description": "",
    "processors" : [
      {
        "json" : {
          "field" : "message",
          "strict_json_parsing": false
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "message": "123 \"foo\""
      }
    }
  ]
}'
```

Closes elastic#92898

…tic#93212)

The `SSLConfigurationReloader` uses a `CompletableFuture` to receive an
as-yet-unconstructed instance of the `SSLService`, but in all cases the
`SSLService` is in fact already constructed so we can just directly pass
it into the constructor.

This PR adds querying-cluster-side support for internal users making
requests cross cluster under the new security model.

For the querying cluster, this involves detecting that a request is
made by an internal user and sending a remote access header with an
empty role descriptors intersection and an embedded authentication
instance identifying the internal user making the request. The
fulfilling cluster will use this header to resolve the correct
privileges for the given internal user (while also enforcing the
privileges of the remote access key used for the request). The
fulfilling cluster side implementation will follow in a separate PR.

Currently, the only use of internal users cross cluster is requests
made by _system to fetch cluster-state from the fulfilling cluster.

Today shard routing assertions are convoluted and even accept some cases.
For example `assert relocatingNodeId == null || state == RELOCATING || state ==
INITIALIZING` permits null `relocatingNodeId` for relocating shard. This change
splits assertion per state in order to simplify and allow stricter invariants.

Similar to `ByteArrayIndexInput`, but for a general `BytesReference` to
avoid having to copy the bytes into a contiguous array.

- There's no real need for the caller's `Logger` just in case an impossible
  situation happens - we will get enough info from the log message.

- Almost nobody uses the `forceExecution` feature, default it to
  `false`.

- Accept an `ExecutorService` rather than requiring the whole
  `ThreadPool` only to look up the executor later on.

…3182)

…lastic#93203)

After elastic#93075 the first settings parameter in ClusterModule#createAllocationDeciders
are expected to be the same settings that ClusterSettings contain,
this was not the case for this test and some of the allocator deciders
used the default config value instead of the one that the test needed.

Closes elastic#93142

This migrates org.elasticsearch.index and org.elasticsearch.indices to use TransportVersion

* enhancement: boolean field to support ignore_malformed

* fix: changes in current builder for BooleanFieldMappers within tests files.

* Updating documentation

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Amy Jonsson <amy.jonsson@elastic.co>

The test uses random vectors, consequently it is possible that vector_1 values are much closer than vector_2 values. Increasing boost to ensure order.

…c#93234)

Some naming adjustments, the ability to set up the cache on a non-frozen
search node and a small API adjustment that makes it so a `ThreadPool`
is not a dependency for using the cache.

…ic#90122)"

This was merged in error without a full CI run, and has some issues.

This reverts commit edcdc43.
This reverts commit 26c0a35.

With this change state persistence gets triggered based on time instead of a counter. This applies only to transforms
during indexing e.g. a batch transform, it doesn't affect other reasons to persist state, e.g. after a checkpoint or a
state change triggered by an API call. The reduction of state persistence triggers transform reduces overhead. State
persistence gets triggered every 60s.

…3076)

InferenceConfig is kept on Version, as that existed before VersionedNamedWriteable came along

…lastic#93076)"

This reverts commit bef85c6.

Today it doesn't work to close an index which holds unpromotable shards.
This commit fixes that.

…ic#93163)

This commit adjusts validation to allow custom password hash
to be provided when executing change user password transport action.

This change relates to elastic#92871, which introduced the original changes
but somehow, the validation on the transport action was missed to be
refactored as well.

When we upgraded to lucene 9.5 (snapshot) with elastic#92957 we initially disable panama-based
mmap directory through a system property. With this commit we remove the system property
and enable java 19 memory segments by default (based on apache/lucene#12033)

This PR moves the postLookup call out of the lookup methods in order to correctly decrement 
the instance counter a single time at the end of the processor call.

…3248)

Add enough documents to the searchable snapshot backing index so
all shards have enough data files to be blocked during recovery.

Closes elastic#91450

…astic#93242)

Re-apply "Migrate VersionedWriteable & NamedDiff to TransportVersion (elastic#93076)"

This reverts commit 48f9609.

…stic#90977)

Instead of synchronously blocking a thread in the management pool, add a listener on removed tasks and calls nodeOperation after all matched tasks have been removed. Also add a scheduled tasks to bail out after the specified wait timeout if the tasks haven't been finished.

Fixes elastic#89564, elastic#90988

Use the new constructor signature for ThreadedActionListener from elastic#93184

Relates elastic#93226

…ion (elastic#93199)

conditional usages of StreamInput/StreamOutput getVersion are refactored to use
TransportVersion instead of Version

The built-in `editor` role allows "all" access to all Kibana
features, but only read access to data indices. This doesn't
work well for functionality that spans Kibana and Elasticsearch
and allows the user to choose their own results index, like
ML data frame analytics.

This change adjusts the notes on the `editor` role to make clear
that in this case an additional role must be granted to give the
necessary access on the data index that the results will be
written to.

… force strategy (elastic#93264)

With the introduction of TransportVersion we want to use this object in transport protocol related code (usages of StreamInput, StreamOutput).
Also with serverless release being more frequent then on-prem the code changes that relied on Node's Version to perform a conditional logic are also refactored to use TransportVersion which has higher granularity

We recently upgraded the antlr version that painless uses to 4.11.1.
Lucene 9.5 is on the same version.
This commit updates ql, eql and sql to the same version

Replaces the usage of the GeoIp processor factory with the DatabaseNodeService when 
getting all databases in the tests.

Adds a comment to `TransportHandshaker` laying out the details of a
transport handshake.

With the merging of apache/lucene#12033 this
testing is no longer necessary as Lucene no longer requires the
--enable-preview flag for enabling the project panama mmap
implementation.

…tic#93293)

Plugins which indirectly implement VersionedNamedWriteable should return
TransportVersion

relates elastic#93242

This PR fixes an issue in authentication rewriting: previously
maybeRewriteRealmRef did not remove domains, even when this was
expected. This has very limited customer impact, since the enclosing
maybeRewriteForOlderVersion method is called in conjunction with
Authentication.encode() in most cases. Authentication.encode() calls
writeTo, which in turn omits domains as necessary, based on a version
check.

It's vital that listeners passed to the `IndexShardOperationPermits` are
not completed more than once, because otherwise the caller might not
release the returned `Releasable` which would leak a permit. It looks
like this is the case today, but one of its callers has some explicit
protection against such a leak. This commit adds some assertions to
demonstrate that this protection is not required.

don't re-create doc values object for every value. This change fixes a
conceptual problem which caused significant overhead.

)

Relates elastic#93202

…lastic#93297)

Ensure that all shards are started before making progress, this
avoids issues when we expect one of the shards to be active but
it isn't.

Closes elastic#91967

…Mapper Size} (elastic#93305)

Relates elastic#93303

Fix type togeher > together

…/10_basic/Mapper Size} (elastic#93315)

Relates elastic#93303

* Add monitoring mappings for es ingest metricset

* Update mappings

* Fix type for type and type_tag

* Fix JSON and update template version

* Update docs/changelog/92950.yaml

elastic#93317)

Point Fleet/Agent ES files to Fleet team instead of Agent Control Plane team.

Co-authored-by: Josh Dover <1813008+joshdover@users.noreply.github.com>

…#93324)

When a datafeed starts up it looks at the job's data counts to decide
where to pick up from where a previous invocation finished. Previously
the datafeed was always getting the data counts from the index. In the
case where a datafeed has been stopped and restarted while its
corresponding job was continually opened this is incorrect. In this case
the data counts need to be obtained from the running job. (In the case
where the job was closed and reopened while the datafeed was stopped
this does not matter, as closing the job will have persisted the
up-to-date data counts.)

This bug has always existed, yet is made much more likely to cause a
noticeable discrepancy by the changes made in elastic#93000.

Fixes elastic#93298

This is a follow up to elastic#93229 to fix one of our example plugin tests.