Improve handling of readonly filesystems

[DaveCTurner]

Today we do not allow a node to start if its filesystem is readonly, but it is possible for a filesystem to become readonly while the node is running. We don't currently have any infrastructure in place to make sure that Elasticsearch behaves well if this happens. A node that cannot write to disk may be poisonous to the rest of the cluster:

• a readonly master-eligible node repeatedly leaves and rejoins the cluster, and may trigger elections by offering a vote that it cannot give
• a readonly data node may be assigned shards (e.g. due to rebalancing) which will then immediately fail (cf Prevent allocating shards to broken nodes #18417). Any shards that are already assigned to the node may also eventually fail too (e.g. when syncing retention leases triggers some IO).

This issue is to improve Elasticsearch's behaviour when a node becomes readonly:

• verify that the data paths are writeable before joining a cluster
• verify that the data paths are writeable before offering a pre-vote
• periodically verify that data paths are writeable, and leave the cluster if they are not

[elasticmachine]

Pinging @elastic/es-distributed

[Bukhtawar]

@DaveCTurner thanks for taking this up. Is this fix already prioritized? Do you think I can pick any of the task since I have some context on the issue based on our discussions

[DaveCTurner]

This work is not yet on our roadmap, but if you have ideas for how to proceed then it'd be good to see them. Note that I'll be unavailable for the next few weeks so don't expect prompt feedback from me.

[pugnascotia]

#25591 feels relevant here - it covers documenting what is a cluster's behaviour when a filesystem crashes but the node(s) remains operational.

[DaveCTurner]

Indeed, solving this issue will also resolve #25591 since here we are aiming to exclude the cases where the filesystem is unusable but the node remains in the cluster.

[DaveCTurner]

For completeness, "readonly" also includes cases such as Disk quota exceeded.

[Bukhtawar]

@DaveCTurner, Just a heads up I would be raising a PR for the issue, this week hopefully.
One clarification though, shouldn't master eligible readonly nodes also be blocked from starting a pre-voting round

[DaveCTurner]

shouldn't master eligible readonly nodes also be blocked from starting a pre-voting round

Sounds reasonable, yes. Or else we block sending pre-votes (mentioned in the OP) and require a pre-vote from the local node before PreVotingRound#handlePreVoteResponse starts the election.

[DaveCTurner]

Actually I think I prefer the latter idea: require a pre-vote from the local node. A side-effect of receiving a pre-voting request is to call Coordinator#updateMaxTermSeen which is best to call as early as possible. If we delayed that until a node stopped being read-only then it could trigger another election in an otherwise healthy cluster, which would be a bit weird.

---

Edit: I'm unsure again; I see advantages on both sides. I think it's a minor point and hopefully easily adjusted later, so let's not dwell on it.