high disk watermark more than flood stage disk watermark

[missinglink]

saw this in the ES6 logs:

java.lang.IllegalArgumentException: high disk watermark [99%] more than flood stage disk watermark [95%]

[orangejulius]

Ugh ok. I guess this repo will have to handle setting the flood stage disk watermark as well, and instruct people to ensure the two values are set appropriately.

[missinglink]

Are those settings not currently coming from this repo?

[orangejulius]

They are configurable, so any terraform project using this repository can set them.

[missinglink]

I had a look into this and unfortunately, the flood_stage watermark isn't the same as the other watermark settings (high/low) in that it's not dynamically updatable, so presumably it needs to be set in elasticsearch.yml instead of updated via a curl request ☹️

curl -XPUT \
  -H 'Content-Type: application/json' \
  -d '
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.flood_stage": "50%"
  }
}' \
'localhost:9200/_cluster/settings?pretty=true'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage], not dynamically updateable"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "persistent setting [cluster.routing.allocation.disk.watermark.flood_stage], not dynamically updateable"
  },
  "status" : 400
}