fix: don't release healthchecker during etcd sync due to compaction

[shreemaan-abhishek]

Description

Root cause of the bug:

When APISIX tries to watch a revision that has been compacted by etcd compaction. This case is difficult to reproduce manually, so adding tests for this would be very unstable.

To reproduce this manually:

• Create an upstream where one node is available and another is not:

curl http://127.0.0.1:9180/apisix/admin/upstreams/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
  {
    "nodes": {
      "127.0.0.1:1980": 1,
      "127.0.0.1:1111": 1
    },
    "retries": 0,
    "name": "测试心跳",
    "type": "roundrobin",
    "timeout": {
      "connect": 60,
      "read": 60,
      "send": 60
    },
    "pass_host": "pass",
    "scheme": "http",
    "keepalive_pool": {
      "idle_timeout": 60,
      "requests": 1000,
      "size": 320
    },
    "checks": {
      "active": {
        "concurrency": 10,
        "healthy": {
          "http_statuses": [
            200,
            302
          ],
          "interval": 1,
          "successes": 2
        },
        "http_path": "/firstwork/healthcheck",
        "timeout": 5,
        "type": "http",
        "unhealthy": {
          "http_failures": 5,
          "http_statuses": [
            429,
            404,
            500,
            501,
            502,
            503,
            504,
            505
          ],
          "interval": 10,
          "tcp_failures": 2,
          "timeouts": 2
        }
      }
    }
  }'

• Create a route that references the above upstream:

curl "http://127.0.0.1:9180/apisix/admin/routes/1" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '
{
  "uri": "/*",
  "upstream_id": "1",
  "name": "测试心跳问题",
  "methods": [
    "GET",
    "HEAD",
    "POST",
    "PUT",
    "DELETE",
    "OPTIONS",
    "PATCH",
    "CONNECT",
    "TRACE"
  ],
  "priority": 0,
  "enable_websocket": false,
  "status": 1
}'

• send a couple of requests so that healthchecker kicks out the unhealthy node:

curl -i "http://127.0.0.1:9080/hello"

• use the following script that increases the latest revision number and performs compaction before apisix is able to watch the latest revision:

rev=$(etcdctl get --order=DESCEND --sort-by=MODIFY --limit=1 --prefix "" -w=json | jq '.kvs[0].mod_revision + 1')
ectl put key val; ectl compact $rev

• wait for the logs that say err: compacted, will read the configuration again via readdir
• after this the healthchecker will be released and unhealthy nodes will be added back. You can confirm this by sending requests to the route created previously some of them will result in 502.

Fixes #10500

Checklist

• I have explained the need for this PR and the problem it solves
• I have explained the changes or the new features added to this PR
• I have added tests corresponding to this change
• I have updated the documentation to reflect this change
• I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

[membphis]

This is a wrong solution and will cause the checker to never be released

[shreemaan-abhishek]

could you clarify more? My changes will not release the checker only when the reason is compaction. There is still another case that can release the checkers properly:

https://github.com/shreemaan-abhishek/apisix/blob/105993f40f34768cd0a86035d01393346fff3a94/apisix/core/config_etcd.lua#L686

[shreemaan-abhishek]

closing in favour of: #10500 (comment)