[Agones] Removing/updating token annotation on a GameServer deoesn't update the Endpoint

[markmandel]

What happened:

I have a Quilkin Agones xDS server, and connected Quilkin proxies.

1. I created and allocated GameServer in Agones with the quilkin.dev/tokens: annotation.
2. I then kubectl edit that GameServer and removed the quilkin.dev/tokens: annotation.
3. I see two endpoints now in the config for my proxy, not a single one with the token removed.

➜  agones-xonotic-xds git:(examples/xonotic-xds) ✗ curl -s http://localhost:9091/config | jq
{
  "admin": {
    "address": "0.0.0.0:9091"
  },
  "clusters": {
    "default": {
      "localities": [
        {
          "locality": null,
          "endpoints": []
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7251",
              "metadata": {
                "quilkin.dev": {
                  "tokens": [
                    "NDU2"
                  ]
                }
              }
            }
          ]
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7251",
              "metadata": {
                "quilkin.dev": {
                  "tokens": []
                }
              }
            }
          ]
        }
      ]
    }
  },
  "filters": [
    {
      "name": "quilkin.filters.capture.v1alpha1.Capture",
      "config": {
        "metadataKey": "quilkin.dev/capture",
        "suffix": {
          "size": 3,
          "remove": true
        }
      }
    },
    {
      "name": "quilkin.filters.token_router.v1alpha1.TokenRouter",
      "config": null
    }
  ],
  "management_servers": [
    {
      "address": "http://quilkin-manage-agones:80"
    }
  ],
  "proxy": null,
  "version": "v1alpha1"
}

What you expected to happen:

The token removed from the endpoint, so an end user could remove a playing user based on their connection token in realtime.

How to reproduce it (as minimally and precisely as possible):

See above.

Anything else we need to know?:

Not sure if duplicated to #585 so reporting separately.

Environment:

• Quilkin version: dev.
• Execution environment (binary, container, etc): Container
• Operating system: Linux
• Custom filters? (Yes/No - if so, what do they do?): No
• Log(s): N/A
• Others: N/A

[markmandel]

Just did a check on the admin server as well, and see the same config

{
  "admin": {
    "address": "0.0.0.0:9091"
  },
  "clusters": {
    "default": {
      "localities": [
        {
          "locality": null,
          "endpoints": []
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7251",
              "metadata": {
                "quilkin.dev": {
                  "tokens": [
                    "NDU2"
                  ]
                }
              }
            }
          ]
        },
        {
          "locality": null,
          "endpoints": [
            {
              "address": "34.127.72.131:7251",
              "metadata": {
                "quilkin.dev": {
                  "tokens": []
                }
              }
            }
          ]
        }
      ]
    }
  },
  "filters": [
    {
      "name": "quilkin.filters.capture.v1alpha1.Capture",
      "config": {
        "metadataKey": "quilkin.dev/capture",
        "suffix": {
          "size": 3,
          "remove": true
        }
      }
    },
    {
      "name": "quilkin.filters.token_router.v1alpha1.TokenRouter",
      "config": null
    }
  ],
  "management_servers": null,
  "proxy": null,
  "version": "v1alpha1"
}

[markmandel]

Just did a local manual test - can confirm the token gets dropped.

Will write an extra piece to the Agones integration tests to automate testing of this, and we can close this issue.