bug: Unable to get the latest upstream node list by service discovery

[haifeng9414]

Issue description

I use consul as the registry center and configuration center of apisix and I implemented a consul service discovery through consul-template. After an application is redeployed, the log shows that the consul service discovery has obtained the latest application IP list, but accessing the application through apisix returns 502, the log shows that apisix uses the old application IP.

I think it is caused by the cache of server_picker:

-- apisix/balancer.lua
local server_picker = lrucache_server_picker(key, version,
                            create_server_picker, up_conf, checker)

When the upstream checker is not enabled, the value of version is equal to route.modifiedIndex .. "&" .. service.modifiedIndex, the change of the application ip address in the registry center will not cause this value to change, this will cause the application to be inaccessible until the server_picker cache is invalidated.

I use consul as service discovery instead of eureka, but I believe that using eureka will also have this problem. I think the version should be obtained when obtaining the service ip address:

-- apisix/balancer.lua
if up_conf.service_name then
    if not discovery then
        return nil, "discovery is uninitialized"
    end
    up_conf.nodes = discovery.nodes(up_conf.service_name)
end

change to:

if up_conf.service_name then
    if not discovery then
        return nil, "discovery is uninitialized"
    end
    up_conf.nodes, up_version = discovery.nodes(up_conf.service_name)
end

Environment

• apisix version (cmd: apisix version): lastest master(251625d)
• OS: mac os/linux

Minimal test code / Steps to reproduce the issue

1. Set discovery: eureka and worker_processes: 1, config eureka address in config.yaml.
2. Deploy a multi-instance application and registry to eureka.
3. Confirm that the application can be accessed through apisix.
4. Redeploy the application and access it through apisix again.

What's the actual result? (including assertion message & call stack if applicable)

access application through apisix returns 502

What's the expected result?

no error

[membphis]

@qiujiayu Do you have time to look at this issue?

[qiujiayu]

Adding up_version is a good solution.

[haifeng9414]

Adding up_version is a good solution.

I don’t know how to determine the up_version. Eureka does not have this attribute. Although the data of consul has ModifyIndex, my consul-template solution cannot get ModifyIndex of the data. In order to avoid this problem first, I used the string formed by the service ip list as up_version.

Is there a better solution, or let the implementation of service discovery be responsible for invalidating the server_picker cache?

[membphis]

I used the string formed by the service ip list as up_version.

json?

sync from eureka -> lua table <- server-picker

When data is synchronized, first compare, and only update to Lua table when changing.

then we can use the Lua table as the key of lrucache, example:

apisix/apisix/plugins/referer-restriction.lua

Line 113 in 5b97223

local matcher = lrucache(conf.whitelist, nil,

[haifeng9414]

Both the configuration and the ip address of the upstream should affect the cache of healthchecker and server-picker. If use table as the cache version, how to know the configuration of the upstream has changed?

I think we can maintain the ip list for each upstream in the service discovery, and use a number to indicate the version of the upstream. When the upstream ip changes, the number will increase, and it will be used as part of the current cache version , this will not make the cache version too long