Error: inserting target into state: inserting target ip:port for upstream xxx: entity already exists

[michaelhsk]

Kong Version 2.1.2
decK Version 2.2

When using upstreams with targets in my kong configuration deck fails with this error when using --select-tag in command line or select_tags in yaml file.
With routes, services, plugins and upstreams without target there is no problem. Only if i add targets to the upstreams.
If i don't use tags everything works fine, also with targets.

First time the configuration will be synct to kong without any problem. But after that every dump, diff or sync shows this error.

Is this a known bug?

[hbagdi]

Can you please share the yaml file so that we can reproduce this error?

[michaelhsk]

ok, added 2 sample files, with these 2 yaml files i could reproduce the problem on a 2nd environment:

upstream.zip

added v-proxy.yaml to ./master_config and sync:
./deck sync --select-tag deck-managed -s ./master_config
creating upstream webhost
creating target 10.10.10.12:443 for upstream ff8dedff-a544-42c7-abd2-6204f5f3eb38
creating target 10.10.10.11:443 for upstream ff8dedff-a544-42c7-abd2-6204f5f3eb38
creating service v-proxy
creating route v-routes
Summary:
  Created: 5
  Updated: 0
  Deleted: 0

added w-proxy.yaml also to ./master_config and sync again:
./deck sync --select-tag deck-managed -s ./master_config
creating upstream webhost2
creating target 10.10.11.12:8443 for upstream 157eb1aa-efa7-4331-a32c-8d6d9cd59f44
creating target 10.10.11.11:8443 for upstream 157eb1aa-efa7-4331-a32c-8d6d9cd59f44
creating service w-proxy
creating route w-routes
Summary:
  Created: 5
  Updated: 0
  Deleted: 0

after that:
./deck diff --select-tag deck-managed -s ./master_config
Error: inserting target into state: inserting target 10.10.11.11:8443 for upstream 157eb1aa-efa7-4331-a32c-8d6d9cd59f44: entity 
already exists

./deck dump --select-tag deck-managed
Error: building state: inserting target into state: inserting target 10.10.11.11:8443 for upstream 157eb1aa-efa7-4331-a32c-8d6d9cd59f44: entity already exists

Don't know if select-tag is needed in command because it is still in config
_info:
select_tags:

• deck-managed

but this makes no difference.

[hbagdi]

This is potentially a bug in decK and needs further investigation. We will look into it and report back soon.

[michaelhsk]

There is nothing new about the bug?

[hbagdi]

@mflendrich Let's get this re-prioritized for investigation.

[michaelhsk]

tested with deck 1.5.1 where #280 should be fixed.
Error already exists.

Seems that #280 don't fix this problem.

[hbagdi]

@michaelhsk I tried to reproduce this based on the files in #233 (comment) but I couldn't.
Can you please provide a new set of reproduction steps? Or if I'm missing something, please let me know.

[michaelhsk]

Meanwhile we are using Kong 2.2.1, deck 1.5.1, same problem on 5 similar environments. All use cassandra as database. Maybe that this makes the difference?

[hbagdi]

Ah, C*. What would help us is if you could use --verbose 2 and record an entire sync/diff event in an environment where this is happening. That would tell us if Kong's API is violating a contract which is causing this failure in decK. That seems unlikely but worth exploring because I don't have more ideas of what's wrong in decK.

[michaelhsk]

Hi, tried it also on a testinstallation with postgres DB, the problem does not seem to arise there.
Again similar installation with cassandra db. the problem occurs again.

Attached you find the output with --verbose 2.
deck_output_verbose2.zip

[mmorel-35]

@hbagdi , wasn't there a kind of flag to generate a throttling for Cassandra? Could it help?

[hbagdi]

@mmorel-35 Good idea. @michaelhsk could you try this again with --db-update-propagation-delay and see if it changes the situation?

[michaelhsk]

tried, unfortunately, does not solve the problem.

[michaelhsk]

Seems that the error has to do with the tags table.
I deleted the rows for the 4 targets in the tags table, after that the following occured:

diff shoes no error, only that 4 targets will be created.
Summary:
Created: 4
...

1st sync: no error, 4 targets will be deleted, 4 targets will be created.
Summary:
Created: 4
Deleted: 4

In cassandra DB i can see that only the delete was done.

2nd sync: no error, 4 targets will be created.
Summary:
Created: 4

After the 2nd sync the targets are really created, and i can see them in the targets table and the rows to the tags table where also addded.

So all entries are completely available again in database.

And now next sync shows the error again.

[hbagdi]

Can you remove tags from your configuration and verify if the error occurs or not?

[michaelhsk]

See my first post ;-).

"If i don't use tags everything works fine, also with targets."

So if i don't use tags as parameter and at _info section in yaml everything works fine.

[hbagdi]

With that finding (I'm sorry for overlooking that), @fffonion probably has some ideas as to why this happens.

[michaelhsk]

Meanwhile we have migrated our test environments to Kong 2.4.1 and it looks that the problem no longer occurs. It seems that it is fixed in kong and not really a deck problem.

[hbagdi]

It seems that it is fixed in kong and not really a deck problem.

Yes, this is a problem in Kong and not decK because it violates the API contract (hence the ping to @fffonion earlier).

Going to close this, please report back if the problem persists and we will re-open.