Fix "Error: could not execute revoke query: pq: tuple concurrently updated"

[boekkooi-lengoo]

Hello and Happy Holidays (or I hope they where),

Yesterday I was working on automating some DB permissions and kept getting a "pq: tuple concurrently updated" error, so today I decided to remove this annoyance (aka it worked after running terraform multiple times but that's just annoying).

I have added an example of the issue in f9879a4 which (for me) consistently shows the issue.
The issues seems to be that updating the same "tuple" (in this case doing a GRANT or REVOKE in parallel) causing postgres to error. For a reference to the issue see https://www.postgresql.org/message-id/CAHyXU0y3M1Znv=MJ7u1y3pSMutALcoryQx9-v8c-WkaTdvvQ7Q@mail.gmail.com and https://community.pivotal.io/s/article/Query-Fails-with-ERROR--tuple-concurrently-updated?language=en_US.

The "fix" (or maybe hack?) I introduced is using a mutex to ensure that only a single GRANT/REVOKE is executed at the same time.
This seems to resolve the issue but if you have a better idea on how to resolve this I would love to try it.

To run the example I used the following commands.

source ./tests/switch_superuser.sh
unset TF_ACC
docker-compose --project-directory=tests up -d 

cd examples/issues/169/
terraform init
terraform apply -auto-approve

To run the fix I build the provider using go build -o ./examples/issues/xx/postgresql/terraform-provider-postgresql and then ran the following commands inside examples/issues/169/.

export TF_CLI_CONFIG_FILE=$(pwd)/examples/issues/169/dev.tfrc
terraform apply -destroy -auto-approve
terraform apply -auto-approve

Thanks for taking the time to look at this and have a great day!
Cheers,
Warnar

[cyrilgdn]

Hi @boekkooi-lengoo ,

Thanks for opening this PR, I don't have so much time these days but I'll try to look at it in detail quickly.
You'll have to merge origin/master to fix the failing tests, it's unrelated to your changes.

What I can already say is that there's already a lock here: https://github.com/cyrilgdn/terraform-provider-postgresql/blob/master/postgresql/helpers.go#L459, which lock only based on the role name. It indeed doesn't manage properly the public specific case (as the query on pg_roles will return an empty result).

Maybe it can manage the public specific use case by locking a fixed integer (e.g.: SELECT pg_advisory_xact_lock(-1)).

Your fix could work but it can slow down state with many grants resources.

[boekkooi-lengoo]

Hi @cyrilgdn,

Thank you for the tip! It seems that using pgLockRole actually also solves the issue at hand since the tuple concurrently updated is occurring when applying the grant for a role (aka https://github.com/cyrilgdn/terraform-provider-postgresql/pull/169/files#diff-ea9ba14211515201ec9fb3fb69a6f8e75b44db83c6c528f56e083bac7d9dc100R78).

[JamesTimms]

Hi, any chance this can get merged. I've experienced the same issue @boekkooi-lengoo has reported and this sounds like the fix!

[cyrilgdn]

Thanks for the fix @boekkooi-lengoo 👍

[der-eismann]

Do you think you could do a timely release? We are stumbling over this as well.

[cyrilgdn]

@der-eismann It will be released this week yes

[boekkooi-lengoo]

Sadly I just tested out v1.15.0 and managed to hit another tuple error but this one is slightly different.
I'm going to write a test case for it and see if it can be reproduced.

[rgl]

FWIW, this is still happening with the latest 1.17.1 version :-(

[mkuchniak]

Same problem on 1.18.0