fix(manager) lock client creation

[rainest]

What this PR does / why we need it:
Add a mutex and lock it when creating Kong admin clients. Attempting to
create multiple workspaced clients simultaneously can result in a race
condition where both clients check for the workspace before either
creates it, both clients will attempt to create the workspace, and the
second client to attempt creation will hit a unique violation.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes CI errors like:

panic: creating workspace: HTTP status 409 (message: "UNIQUE violation detected on '{name=\"notdefault\"}'")

Special notes for your reviewer:

PR Readiness Checklist:

Complete these before marking the PR as ready to review:

• the CHANGELOG.md release notes have been updated to reflect any significant (and particularly user-facing) changes introduced by this PR

[shaneutt]

This patch does seem to fix the immediate issue, however having the calling code being responsible for managing the locks for the underlying race condition makes maintenance and future contributions a bit more difficult. The burden should be on the implementation of the functionality, I suggest:

• move the locking mechanism into the GetKongClient() method (OR the lowest level needed to avoid the collision)
• make the mutex a part of the Config rather than a glob if the lock is moved into a Config method
• whatever method or function the lock ends up moving to add godoc documentation to it to help provide some additional context

[mflendrich]

We can get away without locks altogether if we follow the dependency inversion pattern and instantiate the client only once, and pass it down to the callers (e.g. as a function argument) for reuse.

[shaneutt]

We can get away without locks altogether if we follow the dependency inversion pattern and instantiate the client only once, and pass it down to the callers (e.g. as a function argument) for reuse.

Having a method called GetKongClient() which by name would appear to get you an HTTP client for Kong not be thread-safe is not particularly axiomatic. Since the implementation has this underlying race condition I still think it's best that we add the lock to it, as close to the actual issue as we can, as we should be ensuring thread-safety at the boundary of our APIs in a way that's opaque to callers. That said, I'm not against also sharing a single client to reduce waste.

[rainest]

Went with the lock within the function. We don't currently create meaningfully different clients, but potentially could if we start interacting with separate Kong instances and/or workspaces from a single controller, which we've discussed in the past.

[shaneutt]

One suggestion and then LGTM