Operator canot talk to tenant because of: The Access Key Id you provided does not exist in our records

[sebhoss]

Expected Behavior

The operator should be able to talk to its tenants using the secret provides in the spec.users array of a tenant.

Current Behavior

The operator logs the following error and fails to connect:

I0917 12:33:34.584792       1 event.go:377] Event(v1.ObjectReference{Kind:"Tenant", Namespace:"minio-tenant-loki", Name:"loki", UID:"f15c42e3-701a-4a48-95bb-fa753a346aca", APIVersion:"minio.min.io/v2", ResourceVersion:"2096771256", FieldPath:""}): type: 'Warning' reason: 'BucketsCreatedFailed' Buckets creation failed: The Access Key Id you provided does not exist in our records.

At first, I thought this was another case of minio/minio#20120 but the access key does seem to exist and loki (the service using the minio storage) is able to connect just fine using the very same credentials that the operator is using so I'm sure that the credentials itself are fine. The file system inside the minio pods seems to agree:

$ ls -la /export0/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export1/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export2/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

$ ls -la /export3/data/.minio.sys/config/iam/users/
total 12
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 .
drwxr-sr-x 5 1000 1000 4096 Jun 26 07:34 ..
drwxr-sr-x 3 1000 1000 4096 Jun 26 07:34 <redacted>

The value is the same in each location and each pod. It is exactly the value of the CONSOLE_ACCESS_KEY key inside the storage-user secret that is referenced in the minio tenant. The tenant is not completely new but has been around for a while and the operator was previously able to connect to it and provision it correctly.

Possible Solution

N/A

Steps to Reproduce (for bugs)

1. Take secret from https://github.com/minio/operator/blob/master/examples/kustomization/base/storage-user.yaml
2. Deploy tenant that references that secret, e.g. https://github.com/minio/operator/blob/master/examples/kustomization/base/tenant.yaml
3. ???
4. See operator logs

Context

We use the operator to provision minio tenants. We only just updated the operator to v6 in this environment and were running v5 of the operator previously. I cannot delete and re-create this tenant because it contains production data which we do not want to lose.

Regression

Yes - I'm sure that older versions of the operator were able to connect just fine.

Your Environment

• Version used (minio-operator): v6.0.3
• Environment name and version (e.g. kubernetes v1.17.2): kubernetes v1.29.7
• Server type and version: N/A
• Operating System and version (uname -a): N/A
• Link to your deployment file: We use the official helm chart to deploy the operator

[sebhoss]

I just restarted the operator and now get what I previously described in #2238 ...