host 127.0.0.1:9000: rename /usr/bin/minio /usr/bin/.minio.old: operation not permitted

[pschichtel]

The chain of operator upgrade issues continues after #2305. This time:

main-controller.go:1570] error syncing 'minio/tenant': host 127.0.0.1:9000: rename /usr/bin/minio /usr/bin/.minio.old: operation not permitted

When updating the image in the tenant resource.

The upgrade is from RELEASE.2024-08-29T01-40-52Z to RELEASE.2024-09-13T20-26-02Z.

The rest of the setup is identical to #2305.

[pschichtel]

Manually editing the STS "works around" the issue as last time.

[pschichtel]

I'm wondering: is my setup somehow special or is the upgrade process via operator just not being used much?

[harshavardhana]

It looks like your set-up doesn't allow renames() there is nothing much we can do here. It looks like the image is treated as immutable in your environment.

[harshavardhana]

We have tests already for this that work in regular k8s deployment. We don't ever come across any other non k8s deployments so we never really test it.

If you can reproduce on k8s or openshift we can take a look at how we can handle these cases.

Any other vendor here is out of scope for us.

[pschichtel]

There doesn't seem anything special about the container. If I exec into it as root I can perform the rename without issues. So no syscall restrictions, no read-only FS or anything else. This cluster doesn't have any security enforcement or so. I can confirm the permissions on /usr/bin are 1777 and as uid 1000 I cannot rename the file.

Since I don't often encounter the sticky bit I looked it up again:

For directories, when a directory's sticky bit is set, the filesystem treats the files in such directories in a special way so only the file's owner, the directory's owner, or root user can rename or delete the file. Without the sticky bit set, any user with write and execute permissions for the directory can rename or delete contained files, regardless of the file's owner. Typically this is set on the /tmp directory to prevent ordinary users from deleting or moving other users' files.

/usr/bin and all files in it are owned by root, so it makes sense that uid 1000 wouldn't be allowed to perform the operation.

The pods as setup by the operator have on all containers:

securityContext:
  runAsGroup: 1000
  runAsNonRoot: true
  runAsUser: 1000

So when running the container via docker with uid 1000 as k8s was instructed to, I get the exact same behavior:

$ docker run --rm -it -u 1000 --entrypoint bash quay.io/minio/minio:RELEASE.2024-09-13T20-26-02Z
bash-5.1$ touch /usr/bin/test 
touch: cannot touch '/usr/bin/test': Permission denied
bash-5.1$ mv /usr/bin/{minio,.old.minio}
mv: cannot move '/usr/bin/minio' to '/usr/bin/.old.minio': Operation not permitted

[harshavardhana]

Interesting, let me see if I can make that path fully writable for all users.

[pschichtel]

@harshavardhana did you also republish the latest images?

[harshavardhana]

@harshavardhana did you also republish the latest images?

I did @pschichtel

[pschichtel]

I have this image in production now, let's see how the next upgrade goes :)