Disable fsspec Cache on Integration Tests

[ranchodeluxe]

Addresses #645

I can't find the deeper bug here to ticket with zarr or fsspec but potentially getting close

[ranchodeluxe]

Super fascinating bug. I don't think this is actually minio in the end (for reasons I will explain later). Wondering if this is something with s3fs 🤔

[norlandrhagen]

Oof, well thanks for digging into it!

[ranchodeluxe]

Oof, well thanks for digging into it!

Breakdown

Here's some evidence for it not being just minio:

1. I remove any dynamic paths here and hardcode a reusable one using pathlib.Path('/tmp')

2. I comment out spinning up a pytest minio instnace and hardcode the secrets for my minio container running on my host (see next step) here

3. I then run a dedicated minio container on my host:

docker run \
   -p 9000:9000 
   --name minio666 \
   -e "MINIO_ACCESS_KEY=YOURACCESSKEY" \
   -e "MINIO_SECRET_KEY=YOURSECRETKEY" \
   -v /tmp/:/data \                                  
   minio/minio server /data 

4. I can ssh into that container and trace requests using:

mc alias set 'myminio' 'http://localhost:9000' 'YOURACCESSKEY' 'YOURSECRETKEY'
mc admin trace -v -a myminio

6. Since the problem is intermittent I wait for a successful run to happen and check outputs on minio using mc ls <target>. If everything looks good I know the next runs should ALWAYS succeed b/c the lat key (for the noaa-oisst test specifically) exists already and we are reusing the hardcoded target from step 1 above for the next run

7. More runs either succeed and fail intermittently even with this set up (pytest -s tests/test_integration.py -k 'test_integration[minio_confpath-noaa-oisst]' --run-integration --timeout=500)

8. There is nothing in the verbose mc trace logging that shows bad PUT(s) or 500(s) or anything

9. My hunch is if this was just minio we wouldn't be seeing the tests failing constantly for exactly the same key (which is lat for noaa-oisst test) but different keys across runs. I'm not seeing that. For the other test (gpcp) it's failing for the a different key constantly (time)

Proposals

• Is there a reason these tests can't use an actually dedicated s3 or google storage bucket? Tests run fine against them

• I imagine we don't want to use fsspec.implementations.local.LocalFileSystem for the tests b/c it's not really flexing what this project is about?

• If we truly think it's just minio a possible work around here that will take more time is to do what pangeo-forge-runner does and set up a k3's instance and use minio there. But if it's not just minio it's all for nothing. Thoughts? 🤔

[ranchodeluxe]

Proposals

Is there a reason these tests can't use an actually dedicated s3 or google storage bucket? Tests run fine against them

I imagine we don't want to use fsspec.implementations.local.LocalFileSystem for the tests b/c it's not really flexing what this project is about?

If we truly think it's just minio a possible work around here that will take more time is to do what pangeo-forge-runner does and set up a k3's instance and use minio there. But if it's not just minio it's all for nothing. Thoughts? 🤔

well, if I run minio as a service in k8s locally everything works fine so 🤷 looks like that's the way forward. won't get to that today though

[ranchodeluxe]

Wow, humbled. Things just got more interesting with these failures after the last commits

Cannot get it to fail locally with a similar set up

Looks like we cannot ignore the subtle interaction that is causing this and have to weed it out

[ranchodeluxe]

It was s3fs the whole time!!!!!!! I had to disable cache settings and now everything works 😞 Will swap things back to the docker image and make sure that solves it there too

[norlandrhagen]

LGTM!