State of testing in Pangeo

[alando46]

I'm hoping to understand more about how testing works with Pangeo deployments. After digging around a bit I wasn't able to find any documentation of the exact process, so if this exists and I missed it, please feel free to close this issue. Also, I should mention that I don't have any significant experience with setting up CI/CD so if my apologies in advance if I mistake some concepts.

Is there an overview of the types of tests that are run when a PR is made?

It seems like the following tests may be valuable:

• test that checks that every .yaml file in a helm chart is valid .yaml (found an indentation issue on the latest helm release and am going to submit a pr, but want to make sure there is a test to fix this for future releases)
• test that checks that all dependencies in a dockerfile were successfully installed
• this may be already somewhat covered by https://github.com/pangeo-data/dev.pangeo.io-deploy but it seems like a final test of deploying a cluster, accessing a server and instantiating some dask workers could be useful as well (I believe @scottyhq recently discovered a bug here).
• lastly, maybe a test that checks all helm docker dependencies to make sure they are the latest version.

Just a few ideas. Sorry if this already exists somewhere but just thought I'd raise the issue just in case it doesn't. If these don't yet exist and the community feels they could be valuable, happy to get the ball rolling on getting these tests set up.

[jhamman]

Good points to bring up. Thanks @alando46.

• I think it would be worth while to add a basic minikube deploy test the helm chart repo. This may suffice as a yaml linter/validator or a separate test could be added for this.
• The images in pangeo-stacks will not pass the travis tests or deployment steps if any dependencies are missing. So I think we're fully tested there.
• I'm not sure how you would implement a full integration test where a dask cluster is spun-up from within a jupyterhub but its an interesting idea.
• I'm not sure what you are referring to in your last bullet? Can you expand a bit?

I think we're mostly waiting for someone with some bandwidth to work on these things. If any of these pieces interest you, I'd be happy to work with you to get things integrated.

[alando46]

Thanks for sharing thoughts @jhamman. Please see my responses below

I think it would be worth while to add a basic minikube deploy test the helm chart repo. This may suffice as a yaml linter/validator or a separate test could be added for this.

The minikube deploy seems like a good idea that we could potentially get other test functionality out of.

The images in pangeo-stacks will not pass the travis tests or deployment steps if any dependencies are missing. So I think we're fully tested there.

Ok, thanks for clarifying. There is at least one example (Rasterio) where a package can be successfully installed but not fully function due to dependency issues. Given the importance of GDAL/Rasterio to a lot of geo workflows, would it make sense to brainstorm about an additional layer of functionality testing?

I'm not sure how you would implement a full integration test where a dask cluster is spun-up from within a jupyterhub but its an interesting idea.

I can do some further research on options for this. Tying into the previous comment, it seems like it might be useful to test certain critical functionalities for the most common Pangeo use cases. I'm imagining a series of "feature" tests where a user server is started, a notebook opened, and a series of critical features are tested, maybe including:

• Instantiate dask Cluster and connect to client
• use rasterio to read some data
• process the data in dask cluster
• use xarray to read some netcdf data (not a netcdf person but I believe this is how it's done)
• write something to zarr
• any other critical functionalities

In regards to checking dependencies, again this is coming from somebody without a great sense of testing best practices but it seems like Pangeo has 2 core docker dependencies (right now ZeroToJupyter for notebook and worker images). Would it make sense to configure something to check that the images referenced in the Helm Chart are the latest version of these images?

[jhamman]

@alando46 - I've opened pangeo-data/helm-chart#81 to get the helm chart CI conversation going.

[jhamman]

@alando46 - I would be interested in hearing your ideas for integration testing. I have a few ideas.

1. I can see it being useful to build a basic integration test suite. This could be a standard py.test-like setup with a handful of small but comprehensive tests. You could put this somewhere logical in the docker image and test routinely (as part of docker-stacks or hubploy builds) as:

   docker run -i -t pangeo/base-notebook:latest "py.test"
   

   These wouldn't necessarily use kubernetes but would go a long way toward covering version compatibility issues.
2. You could also connect to a local minikube cluster and run some of the same test suite that uses dask-kubernetes (or other cloud/kubernetes tools).
3. Finally, you could connect to a remote kuberntes cluster (e.g. something running on GCP) and run larger integration tests. You'd want to be careful about when/how these are run just because the cost could add up quickly.

To be clear, none of this exists right now but depending on what you testing, these are all options.

[alando46]

What sorts of integration tests do you see as being best suited to run in a single container?
The tests, or maybe what kubernetes calls "End-to-end" (e2e) testing I'm envisioning might be something along the lines of:

# can connect to dask workers
def test_connect_to_dask():
    from dask_kubernetes import KubeCluster
    from dask.distributed import Client
    cluster = KubeCluster(n_workers=4)
    client = Client(cluster)    
    assert isinstance(client, distributed.client.Client)

# rasterio is talking to dask
def test_open_rasterio():
    # if chunks are present dask array is created
    assert isinstance(xr.open_rasterio('something.tif', chunks=(5,5)), xarray.core.dataarray.DataArray)

# rasterio is working with gdal
def test_open_rasterio():
   # if no chunks are present xarray loads np array into memory (using gdal)
   assert isinstance(xr.open_rasterio('something.tif', xarray.core.dataarray.DataArray)

It seems like one way forward could be to create some sort of default binder configuration which, once a user configures their cluster, could run automatically on each commit to master (if they wanted). Originally I thought about trying to configure Minikube for this as we discussed, but after doing about 30 minutes of internet research, I was unable to find anyone else testing k8s deployments in such a fashion. My inference is this could mean it's not a best practice.

Even if it turns out it makes sense to do some sort of Minikube testing, the workflow of vcs->repo2docker->download images and deploy on local k8s->deploy on production k8s seems like it could introduce a lot of complexity, especially considering how to integrate this with the existing CI/CD tools. In terms of costs of testing on the cloud, certainly they're there, but running quick tests and immediately tearing things down seem cheaper than no tests and working for X amount of time until you realize something is broken.

Validation - End-to-end Testing:

https://kubernetes.io/docs/getting-started-guides/ubuntu/validation/

Additional information:

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md

thoughts @jhamman?

[jhamman]

I think it will be a constant challenge to just test integration points and not just the core functionality of individual libraries. For example, the test suites in dask-kubernetes and xarray should cover your three tests -- unless there is something unique to your use case (e.g. the configuration of the k8s cluster or the input COGS).

I'm not apposed to testing on a production k8s cluster either but I think it should be possible to also test on a minikube like thing.

[alando46]

@jhamman this is great feedback, thanks for sharing. I think I'm still trying to get the hang of integration test best practices, but for now am going to spend time exploring what could be useful tests to run during build-time per your suggestions here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rabernat]

Let's keep this alive.

@alando46 - haven't heard from you in a while! How are things going on your end.

In terms of testing, you might be interested to see the stuff we have going in:

• https://github.com/pangeo-data/pangeo-datastore - continuous integration of intake catalogs and cloud-based zarr datasets
• https://github.com/pangeo-data/pangeo-stacks/ - automatic building and testing of pangeo core docker images

[alando46]

@rabernat hey there! Things have gotten pretty busy at Radiant these last two months and I've had to temporarily put my Pangeo contributions on hold. But the dust is settling somewhat and I plan on picking up where I left off soon. Thanks for sharing those links! I'll probably initially submit as a WIP pr, but regardless hope to have something up in the very near future.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.