Add Docker image for ppc64le and CUDA

[jakirkham]

This adds a Docker image that is an amalgamation of the CUDA Docker image and the ppc64le images to enable building ppc64le packages that have CUDA portions enabled as well.

cc @jayfurmanek

Checklist

• Used a fork of the feedstock to propose changes
• Bumped the build number (if the version is unchanged)
• Reset the build number to 0 (if the version changed)
• Re-rendered with the latest conda-smithy (Use the phrase @conda-forge-admin, please rerender in a comment in this PR for automated rerendering)
• Ensured the license file is being packaged.

[jayfurmanek]

This is great! Thanks!
Are there too many license problems with putting CUDA into its own conda pkg like defaults does?

[scopatz]

Are there too many license problems with putting CUDA into its own conda pkg like defaults does?

Yes, sadly....

[jakirkham]

Are there too many license problems with putting CUDA into its own conda pkg like defaults does?

So this contains the nvcc compiler. I'm not aware of a Conda package that includes this. Is there one you know of?

If you are referring to cudatoolkit, we do plan to leverage that package from defaults and attach it as runtime dependency of packages built with nvcc. Please see this code for context.

[jakirkham]

@jjhelmus, do you know if it would be possible to get some updated cudatoolkit packages in defaults for ppc64le?

[jayfurmanek]

I work on IBM's WML-CE and we have both cudatoolkit and cudatoolkit-dev packages:

https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/#/

The -dev package has nvcc and the lot. We moved to conda just this year so we started with cuda10.1 and will continue from there. Older CUDA versions have various problems on ppc64le anyway - some better than others.

Packages in WML-CE are free to use and there is a auto-acceptable license that comes along with.
I'm not sure it helps here though, being external. We could contribute the recipe, but then you're back to the CUDA license terms.

[jakirkham]

Yeah we can look into this as well.

Initially our thought was it would be nice to use the Docker image to simplify the install and distribution process. They've also worked reasonably well for our use cases.

Hopefully we can move to Docker multi-arch images in the future to improve organization and maintenance here. ( #102 )

[jjhelmus]

At this time we are not building GPU packages for the linux-ppc64le platform in defaults so there are no plans to update the cudatoolkit package.

[jakirkham]

Completely understand Anaconda not building out the stack if it doesn't make sense. However would it be possible to get an updated cudatoolkit package alone? This would be useful for building out GPU packages in the open. Or are there other concerns with doing this that I'm missing?

[jakirkham]

Have raised issue ( AnacondaRecipes/cudatoolkit-feedstock#4 ) about ppc64le packages for cudatoolkit.

[isuruf]

@kkraus14 mentioned that they were asking NVIDIA internally about distributing these through conda-forge. Any updates?

[jakirkham]

Nothing yet.

[isuruf]

@conda-forge/core, this is ready for review.

The hold up was caching of defaults::cudatoolkit which was done for amd64, but there's no such package for ppc64le. I've removed caching because of licensing issues, we can't ship it (even if including it as part of the docker image). We can ship the docker image with the base layer from nvidia because we aren't shipping the base layer and also because there's a special licensing term.

[jakirkham]

I don’t understand how this can be useful without that cudatoolkit package. Also I don’t follow the comments about licensing. This should wait until the cudatoolkit package issue is addressed.

[isuruf]

@jakirkham, this is useful for packages using only the driver API. They don't need cudatoolkit.

About licensing, we are not distributing cudatoolkit in conda-forge as a conda package because of licensing issues. If that is the case, how can we redistribute the defaults::cudatoolkit package in dockerhub?

[jakirkham]

If we don’t have cudatoolkit, this will still not work for many use cases and will be confusing to users. Let’s work with Anaconda to do this right.

The Docker image is a special case in NVIDIA’s licensing that allows the distribution of the CUDA toolkit in the image. It in fact already ships with everything in the cudatoolkit package and then some.

As I’m taking time off atm, I’d very much appreciate pushing pause here. This has been waiting for a while. Waiting until next week to discuss this should be fine.

[isuruf]

Okay, let's not merge this for a week, but we can keep the discussion going.

[jakirkham]

Just to be clear I don’t think this should be merged without addressing the things mentioned above.

[isuruf]

@jakirkham, this image is useful for packages using CUDA driver API which doesn't need cudatoolkit. Also this image is useful for people who would want to use cudatoolkit image from IBM as mentioned above, but in a custom organization. This would make it easier for me. Any other thoughts about the usefulness of the image without cudatoolkit?

[isuruf]

ping

[jakirkham]

Thanks @isuruf! Adding a comment seems sufficient for getting this point across. If we find people are still confused, we can figure out more appropriate messaging then. LGTM

[jayfurmanek]

Looks to be as good as we can make it at the moment. Thanks @isuruf

[isuruf]

Thanks @jakirkham for the work and everybody for the review.

[jakirkham]

Have also raised issue ( #141 ) about this so we can track this need/redirect people there.