Policy on adding new images

[mathbunnyru]

It would be nice to come up with a policy when we add new images and when we don't.
#1936

Another question is - do we allow ourselves to remove some images (stop building new versions) for some image.

[consideRatio]

An incomplete history

I was thinking that we could extract some historical decisions for reference going onwards and arrived to this as a starting point.

• Initial stacks #1
  Added minimal-notebook, scipy-notebook, r-notebook. At this point, also all-spark-notebook, datascience-notebook, and julia-notebook was considered.
• Notebook stack for Python w/ Spark on Mesos #3
  Added pyspark-notebook
• Notebook stack Scala, Python, and R w/ Spark on Mesos #5
  Added all-spark-notebook
• New docker stack for a headless kernel #42
  Added minimal-kernel
• Base notebook stack #209
  Added base-notebook by refactoring minimal-notebook
• Discuss: Remove minimal kernel #213
  Removed minimal-kernel
• add tensorflow book #266
  Add tensorflow-notebook
• Add a tiny alpine based notebook #356
• Add PHP Stack #444
• Add ruby-notebook for Ruby 2.4.2 stack #486
• Add PyTorch notebook #533
• [WIP] Deprecate pyspark-notebook, r-notebook, Mesos support #693
• Add sparkmagic-notebook Image #974
• Add nvidia based notebooks to stack #1196
• Add octave notebook with popular packages #1204
• Add SensiML notebook, based on TensorFlow notebook #1234
• Add nteract front end #1327
• Add datascience-python-r-notebook #1569
• Snowpark notebook #1745
• Create base image to base-notebook for non-server Jupyter applications #1825
  Adds docker-stacks-foundation by refactoring base-notebook
• Add jupyter/julia-notebook #1926
  Added julia-notebook
• Add jupyter/pytorch-notebook #1936
• Add image with Linux Desktop environment in it #1961

[mathbunnyru]

Thanks @consideRatio. This is really helpful.

[mathbunnyru]

So, I have a list of a few things we should consider when adding new images (I also express my own opinion here and people might disagree with it and it's ok):

1. Popularity. We should not add a separate notebook, which will contain just YOUR_FAVOURITE_RARE_PACKAGE_HERE, just because it's someone's favourite package. If this package is really popular, it's good. The software added should be modern and well-supported.
2. Added value and potential user database growth.
3. Consistency with current images. I don't think we should add something completely different, for example, an image with C++ Jupyter Kernel.
4. Maintenance and especially build time. Since we're using Docker, what's inside the container doesn't affect our build workflows (most of the time). But some images might be heavy to build, especially since we upload/download images as archives.
5. Any special hardware/equipment - I don't think we should be adding ppc64le images, for example. On the other side, I'm ok adding GPU images stack, if someone is ready to implement this without sacrificing build time (it's possible, but needs some work to do) and if the license allows us to do so.
6. We should accept the idea that we can't live in a model where "our images fit 95% of needs". I think the correct model - you should choose an image closest to your needs and then install a few packages on top of this image.
7. I think we should stick to the philosophy of using the latest software available. It worked quite well for us.

[mathbunnyru]

The message above is my opinion (which might easily change) and not a statement 🙂
Please, feel free ti share your ideas.

[twalcari]

With regards to point 5: adding GPU-support is non-trivial because the version of CUDA that you bundle is an extra confounding factor. For example, PyTorch has separate package indexes for CUDA 11.7 and 11.8. Do you bundle cuDDN or not? When the version of CUDA bundled into the docker image is newer than what's supported by the Nvidia drivers on the host system, it will crash, etc.

[mathbunnyru]

Thanks, @twalcari, that's a good point.

I think we need a way to build all the images tree in parallel.
And then to be able to tag the whole tree correctly.

For example, we can build the images tree with simple Ubuntu (as now), and also build the images tree for cuda 11.7 (and add some tag prefix like cuda11.7- for all these images) and cuda 11.8.
And have options for cuDDN as well.
Obviously, we can build all this in parallel.

So, if someone wants to add templating based on the root image, even without adding gpu-support, it seems to be a nice feature to have.

We will also have to take a look at licensing of cuda-based images, I think one of the reasons they were not added was not being able to redistribute the images because of the cuda license (I'm not sure about this one).