Add Dockerfile for Ubuntu 20.04 (focal fossa). Also fixup docker file to

[cdmatters]

build xenial properly and remove ncurses dependency.

[heiner]

Adding @edran who coded this originally (thanks! <3).

I'd love to understand what this is even doing? We're pushing images for different Ubuntu versions to some kind of image hosting service?

[cdmatters]

Adding @edran who coded this originally (thanks! <3).

I'd love to understand what this is even doing? We're pushing images for different Ubuntu versions to some kind of image hosting service?

My understanding is this is pushing to fairnle's dockerhub which allows third parties to simply develop/pull in our existing images as a base layer, rather than building from scratch. These commands only run on releases though, so it makes sense that the last dockerhub releases were on 30th september to coincide with release of 0.5.2.

My concern is i don't have access to the dockerhub so i dont know how focal will play with there not being a registered repo there (do I need to set one up?), and also, how would we feel about moving fairnle/nle to be the more explicitly labelled fairnle/nle-bionic? The image has only had 200 or so pulls, so it might be okay to have a breaking change, but up for discussion.

PS: Also: hi @edran nice to e-meet you :) Great work on nethack!

[heiner]

Re: the breaking change: Fine by me. We could also add nle-latest?

I'll see if I can add you to the dockerhub group in the morning.

[edran]

What @condnsdmatters says is correct (hello! 👋 ). The hub is useful because it enables anyone to build upon existing images, as well as providing us with a way to deal with CI correctly (one day... right now we are mostly using GH / CCI's native envs, which is maintenance cost that is probably not necessary).

As for improvements, I would suggest the following:

• Definitely add @condnsdmatters to the fairnle hub org. If any of you give me your account id, I can do that immediately.
• Since the images are essentially $build_env + $nle_version_on_build, we could push an image every time the respective dockerfile is updated (which would most likely update the build env) and use that in CI. I actually thought I had done that already, however looking at the latest daily CI logs, a lot of things have been failing for quite a few weeks, so something is definitely dodgy and needs to be reviewed. I normally would have kept track of that, but thesis write-up has taken over everything :( Anyhow, I can probably look into that over the weekend if it's necessary.
  • I also wouldn't do this by creating another repository (e.g. nle-latest), but a tag dev/master to each of the existing repos. EDIT: yep, this should be happening already: https://github.com/facebookresearch/nle/blob/master/.github/workflows/build_docker.yml#L47
• Move the nle tag to use the focal images, and create nle-bionic. This is fairly straightforward, and there's no need to maintain backward compatibility wrt. the main nle tag, as it was always meant to be the "most updated" build env + nle (and docker users tend to version their images anyway).

[edran]

FYI docker/README.md also needs to be updated.

[cdmatters]

* Definitely add @condnsdmatters to the fairnle hub org. If any of you give me your account id, I can do that immediately.

Great! My dockerhub username is condnsdmatters as well. Looks like I last used it four years ago!

* Move the `nle` tag to use the focal images, and create `nle-bionic`. This is fairly straightforward, and there's no need to maintain backward compatibility wrt. the main `nle` tag, as it was always meant to be the "most updated" build env + nle (and docker users tend to version their images anyway).

I've changed my mind on this, and actually propose we keep the "main" image as 18.04, given that's what we use internally. I think loosely encouraging usage of this image can only help us for the time being. My recent change keeps 18.04 as repo fairnle/nle and creates fairnle/nle-focal.

Finally wrt to removing NVIDIA etc... I brought this up w @heiner and see it both ways. I appreciate that people now have to go through a bit more docker hassle, but also it is likely people are going to have to do some customization anyway - no matter what. In light of that it is probably cleaner for the nle repo to maintain a bare minimal dockerfile/image of how to get set up with nle, which people can adapt as needed. Happy to discuss further.

[edran]

My recent change keeps 18.04 as repo fairnle/nle and creates fairnle/nle-focal.

Fair enough, overall 18.04 is also fairly stable and not going anywhere for at least a couple more years, so it's not a bad decision.

My dockerhub username is condnsdmatters as well.

Done, you should be able to access the org. I've also created a nle-focal repo, so in theory it should be pushable.

a bare minimal dockerfile/image of how to get set up with nle

I see your point, and for a more standard project I would 100% agree with you; in this case, however, I think that by default we should be providing images that are "ML / DL ready", since (a) that's what the great majority of our users would want to use with NLE, and (b) benchmarks tend to be most successful when they have as little bootstrapping friction as possible.

That said, I don't feel strongly enough about it, so I won't put up much of a fight if you decide otherwise :)

[cdmatters]

I see your point, and for a more standard project I would 100% agree with you; in this case, however, I think that by default we should be providing images that are "ML / DL ready", since (a) that's what the great majority of our users would want to use with NLE, and (b) benchmarks tend to be most successful when they have as little bootstrapping friction as possible.

That said, I don't feel strongly enough about it, so I won't put up much of a fight if you decide otherwise :)

I sympathise with this, and am pretty pro having one 'ML ready' image. But rathen than including vim etc ourselves, perhaps we could build off a widely used image (there must be a pytorch one, or maybe even jupyter etc)

[heiner]

So when I tried debugging the recent memleak fiasco I wanted something that builds fast so I threw out PyTorch etc, and ubuntu:latest seemed more "standard".

Now that we have Dockerfile.valgrind too I'm not against going back to the nvidia ubuntu images that ~somehow support GPUs (I'm not 100% sure how to use them but that's a different matter). I'd still not install PyTorch. Our users may also use jax, tf, etc.

[cdmatters]

base images updated. if someone hits accept i will merge.

[heiner]

Cool, thanks.

Let's perhaps give @edran a chance to offer an additional take.

[edran]

LGTM, great job Eric!

Using a distro matrix for the docker job declaration is probably good. It is sad GH action doesn't support yaml anchors (or a better language...).

[heiner]

Cool, thanks for this @condnsdmatters!