From f80afe27e285ac73e5b2ca1c9bcc105cc2824cf8 Mon Sep 17 00:00:00 2001 From: z8v Date: Thu, 7 Sep 2023 14:04:22 +0300 Subject: [PATCH 1/8] Add tutorial for setting up the remote execution infra --- docs/remote-execution-tutorial.md | 131 ++++++++++++++++++++++++++++++ 1 file changed, 131 insertions(+) create mode 100644 docs/remote-execution-tutorial.md diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md new file mode 100644 index 000000000..ec82c4b50 --- /dev/null +++ b/docs/remote-execution-tutorial.md @@ -0,0 +1,131 @@ +# Bazel remote execution using `rules_nixpkgs` + +During a Bazel build with `rules_nixpkgs`, a series of Nix paths will be created on the local +machine, which are then referenced by Bazel either directly or indirectly. The challenge with remote +execution is to ensure that the executors have those Nix store paths available. + +To accomplish this `rules_nixpkgs` will copy those paths, through SSH, to a remote Nix server. Then +the paths can be made available to remote executors through a read-only NFS mount. + +## Steps + +### Setup the Nix server + +The first step would be to setup a Nix server. We assume a Debian/Ubuntu for the following steps but +any Linux system should do. + +1. Install nix +2. Install the NFS server package +```bash +sudo apt-get update +sudo apt-get install nfs-kernel-server nfs-common +``` +3. Edit the NFS server's export file (`/etc/exports`) + +The following line assumes an internal private CIDR where the mount can be used (`10.0.0.0/16`). You +can adjust according to your setup. + +```conf +/nix/store 10.0.0.0/16(ro,sync,no_subtree_check) +``` +4. Start the NFS server +```bash +sudo systemctl start nfs-kernel-server +``` + +### Setup the executors + +The goal of this step is to mount the exported file share from the Nix server. This will enable the +executors to have the current state of the Nix store. + +The exact setup will different depending on what kind of executor you're using. Below are some +examples of such configurations. + +### VM server + +1. Install the nfs packages +```bash +sudo apt-get install nfs-common +``` +2. Mount the file share + +For this we'll need an entrance like the following on the `/etc/fstab` file. +```conf +host_ip:/nix/store /nix/store nfs ro,nfsvers=3 0 0 +``` + +### [BuilbBarn](https://github.com/buildbarn) on Kubernetes + +Buildbarn provides Kubernetes manifests that you can use to deploy an executor. In order to make it +compatible with the `rules_nixpkgs` we'll need to mount the NFS share. Luckily this is supported +already on Kubernetes. + +1. Fetch the Buildbarn manifests from https://github.com/buildbarn/bb-deployments/tree/master/kubernetes + +2. Adjust the worker manifest. + +Update the `Deployment` spec file of the +[worker](https://github.com/buildbarn/bb-deployments/blob/d142377ce90d48407f01ca67a7707d958de38936/kubernetes/worker-ubuntu22-04.yaml) +to include the NFS share mount on the `runner` container: + +```yaml +... +spec: + template: + spec: + containers: + ... + name: runner + volumeMounts: + ... + - name: nfs-vol + mountPath: /nix/store + volumes: + - name: nfs-vol + nfs: + server: 10.0.0.1 # Replace with the NFS server IP + readOnly: true + path: /nix/store + ... +``` + +### Configure `rules_nixpkgs` for remote execution + +The final step is to configure our Bazel project to use the Nix server and remote execution. + +1. Enable copying of Nix paths to the remote server. + +This can be done by setting the `BAZEL_NIX_REMOTE` environment variable. This should be the name of +any entry on the [SSH config](https://www.ssh.com/academy/ssh/config) file where all the +authentication details are provided. + +``` +$ cat $HOME/.ssh/config + +Host nix-server + Hostname 10.0.0.1 + IdentityFile ~/.ssh/nix-server + Port 2222 + User nix-user + +export BAZEL_NIX_REMOTE=nix-server +``` + +2. Configure remote execution. + +We can't give exact instructions for this steps because it on your specific setup and the executors +or third party service you're using. + +Overall this should not affect the way `rules_nixpkgs` works once the Nix paths are available on the +executors. + +Example config for Buildbarn: + +```conf +build --remote_timeout=3600 +build --remote_executor=grpc://: +``` + +You can use one of the +[examples](https://github.com/tweag/rules_nixpkgs/tree/master/examples/toolchains) to test this +setup. From 55b586e4229d74918a2fc7c88ff26ddb8f01e364 Mon Sep 17 00:00:00 2001 From: z8v <86471569+z8v@users.noreply.github.com> Date: Sun, 10 Sep 2023 11:22:34 +0300 Subject: [PATCH 2/8] Update docs/remote-execution-tutorial.md Co-authored-by: Ben Radford <104896700+benradf@users.noreply.github.com> --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index ec82c4b50..e3d59989d 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -38,7 +38,7 @@ sudo systemctl start nfs-kernel-server The goal of this step is to mount the exported file share from the Nix server. This will enable the executors to have the current state of the Nix store. -The exact setup will different depending on what kind of executor you're using. Below are some +The exact setup will be different depending on what kind of executor you're using. Below are some examples of such configurations. ### VM server From 0efecd42c4bfa898af005bb72fcc2aeddda2bc2c Mon Sep 17 00:00:00 2001 From: z8v <86471569+z8v@users.noreply.github.com> Date: Sun, 10 Sep 2023 11:22:40 +0300 Subject: [PATCH 3/8] Update docs/remote-execution-tutorial.md Co-authored-by: Ben Radford <104896700+benradf@users.noreply.github.com> --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index e3d59989d..7012de89e 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -49,7 +49,7 @@ sudo apt-get install nfs-common ``` 2. Mount the file share -For this we'll need an entrance like the following on the `/etc/fstab` file. +For this we'll need an entry like the following in the `/etc/fstab` file. ```conf host_ip:/nix/store /nix/store nfs ro,nfsvers=3 0 0 ``` From a8060b1cfbfa23b93f0c1b15cf29b6e5f7bfbf4e Mon Sep 17 00:00:00 2001 From: z8v <86471569+z8v@users.noreply.github.com> Date: Sun, 10 Sep 2023 11:22:45 +0300 Subject: [PATCH 4/8] Update docs/remote-execution-tutorial.md Co-authored-by: Ben Radford <104896700+benradf@users.noreply.github.com> --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index 7012de89e..2c6f9a3b9 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -54,7 +54,7 @@ For this we'll need an entry like the following in the `/etc/fstab` file. host_ip:/nix/store /nix/store nfs ro,nfsvers=3 0 0 ``` -### [BuilbBarn](https://github.com/buildbarn) on Kubernetes +### [BuildBarn](https://github.com/buildbarn) on Kubernetes Buildbarn provides Kubernetes manifests that you can use to deploy an executor. In order to make it compatible with the `rules_nixpkgs` we'll need to mount the NFS share. Luckily this is supported From aa703bd84a74f8651fb9e2c5afe35a270fd82b98 Mon Sep 17 00:00:00 2001 From: z8v <86471569+z8v@users.noreply.github.com> Date: Sun, 10 Sep 2023 11:22:52 +0300 Subject: [PATCH 5/8] Update docs/remote-execution-tutorial.md Co-authored-by: Ben Radford <104896700+benradf@users.noreply.github.com> --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index 2c6f9a3b9..11c5afca4 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -96,7 +96,7 @@ The final step is to configure our Bazel project to use the Nix server and remot 1. Enable copying of Nix paths to the remote server. This can be done by setting the `BAZEL_NIX_REMOTE` environment variable. This should be the name of -any entry on the [SSH config](https://www.ssh.com/academy/ssh/config) file where all the +an entry in the [SSH config](https://www.ssh.com/academy/ssh/config) file where all the authentication details are provided. ``` From 7f87fae317154afd7c6ed1fc42c15b7d4f9f54be Mon Sep 17 00:00:00 2001 From: z8v <86471569+z8v@users.noreply.github.com> Date: Sun, 10 Sep 2023 11:23:05 +0300 Subject: [PATCH 6/8] Update docs/remote-execution-tutorial.md Co-authored-by: Ben Radford <104896700+benradf@users.noreply.github.com> --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index 11c5afca4..043aa679b 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -113,7 +113,7 @@ export BAZEL_NIX_REMOTE=nix-server 2. Configure remote execution. -We can't give exact instructions for this steps because it on your specific setup and the executors +We can't give exact instructions for this step because it depends on your specific setup and the executors or third party service you're using. Overall this should not affect the way `rules_nixpkgs` works once the Nix paths are available on the From d084bb4190bc10dd50a0737b11287896d4802483 Mon Sep 17 00:00:00 2001 From: z8v Date: Sun, 10 Sep 2023 16:20:33 +0300 Subject: [PATCH 7/8] Small wording improvements --- docs/remote-execution-tutorial.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index 043aa679b..535cc0c47 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -11,8 +11,8 @@ the paths can be made available to remote executors through a read-only NFS moun ### Setup the Nix server -The first step would be to setup a Nix server. We assume a Debian/Ubuntu for the following steps but -any Linux system should do. +The first step would be to setup a Nix server. Although we base the following instructions on a +Debian/Ubuntu system, most Linux distributions should work. 1. Install nix 2. Install the NFS server package @@ -35,11 +35,11 @@ sudo systemctl start nfs-kernel-server ### Setup the executors -The goal of this step is to mount the exported file share from the Nix server. This will enable the -executors to have the current state of the Nix store. +This step involves mounting the exported file share from the Nix server, allowing the executors to +access the current state of the Nix store. The specific procedures may differ based on your executor +type. -The exact setup will be different depending on what kind of executor you're using. Below are some -examples of such configurations. +Below are examples for various configurations: ### VM server From 02723345703c67a6262bc693f278a0accab9c8ee Mon Sep 17 00:00:00 2001 From: Andreas Herrmann Date: Wed, 11 Oct 2023 13:26:05 +0200 Subject: [PATCH 8/8] Update docs/remote-execution-tutorial.md --- docs/remote-execution-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/remote-execution-tutorial.md b/docs/remote-execution-tutorial.md index 535cc0c47..800268ba1 100644 --- a/docs/remote-execution-tutorial.md +++ b/docs/remote-execution-tutorial.md @@ -1,6 +1,6 @@ # Bazel remote execution using `rules_nixpkgs` -During a Bazel build with `rules_nixpkgs`, a series of Nix paths will be created on the local +During a Bazel build with `rules_nixpkgs`, a series of Nix store paths will be created on the local machine, which are then referenced by Bazel either directly or indirectly. The challenge with remote execution is to ensure that the executors have those Nix store paths available.