Add a way not to use the user's computer as a cache

[tobiasBora]

Sum up: would it be possible to never download any store on the client computer?

First, thank you for all the good work.

Then, I'm here because there is a fundamental problem that makes me unable to use the very promising nixops, while I'd love to use it instead of other non nix-based alternatives. Let's begin with a small analogy. Let's imagine that you work in a big company that makes unicorn food. To produce this food, the company needs one big machine named "SuperRainbowMachine", which is very big and expensive. Do you think that it would be "normal" for the boss of this company to force all the workers of this company to buy and keep a "SuperRainbowMachine" at home, even if they don't even have a unicorn? Of course not, you would refuse it. But because you are (really) nice, you finally accept it. And now your boss comes back to you and tell you "Each time you want to produce a new piece of food, you need to produce it in our company using the SuperRainbowMachine, then you need to bring this piece of food at home, and bring it back to the company place in order to put it in these bags". Would you find it logic? Of course not.

Well, that's more or less the feeling I have when I try to use nixops. Indeed, I have a raspberry pi 3 model B+, that runs nixos. Now, I'd like to put the configuration on a git server, modify it from my laptop, my mobile phone, my work computer... and deploy it to the raspberry pi. So I created the two files for configuration and network description, as explained in the documentation, and I began to deploy it:

nixops deploy -d test

I would expect this command to copy the configuration on the raspberry pi, and remotely run a nixos-build switch, and everything would be done. However, my laptop started to download tons of things locally from the binary cache... I asked in the irc channel, and clever (thank you very much for you help) explained me that this was normal, and that all the packages that are installed on the raspberry pi are first downloaded to the local computer, then sent to the raspberry pi, and also these files must be present in the local store for a long time, as soon as you want to keep editing the raspberry pi without redownloading everything. Still weirder, when the configuration is not in the binary cache, my computer tries to compile it, and because my computer is x86_64 and the raspberry pi is aarm64, it just fails... So because of that clever told me to set-up the raspberry pi as a builder... So to sum up, my laptop will ask to the raspberry pi to compile a program, then this program will be send to the laptop, and then the same program will be used later by the raspberry pi... Still still weirder. The raspberry pi needs to have the same nixpkgs revision as the client, so I need to pin a specific revision (thanks again clever):

nixops modify -d router -I nixpkgs=https://github.com/nixos/nixpkgs/archive/ce0d9d638de.tar.gz deployments/router_deployment.nix

But all these things seems crazy to me, and I really don't understand why we need to store the server store on the laptop side. It does not even have the same architecture, how can it be useful! And then, why do you need to have the same revision of nixpkgs on both the raspberry pi and the client? For me all the point of nix is that because it's declarative, you can deploy it on a server by just copy/pasting the configuration files and that's all. So why shoud we use all these things to just do a simple deployement? There is also a problem as soon as you have several computers that can push to the raspberry pi: first it means that all the clients need to have a local copy of the full store of the server, while they will never use it... and my phone can't just handle that! For me, you should be able to deploy a configuration on a server no matter your architecture is, no matter which revision of nix you are using, no matter which available space you have... All of this should be the job of the raspberry pi !

So to sum up, I'd like to know why nixops is designed that way, and also I'd like to know if it would be possible to avoind any store duplicte on the client side.

Thank you!

[aij]

Nixops is really meant for managing a bunch of servers rather than just one.

For example, when I run nixops deploy on my laptop, it will evaluate the configuration for the whole deployment once, using my configuration and custom nixpkgs (even changes not committed or pushed anywhere else). Then it will fetch and/or build the necessary packages only once, and then just copy them to all my servers over the local network.

It sound like what you want is the exact opposite though. If you only have one server, and want to be able to edit it from any number of clients, why not keep the configuration on the server and edit it from any client you want?

Anyway, I think if you rephrase your questions in light of 1 client + N servers, most will be self-evident. Let me know if any are not.

One additional puzzle piece is that nixops supports managing secrets (like encryption keys) without writing them to the server's disk, so copying the entire configuration would be problematic rather than just unnecessary.

[tobiasBora]

Thank you for your answer.

Well for me it's important to do the distinction between several types of entities (we can add more, but these are the fundamentals I think):

• the developpers
• the servers
• the build servers that compiles and store the binaries (that could also include Hydra)

The goal of the developpers (called client above) is to describe the specification of the different servers. The role of the servers is to run the end application. The role of the build servers is to optimize the process in order to avoid multiple compilation. For now, the developpers and the build servers are mixed in a single entity, which gives the awkward problems I listed above. A concrete example of application would be this one:

What would you do, for example, if you are a big company, with 100 servers that, all together, require 60Gb of memory for the applications, and 200 developpers? With the current system, the 200 developpers would need to:

• compile all the applications of the 100 servers (may be avoided by asking to each developper to add a given build server)
• store all the applications on the local machine (you are talking about keys, so it means if I follow that in this case all the employeurs would have the keys of all the servers on there computer... that's not really secure isn't it?)

So it means that each developper will compile and store 60Gb of binaries, i.e. hours of useless compilation and 12000Gb of useless data...

While with the 3 entity model described above, we can configure one server to be the build server (it can even be optional if we have no costly build because we are using the Hydra binary cache), and that way we compile at most one time the binaries, the developpers store nothing but a few text files for the configuration, and each server stores only the binaries he needs. So a total of 60Gb if we use only Hydra (optimal value), or 120Gb if the build servers build everything... Quite different from the 12000Gb above!

Finally, converning your remarks about secrets, I understand that one need to carefully choose how they are handle. But lot's of tools can deal with them without overcharging the developpers (see Salt for example), and I think that the issue you point out can be easily solver by simply sending to the server X only the configuration (and secrets) of server X.

-- EDIT --
If you still think that it's not the point of nixops to tackle multiple developpers (which would be really sad, but after all it's your choice), do you know any other project that can deal with it?

[coretemp]

Perhaps this helps to use NixOps in a team: https://blog.wearewizards.io/how-to-use-nixops-in-a-team.

[samuela]

I'm facing more or less the same dilemma. It came as quite a surprise to me that nixops puts the burden of building the server configurations entirely on the client machine and not on the servers themselves. It's not at all what I expected and like OP it's also a dealbreaker for using nixops for me since I need the ability to control a fleet of machines from multiple clients that are not all running NixOS.

[samuela]

Perhaps this helps to use NixOps in a team: https://blog.wearewizards.io/how-to-use-nixops-in-a-team.

This link seems to be broken. Is there an updated url?