Toward a stable filesystem api

[nrdxp]

Second Draft

We need to adapt to Nix now being a language with a proper module system, considering flakes to be essentially equivalent to a module in other languages. This should help create clear and distinct lines between different elements of the system, and keeps private code from leaking. For example, profiles could be it's own subflake of DevOS, which exports all profiles as nixosModules to then be consumed by the parent flake. Not every directory need be a flake, and many of the existing directory folders could be grouped together in a shared flake to reduce the complexity of the filesystem layout, without sacrificing any of the already established conveniences of DevOS.

This has the added benefit of allowing power users to pull only relevant pieces of DevOS, such as it's lib, in a flake ref: "github:divnix/devos?dir=lib".

Impetus

In a word: complexity.

From the very first moment of deploying DevOS to the world, I knew the fs hierarchy had to change. I was hoping to illicit feedback from the community. As an example, one of the complaints I hear echoed is the default imports, which happen it different ways in different places, and if kept at all should definitely be reduced to a simpler implementation. This issue will remain open until a solid layout has been decided on, so we don't have to continue changing it out from under users. I want a flake centric approach, and internal flakes seem like the right way to manage access, at least from a high level.

For example, with this approach we should be able to maintain the convenience of the current doc setup, i.e a README.md in every directory explaining it's use, without actually needing the book.toml and SUMMARY.md in the root. We can just pull in the src flake to the doc flake, and overlay the book.toml onto it to generate the book.

Once a demo is achieved that maintains all the established conveniences of current DevOS, and a proof-of-concept for #127, I will open a PR that we can then scrutinize.

Changes

We would turn off the github template, and generate the template as a derivation with the docs optionally included for users. This will eliminate one level of nesting for users, while also eliminated irelevant code from the users template.

Goal

This won't change once 1.0 is reached, so we need to get it right, aiming at convenience for users without restricting what they can do.

---

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

[Pacman99]

Can you explain what you mean by out? And why things like overlays and pkgs go there?

And I'm not sure how to fix this or if it should be changed, but its pretty annoying that to edit a profile you now have to go three directories down(including profile folder). I feel like profiles are the most used folder.

But overall I do like the separation of different folders.

[nrdxp]

@Pacman99, I agree, profiles are only two levels deep if you look carefully, but this still may not be optimal. Might have gone overboard with nesting, and still not sure exactly how many internal flakes we should have. Just wanted to brainstorm here. We can take as long as we need to get it right, but I do want to get a PR open fairly soon, so we can start scrutinizing actual code.

out is short for flake outputs. Essentially each output would have it's own folder, and for that output, the flake would do an import, e.g. nixosConfigurations = import out/nixos. A big complaint we have is complexity, and I was trying to make it more obvious for users. We could match the folder names exactly, i.e. outputs/nixosConfigurations as well.

In general, don't take the visual in the OP too seriously, it is really meant more as a cue to help everyone understand which direction my brain was going. I'm sure the final product will look much different.

[blaggacao]

I'm going to pack top level with other things:

./Mgt # docbook for startup project management, like operating manual, strategy, etc.
./Accounting # beancount subfolder to keep a bencount accounting up and going
./Receipes # for recie collection, maybe dokbook (it's a restaurant)
...

So keeping top level clean(er) is indeed a thing that would be useful for my use case.

[nrdxp]

Indeed, the nesting can be mitigated mostly by generating the template from the src dir, so when users go to pull the template, they only have the contents of src, and maybe the docs.

[blaggacao]

Yeah, to include docs would be good, since it is a very handy trampoline for extending on all sorts of documentation.

[nrdxp]

We should be able to actually let users decide if they want docs or not. A naive implementation is one template with, one without the docs.

[nrdxp]

Since this is 1.0, and nickel seems like a big part of the "nix" future, I would really like a contract to guarantee the spec. This may be possible already.

[nrdxp]

ping: update proposal to second draft, accentuating thesis.

[tgunnoe]

https://github.com/divnix/devos/blob/core/pkgs/flake.nix I dont understand the idea of subflakes, can you explain your reasoning? Or is there an old topic I can look at about it?

This flake seems like a lot of code to just parsing version/rev numbers from pkgs, and I don't see the gain. Before I was able to just as easily nix run '.#<pkg>' and items in the pkgs/ directory would match what could potentially go upstream to nixpkgs. And now they wont. Plus, arent you doing flake tasks in multiple places now? At first usage it seems to add complexity for no gain.

[blaggacao]

The main advantage is that nix flake update does dealing with the hashes, which I'd consider is not very elegantly solved in nix proper, from the perspective of a +- newcomer.

The main disadvantage is that it introduces a new specialArgs srcs from which to pull sources. Hence any package declaration would need a tiny transformation before being upstream-able, but at the gain of easier package updates.

{
  src = srcs.mypackage-flake-input;
}
# vs
{
  src = fetchurl ( ... );
}

Other than that, the behavior, especially nix run '.#<pkg>' should be exactly the same, otherwise it's a bug.

[nrdxp]

I go back and forth on this. On the one hand, modifying the hash to upstream really isn't so much work. On the other, it kinda makes the source opaque, and the interface for subflakes isn't the best atm. I'm hoping there will be some usability improvements for internal flakes at some point that decide the issue in my mind. I want something like inputs.some-input.url = "self:some/subflake".

The way the lock file is updated is kinda strange as well, and seems to cause some strange issues.

The good news is, you can just not specify your source in pkgs/flake.nix if you'd rather do it the old way, that's a perfectly acceptable option. If that wasn't the case, I might be tempted to revert for now.

Either way, I'll probably hold off on adding more subflakes until this becomes easier to deal with.

[tgunnoe]

Yeah I think its a lot of complexity just to deal with 1 rev hash, that will need to be undone anyways before being added to nixpkgs.

And yes, i already had that exact issue with it-- updating the 'subflake' leads to a bad NAR hash to your main flake. Thats originally why I had to inspect the pkgs/flake.nix in the first place.

Also the way its called is done by relative path at the moment. Which when I need to act on my flake repository locally i have to be within the exact root directory where the flake is located.

I think it's not a directory where you'd want to mass update the hashes/commits of repos either, because you're in the progress of getting them upstreamed to nixpkgs individually and not all at once. So I don't see the utility of nix flake update but I do see a lot of added complexity.

Just my feedback after having updated from a state before "suites", which ive come to like!

[blaggacao]

Also, I might add, that it sometimes becomes a little redundant to re-declare a ref, usually a version tag that also holds the version derivation argument.

Maybe ther would be a way to revert this and rather think of how to otherwise helpmpeople withbupdating their hashes (it's not too bad, but it is annoying).

[Pacman99]

With the recent changes in 0.9 and the dropping of extern in #264, are there still any problems with the filesystem api?
I really file structure as it is right now, every folder and file has a clear purpose, there aren't too many folders. And with the switch to devos as a library, you can really easily change it.

[nrdxp]

I think we are definitely close. Maybe we should leave this open, at least until the next release, and see what state we are in at that time.

[blaggacao]

Closable via #293

[nova-nowiz]

should this issue be unpinned as it is closed?