lib: init and use mkFlake and evalFlakeArgs #218

Pacman99 · 2021-03-26T19:13:46Z

Add general lib functions to create a devos flake. These functions do not enforce a file system structure, but just provide an api to create a flake.
The options that have types do not allow paths, see this comment for my reasoning.
Module system allows for type checking and storing all data about the flake within one comprehensive evaluated module. It also allows for adding defaults, so folders and files are no longer required. You can safely delete the userSuites or just not pass modules if you aren't using those features, and devos won't error out like it does now.

Then in devos's flake.nix we now call mkFlake and pass the arguments to enforce this template's file system structure.

I mentioned this in other places, but I do not want to change any of devos's api in this PR. So for example @blaggacao has brought up that extern might be somewhat necessary and could be changed. But I'm still going to keep extern in this PR since thats how devos works right now. Then after the result of #214 and/or #152 we can update mkFlake and devos's flake.nix appropriately in separate PR's.

Thank you @blaggacao for the great ideas! Credit goes to him for the idea of making mkFlake more general. It took a bit for me to understand, but this is way better.

Pacman99 · 2021-03-26T20:05:04Z

Moved templates and devosOptionsDoc out of mkFlake, since they are devos specific. devosOptionsDoc is added to pkgs, I think that would be the best place to add it, following the devos api.

Pacman99 · 2021-03-26T20:09:51Z

Dependent PR's:

Pacman99 · 2021-03-26T20:21:46Z

Can't build devosOptionsDoc It just doesn't seem to exist. I wonder if its because of the issue with flattenTreeSystem. It shows up in overlay but doesn't show up in the packages output.
Yup! dropping flattenTreeSystem fixes this 🤷

nrdxp · 2021-03-26T21:10:42Z

I want to filter packages by system, but I don't think we actually need the "flatten-tree" funcitonality anyhow, since pkgs/default.nix is already flat. Perhaps we could make it simpler and invert the logic, so that if a system isn't explicitly specified, instead of being filtered out as it is now, it is automatically included in all systems. This encourages packages that should be systemed to do so, or CI won't pass, while allowing derivations that can run on any system without issue do so automatically.

Pacman99 · 2021-03-27T04:55:31Z

lib/mkFlake/evalArgs.nix

+          type = path;
+          default = "${self}/hosts";
+          defaultText = "\${self}/hosts";
+          apply = toString;


toString is necessary here otherwise for some reason the mk* functions create a new store path with just that folder and then nix fails because of an access to a restricted store path. either way its better to pass these paths as strings.

Pacman99 · 2021-03-27T05:23:12Z

Added a doc section, 'use devos as a library!' which is an updated version of the 'use-mkdevos' doc from the other PR.

Pacman99 · 2021-03-29T17:41:54Z

lib/devos/mkHosts.nix

@@ -1,6 +1,6 @@
 { lib, dev, nixos, inputs, self, ... }:

-{ dir, extern, suites, overrides, multiPkgs, ... }:
+{ dir, devos, extern, suites, overrides, multiPkgs, ... }:


So users of mkflake can safely delete the core profile or cachix module.

Pacman99 · 2021-03-29T17:48:19Z

lib/mkFlake/default.nix

+      checks =
+        let
+          tests = nixos.lib.optionalAttrs (system == "x86_64-linux")
+            (import "${devos}/tests" { inherit pkgs; self = devos; });


I think passing devos as self is optimal for now. We know devos will always have self.nixosConfigurations.NixOS, while end user might delete the default hosts. And we still have test coverage of self with deploy-rs checks. But in the future I think it would be good to pass both self and devos. And have a mix of testing devos's hosts, and the users hosts/profiles. But lets save for a future PR.

nrdxp · 2021-03-30T19:15:12Z

What's left to do here, before review?

Pacman99 · 2021-03-30T19:37:55Z

What's left to do here, before review?

I just went through a few cleanups today and improved the option descriptions. This should be ready for review. I don't plan on adding anything else in the PR.

Pacman99 · 2021-03-30T19:41:09Z

lib/mkFlake/evalArgs.nix

+          default = "${self}/hosts";
+          defaultText = "\${self}/hosts";


Defaults for all path options do end up adding some file system structure. I think its ok since it is a path option. But just wanted to note that.

lib/devos/mkProfileAttrs.nix

Pacman99 · 2021-04-01T18:41:22Z

lib/mkFlake/default.nix

+    lib = import "${devos}/lib" {
+      inherit self nixos;
+      inputs = inputs // self.inputs;
+    };


Just added this, so we can also extend devos api to flake inputs. Creates opportunity to do things based on what input exists, allowing integrations to become optional.

for example, // optionalAttrs (inputs ? home) { home-manager = ... }

module describing devos flake arguments

general lib function - not devos specific

this is where we create devos's template structure

generated mkFlake options documentation

Pacman99 · 2021-04-01T20:50:17Z

bors try

bors · 2021-04-01T20:51:00Z

try

Build failed:

ci/hercules/evaluation

It is not generally useful for everybody to export devos-lib. We can reintroduce a lib-exportin interface, once we conceputualize the private library sharing use case properly.

currently, there is no need for a recursiveUpdate, and since flake requires flattened trees, there might never be a good enough reason to use it at the expense of introducing a dependency on lib.

evalFlakeArgs is a private function to mkFlake, so there is no reason expense going over the top level namespace. Unfortunately, keeping boundaries around future devos-lib, this also means we cannot introduce a dependency on pkgs, and vie versa we cannot introduce a dependency on devos-lib lighthartedly (without a clear api) on pkgs. While this is genuinly useful, we ight need to find a cleaner and more organized way to implement this in the future.

we should consequently avoid variable redefintions while traversing through layers and interfaces. self is a flake native term and refers to a handle to the flake repo itself.

since lib is structured after upstream lib (lists, attrs, strings), and it is even conceived as lib.makeExtensible, it does not appear in line to namespace devos specific library additions: not only is the name `dev` little self explaining, we also have the `os` namespace, for if library functions are devos specific. The rest are just confenience functions which might as well ladn upstream one day.

A series of "clenups"

Pacman99 · 2021-04-03T15:18:20Z

More dependent PR's:

drop lib's self dep and pass data to each function #228
make core an unecessary profile and add to suites #229, this should help with path issue, but its not necessary

With those PR's, the current "black magic" will no longer be necessary. lib only needs to be initialized once and self will just be self across lib, the flake that is calling each lib function. And devos isn't necessary.
The only remaining issues is tests and shell. I think we can just expect users of mkFlake to leave those folders for now. In future, shell should prob be moved to lib and only the compat part should be in the root folder. And we could add a tests argument for mkFlake - that will require more tests rework, so not planning to include here.

231: Move flake implementation logic to lib r=blaggacao a=Pacman99 This is a simpler version of #218 that moves flake logic to lib and adds a module to evaluate devos. This DOES NOT support out of tree usage, so if you were following any of the previous PR's, the doc sections/examples to use devos as a library will not work. There is work to make a cleaner api and only then will out of tree support work. Until then, this is still useful to simplify devos and clean up a lot of the implementation logic. Co-authored-by: Pacman99 <pachum99@gmail.com>

231: Move flake implementation logic to lib r=nrdxp a=Pacman99 This is a simpler version of #218 that moves flake logic to lib and adds a module to evaluate devos. This DOES NOT support out of tree usage, so if you were following any of the previous PR's, the doc sections/examples to use devos as a library will not work. There is work to make a cleaner api and only then will out of tree support work. Until then, this is still useful to simplify devos and clean up a lot of the implementation logic. Co-authored-by: Pacman99 <pachum99@gmail.com>

Pacman99 · 2021-04-10T15:54:58Z

mostly implemented in #231

Pacman99 force-pushed the mkflake branch from b2bf93c to ea4f08d Compare March 26, 2021 20:03

Pacman99 requested review from blaggacao and nrdxp March 26, 2021 20:20

Pacman99 force-pushed the mkflake branch from ea4f08d to 43d8878 Compare March 27, 2021 04:53

Pacman99 commented Mar 27, 2021

View reviewed changes

Pacman99 force-pushed the mkflake branch 3 times, most recently from 37cf3da to afefcd4 Compare March 27, 2021 05:00

Pacman99 force-pushed the mkflake branch 2 times, most recently from 95f34a3 to 22dc867 Compare March 27, 2021 05:29

nrdxp mentioned this pull request Mar 27, 2021

Drop flattenTreeSystem and use custom function for filtering packages #220

Merged

Pacman99 mentioned this pull request Mar 28, 2021

Move host and suite implementation logic to lib #216

Merged

Pacman99 force-pushed the mkflake branch from 22dc867 to 7e4df95 Compare March 29, 2021 17:39

Pacman99 commented Mar 29, 2021

View reviewed changes

Pacman99 force-pushed the mkflake branch 5 times, most recently from a7bbd57 to ba81137 Compare March 30, 2021 18:14

Pacman99 commented Mar 30, 2021

View reviewed changes

lib/devos/mkProfileAttrs.nix Outdated Show resolved Hide resolved

Pacman99 force-pushed the mkflake branch from cde5eeb to 6c162e9 Compare March 30, 2021 19:58

Pacman99 commented Apr 1, 2021

View reviewed changes

Pacman99 mentioned this pull request Apr 1, 2021

mkDevos and evalDevosArgs: Move flake implementation logic into lib #206

Closed

Pacman99 added 9 commits April 1, 2021 13:47

mkHosts: get devos arg for core and cachix

919cab9

lib: init evalFlakeArgs for mkFlake args

b01f5d1

module describing devos flake arguments

lib: init mkFlake to create a devos flake

09ca1b6

general lib function - not devos specific

flake: use mkFlake

3e7b183

this is where we create devos's template structure

pkgs: add devosOptionsDoc - devos specific

2161153

doc/start: add section for using mkflake

c30cf5b

doc: init mkFlakeOptions.md

a868ed1

generated mkFlake options documentation

mkFlake: also pass self.inputs to lib

683c606

mkHosts: try fix with toString

541bf89

Pacman99 force-pushed the mkflake branch from 1cccd39 to 541bf89 Compare April 1, 2021 20:50

bors bot added a commit that referenced this pull request Apr 1, 2021

Try #218:

5171c64

David Arnold and others added 8 commits April 1, 2021 21:10

mkFlake: do not export soon-to-be devoslib on each repo

9e51359

It is not generally useful for everybody to export devos-lib. We can reintroduce a lib-exportin interface, once we conceputualize the private library sharing use case properly.

cleanup: ironing out flake code

fac0869

currently, there is no need for a recursiveUpdate, and since flake requires flattened trees, there might never be a good enough reason to use it at the expense of introducing a dependency on lib.

flake.nix: recursiveUpdate is unecessary here

2c348a6

lib: self is self is self

f68b94e

we should consequently avoid variable redefintions while traversing through layers and interfaces. self is a flake native term and refers to a handle to the flake repo itself.

Merge branch 'mkflake' into da/mkFlake/simplify-2

3e576ed

Merge pull request #5 from blaggacao/da/mkFlake/simplify-2

fe7b0c7

A series of "clenups"

This was referenced Apr 3, 2021

drop lib's self dep and pass data to each function #228

Closed

Move flake implementation logic to lib #231

Merged

Pacman99 closed this Apr 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lib: init and use mkFlake and evalFlakeArgs #218

lib: init and use mkFlake and evalFlakeArgs #218

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

nrdxp commented Mar 26, 2021 •

edited

Loading

Pacman99 Mar 27, 2021

Pacman99 commented Mar 27, 2021

Pacman99 Mar 29, 2021 •

edited

Loading

Pacman99 Mar 29, 2021

nrdxp commented Mar 30, 2021

Pacman99 commented Mar 30, 2021

Pacman99 Mar 30, 2021

Pacman99 Apr 1, 2021

Pacman99 Apr 1, 2021

Pacman99 commented Apr 1, 2021

bors bot commented Apr 1, 2021

Pacman99 commented Apr 3, 2021 •

edited

Loading

Pacman99 commented Apr 10, 2021

lib: init and use mkFlake and evalFlakeArgs #218

lib: init and use mkFlake and evalFlakeArgs #218

Conversation

Pacman99 commented Mar 26, 2021 • edited Loading

Pacman99 commented Mar 26, 2021 • edited Loading

Pacman99 commented Mar 26, 2021 • edited Loading

Pacman99 commented Mar 26, 2021 • edited Loading

nrdxp commented Mar 26, 2021 • edited Loading

Pacman99 Mar 27, 2021

Choose a reason for hiding this comment

Pacman99 commented Mar 27, 2021

Pacman99 Mar 29, 2021 • edited Loading

Choose a reason for hiding this comment

Pacman99 Mar 29, 2021

Choose a reason for hiding this comment

nrdxp commented Mar 30, 2021

Pacman99 commented Mar 30, 2021

Pacman99 Mar 30, 2021

Choose a reason for hiding this comment

Pacman99 Apr 1, 2021

Choose a reason for hiding this comment

Pacman99 Apr 1, 2021

Choose a reason for hiding this comment

Pacman99 commented Apr 1, 2021

bors bot commented Apr 1, 2021

try

Pacman99 commented Apr 3, 2021 • edited Loading

Pacman99 commented Apr 10, 2021

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

Pacman99 commented Mar 26, 2021 •

edited

Loading

nrdxp commented Mar 26, 2021 •

edited

Loading

Pacman99 Mar 29, 2021 •

edited

Loading

Pacman99 commented Apr 3, 2021 •

edited

Loading