Lua/luarocks packaging improvements #63108
Conversation
Changes: - Fetches rocks and builds Nix expressions for them in parallel - Passes 'maintainers' list to luarocks-nix - Constructs the luarocks argument list more cleanly, by using an indexed array - Made indentation consistent
Summary of resulting package updates:
- bit32: init at 5.3.0-1 (same as current hand-written derivation)
- busted: 2.0.rc12-1 -> 2.0.rc13-0
- compat53: init at 0.7-1 (same as current hand-written derivation)
- cqueues: init at 20171014-0 (same as current hand-written derivation)
- cyrussasl: init at 1.1.0-1 (same as current hand-written derivation)
- lrexlib-pcre: init at 2.9.0-1 (vs 2.8.0 in current hand-written
lrexlib derivation)
- luadbi and backends (luadbi-{mysql,postgresql,sqlite3}): init at
0.7.2-1 (vs 0.7.1 in current hand-written derivation)
- luaexpat: init at 1.3.3-1 (vs 1.3.0 in current hand-written
derivation)
- luafilesystem: init at 1.7.0-2 (same as current hand-written
derivation)
- luaossl: init at 20190612-0 (vs 20181207 in current hand-written
derivation)
- luasec: init at 0.8-1 (same as current hand-written derivation)
- luasocket: init at 3.0rc1-2 (same as current hand-written derivation)
- luasql-sqlite3: init at 2.4.0-1 (vs 2.3.0 in current hand-written
luasqlite3 derivation)
- rapidjson: 0.5.1-1 -> 0.5.2-1
- stdlib: init at 41.2.2-1 (vs 41.2.1 in current hand-written
derivation)
The newer version silences some superfluous warnings we were previously getting in Nix builds using luarocks.
Summary of main changes: - Now makes use of luarocks dependency resolution (builds will fail if rockspec dependencies are unmet) - Renamed argument `external_deps` -> `exernalDeps` and add functionality to handle external dependencies that are multiple-output derivations - Added an `extraVariables` argument for appending to the contents of luarocks config `variables` table - The `rockspecFilename` argument default is now actually used - The `disabled` argument can now be overriden with a less-restrictive check, as it now just sets `meta.broken` instead of throwing an error during eval - The `doCheck` argument is now actually honored if set to `true`
Shados
force-pushed
the
lua-packaging-improvements-pr
branch
from
c31b759
to
dd85416
Compare
|
Force-push to fix an eval failure in a broken package -_-''. |
luaPackages replaced by generated ones:
- bit32
- compat53
- cqueues
- luacyrussasl -> cyrussasl (luarocks name)
- luaexpat
- luadbi -> luadbi front-end module + separate backend modules
luadbi-{mysql,postgresql,sqlite3}
- luafilesystem
- luaossl
- luasec
- luasocket
- luastdlib -> stdlib (luarocks name)
- lrexlib -> lrexlib-pcre (we already have lrexlib-gnu and
lrexlib-posix, lrexlib-pcre however appears to be the variant used in
mudlet, which is the only current dep in nixpkgs)
- luasqlite -> luasql-sqlite3 (luarocks name)
- lfs -> luafilesytem (we literally had two manually written
luafilesystem expressions, under different names)
Changes and additions to overrides to generated luarocks packgaes,
including:
- busted: Install bash completions along with the zsh ones
- cqueues:
- Perform minor surgery on the rockspec to allow using a single
rockspec to build for all supported Lua versions
- Add a patch by @vcunat to work around a build issue
- luuid: Wrote a tiny patch to allow for Lua 5.1/Luajit compatibility
- General changes:
- Sorted the packages
- Attempted to make the formatting consistent
- Preferenced `.override` instead of `.overrideAttrs` wherever
possible
Minor changes to other packages to adjust for the Lua package changes:
- luakit expression simplified
- prosody expression simplified; but users will now need to specify the
luadbi backend module they intend to use in withExtraLibs
- knot-resolver inputs correctd
- mudlet inputs corrected (although this package was and should still be
broken)
Shados
force-pushed
the
lua-packaging-improvements-pr
branch
from
dd85416
to
274715c
Compare
|
@teto please take a look, feel free to ping me on IRC if you want to discuss. |
There was a problem hiding this comment.
wow amazing, thanks a lot for doing this. I've had maybe half of the changes in a tree for a year but never managed to make it into PR material. You seem to have addressed many lingering issues which is great.
For reference:
- the documentation PR doc: update lua documentation #55302
- and this metaissue which I might be able to cross off some things thanks to your effort Meta issue about lua ecosystem progress #56398
|
@teto I've added a commit with those changes to |
|
I am all for merging but since it affects lots of packages, I would like some other commiter to run a nix-review on it (too hard on my current setup). |
|
I ran a nix-review on my small laptop torchPackages seemed to be broken beforehand, as for lua52Packages.ltermbox or digesti marked as broken, it's more like they should not be available for this interpreter. We might need to change that to make the messages clearer. Anyway it was there before the PR. Important packages seem to build fine even though it might be worth to test them. |
|
thanks a lot ! |
|
With this PR error work luadbi Variant with not worked |
|
@Izorkin I've opened a PR to fix that. In the mean-time, you could work-around it with e.g.: {
nixpkgs.overlays = [
(self: super: {
prosody = super.prosody.override {
withDBI = true;
withExtraLibs = [
(super.luaPackages.luadbi-mysql.override({
extraVariables = ''
MYSQL_INCDIR='${super.mysql.connector-c}/include/mysql';
MYSQL_LIBDIR='${super.mysql.connector-c}/lib/mysql';
'';
}))
];
};
})
];
} |
|
With this config - worked! |
|
Found new error in Prosody |
|
@Izorkin looks like this is an upstream issue. Based on some investigating, it looks like this is the current official (?) upstream for luaexpat, and is what most distros are using. The 1.3.0 tarball from there has the same SHA256 as the 1.3.0 tarball in the luaexpat-1.3.0-1 luarock, so I'll make a PR to downgrade to that for now. Longer term, we'll see how things go on the prosody issue tracker for it (I'm subscribed to it now). In the mean-time, you can override {
nixpkgs.overlays = [
(self: super: {
prosody = super.prosody.override {
luaexpat = pkgs.luaPackages.luaexpat.override({
version = "1.3.0-1";
src = pkgs.fetchurl {
url = https://luarocks.org/luaexpat-1.3.0-1.src.rock;
sha256 = "15jqz5q12i9zvjyagzwz2lrpzya64mih8v1hxwr0wl2gsjh86y5a";
};
});
};
})
];
}P.S. It would have been very useful to mention you wrote NixOS tests for prosody + mysql when you mentioned the luadbi-mysql issue :P. Could have saved me a bit of time writing a minimal one to test the fix. |
|
Thanks, worked. |
Matches version used on most distros. Fixes an issue with prosody. Detailed reasoning behind this can be found [here](NixOS#63108 (comment)).
Matches version used on most distros. Fixes an issue with prosody. Detailed reasoning behind this can be found [here](#63108 (comment)).
|
Merging this PR broke auto-setting of variables like |
|
thanks for bisecting. |
|
@vcunat As @teto pointed to, the issue is that Personally, I think I would prefer converging on the
Thoughts? *: Although the given example is a tad odd... |
Up to now the variables were set regardless of the patterns expanding to anything usable, apparently... so it worked for me to bisect the problem. Of course, more real test would be like 6fdd315 or more generally: |
|
@vcunat OK, no, I was confused. It definitely has Lua lib files, and no shared library objects. The variables are only set if the package creates the respective |
Motivation for this change
I got annoyed by some issues in the existing luarocks packaging infrastructure while trying to use it.
Things done
sandboxinnix.confon non-NixOS)nix-shell -p nix-review --run "nix-review wip"--dry-runand built the attrs separately for easier debugging./result/bin/)nix path-info -Sbefore and after)buildLuarocksPackagedocumentation, something this PR does not change ;).Details of the changes are given in the commit messages. Broadly:
update-luarocks-packagesscriptbuildLuarocksPackagesI've done some history editing to break up the commits into somewhat-logical groupings to make things easier to look through. Interim commits don't necessarily represent a working state for nixpkgs, though.
Some possible issues:
withDBIargument won't work on its own now, asluadbihas been split into the front-endluadbimodule and the back-endluadbi-mysql,luadbi-postgresql, andluadbi-sqlite3modules. I think it should be sufficient to build prosody with arguments likewithDBI = true; withExtraLibs = [ lua.pkgs.luadbi-sqlite3 ];, for example, but I don't have a prosody setup to test with.builtinbuild type, which is theoretically cross-platform. But I don't have a mac to test this assertion with.torchPackageswere broken before, and still are now. I spent some time and effort looking into getting them to build, but while most of it isn't too bad, torch itself and some of its dependent torch-specific packages do some fairly wonky things and make bad assumptions in their cmake files.luabinary and the install directory for torch are under a shared$PREFIX, and that it is OK to write to the system Lua share directory.