Working module/package system

[joelpurra]

The current package system is not a mess - this should prove things work. I'll read #672 soon.

See also #659 (comment). If you haven't tried jqnpm yet, please clone it right away. There is no need for me to try to convince you if you won't even try the POC of a working system.

[joelpurra]

In this POC I haven't fixed importing relative modules ./other-file.jq. Nor do I use $PACKAGEROOT/jq.json for package main file path lookups yet. But they're on their way,

[zopieux]

If this idea is to be implemented upstream one day, please consider choosing another name for the PACKAGEROOT env var. It is way too generic as it does not include JQ. Env var are messy enough by themselves, would be great if library developers could choose the least-conflicting names possible!

I would suggest JQPKGPATH or just JQPATH, because it's more like a search path than a root, isn't it?
You could reuse the PATH syntax used by plenty of well known programs, eg.:

export JQPATH="/usr/share/lib/jq;$HOME/.jqpkg"

[joelpurra]

If this idea is to be implemented upstream one day, please consider choosing another name for the PACKAGEROOT env var. It is way too generic as it does not include JQ.

@zopieux: No worries -- it's not an env var, even though it looks like one. It is used internally when resolving context-dependent paths; as is $ORIGIN. It is also mostly meant to be used internally in the default module search path.

That said, these strings could also be prefixed internally -- although I'm not sure it's necessary.

[zopieux]

My bad, thought that was part of the finding-lib-directories stuff.

[joelpurra]

@zopieux: it is part of finding lib dirs for the package system in this pull request -- unless we're accidentally talking about different things -- but $PACKAGEROOT is never resolved against any env vars.

[joelpurra]

Code comments by @nicowilliams were lost during rebase; here are the direct commit links:

joelpurra@42d20ef Find package root, look for packages under 'P/.jq/packages' (proof-of-concept)

joelpurra@1218024 Save if jq was started with -f (a filter file, a filter fifo file/pipe) or an inline filter

[joelpurra]

@nicowilliams: pushed updated code, hopefully addressing your comments.

Further things to work on:

1. Detect the path to the main jq file in a package through .main in jq.json (default is P/jq/main.jq).
2. Automatically use the main jq file if no jq -f code.jq or jq '.some.inline-filter' has been specified.
3. Test (or fix jqnpm tests for) relative module paths work (starting with ./, such as ./other.jq).

[joelpurra]

The default file path P/jq/main.jq is now used if no -f nor direct filter has been specified.

[nicowilliams]

You can see my fixes in the package-root branch of my fork of jq. Let me know what you think.

[joelpurra]

@nicowilliams: great! I'll look at nicowilliams@3cd687c and nicowilliams@2afc83a.

[joelpurra]

@nicowilliams: regarding comment in joelpurra@784fbc3 for "Use P/jq/main.jq if no -f and no direct filter"; this is the old packages/projects difference confusion.

Packages (only functions, only imported as modules) and projects (non-functions are allowed, can't be imported, are used directly) are very similar and they're treated the same by jqnpm in terms of installation/management. Guess projects can be seen as "entry-points" -- this is where you'd write code, while depending on (other people's) packages.

You won't execute jq inside a package, but inside a project. The use case is cat input.json | jq which would automatically execute all of the code in P/jq/main.jq with dependencies. As projects are "entry-points," it's inside a project folder your ordinary jq user would write and execute code.

"Use P/jq/main.jq if no -f and no direct filter" is not a critical feature and I'm open to comments. It's partially connected to the (current lack of) P/jq.json handling in jq, as users shouldn't have to type the .main path set in P/jq.json every time they want to execute the project.

This may be superseded by jqnpm install automatically creating/linking projects' executables to ~/.jq/bin (expected to be in $PATH) which would trigger the same functionality by pointing jq to P/.

#!/usr/bin/env jq --SOMEFLAG /path/to/P

[nicowilliams]

@joelpurra But deciding on such things based on $PWD is very different from just about all other interpreters, and to me, asking for much trouble and very wrong. I don't mind forcing users to use #!jq -f and I don't mind forcing users to use -f on the command-line. Why is using -f undesirable?

(The default program, when we have a tty, is now ., FYI.)

[joelpurra]

@nicowilliams: doesn't mean just about every other interpreter is right. I'll drop the commit though. It was meant to come later anyways, but it was easier to implement than reading jq.json from inside jq (in a smart way).

Edit: done.

[nicowilliams]

And BTW, the tests pass for me on Linux. There was an infinite loop in the package dir search code. You saw I removed all the logic for detecting that we've reached the root directory and replaced it with checking when dirname() no longer shortens current_path -- this is portable and safer, and outsources the dirty work to libc.

[joelpurra]

@nicowilliams: checking the length is much smarter than my solution, thanks! All that should be merged (already).

[nicowilliams]

Thanks. I'll do a final review the next couple of days.

[joelpurra]

@nicowilliams: the tests in jqnpm are more evolved (emulating remote git server and the local cache, etcetera), so please check them out. I'll keep the -f emulation in jqnpm execute for now.

I'll look at import relative to the currently parsed file (instead of the initial -f or $PWD) as well (see the jq-import-direct-file-paths test). Not sure I know how to access jq.json to get .main for a non-default P/jq/main.jq path from inside an executing jq though. Suggestions to both work items are welcome!

[nicowilliams]

@joelpurra Import relative to module doing the import is already supported.

As to the main.jq thing, for now, hardcode it, but if you can read a .json file from C using jv_load_file(). Though frankly I prefer that the main file be named after the module's name anyways.

[joelpurra]

@dtolnay: haven't rebased my jq fork since May, so it looks like this pull-request is a bit outdated. The last thing I did was incorporate @nicowilliams' suggested changes (#673 (comment)). Feature-wise this is a good start for a module system; I've also implemented jqnpm plus bunch of packages for both testing and actual use.

From the first comment (#673 (comment)):

In this POC I haven't fixed importing relative modules ./other-file.jq. Nor do I use $PACKAGEROOT/jq.json for package main file path lookups yet. But they're on their way,

@nicowilliams says (in the previous comment #673 (comment)) relative paths are supported, but I haven't confirmed in jqnpm myself. Barring any conflicts since I last ran the test suite in jqnpm this should be good to go. Please use the joelpurra/jqnpm@jq-package-root branch when testing this pull-request.

@dtolnay: you seem to be on a roll -- care to look at the merge conflicts in this pull request? C isn't my forte, but you seem fluent =)

By the way: gave core jq developers/contributors collaborator access to joelpurra/jqnpm, so as to not hold up v1.5 more than necessary -- go ahead and implement minor fixes.

[joelpurra]

@nicowilliams, @dtolnay: rebased and pushed. Please test, lint etcetera to make sure the C code is alright.

@pkoppstein, @slapresta, @wtlangford, @agordon, others interested: please test this pull request against joelpurra/jqnpm@jq-package-root.

Testing jqnpm on your machine with tests/all.sh (with $PATH set to your jq build and jqnpm clone) would be a good start, creating and publishing your own package would be wonderful!

[joelpurra]

Any news on this?

[nicowilliams]

Re-reviewing this and my fixups, I think the only part we should take is the .jq directory search (and the /packages/). The search should be up from the directory containing the calling jq program file (or $PWD if the program is not in a file) up to $HOME (if available) or the root.

I don't really like having to hard-code things like /proc/ and /dev/ for detecting ways to name a program file that isn't a regular file (but where stat() may say that the thing is a regular file anyways). Making users have to specify an option for saying "search from $PWD" seems fine to me.

[joelpurra]

Haven't updated my forked jq branch joelpurra:package-root since July 2015, so it's 100+ commits behind jq's HEAD (and I have not been tracking relevant/related changes). Either way, made it easier to brew install this particular --devel fork/branch to be able to use jqnpm more easily. Just an FYI.

brew tap joelpurra/joelpurra
brew unlink jqnpm
brew unlink jq
brew install jqnpm --devel

jq -f file/with/deep/imports.jq