Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prep for a 1.5 release -- what else do we need? #657

Closed
nicowilliams opened this issue Dec 27, 2014 · 32 comments
Closed

Prep for a 1.5 release -- what else do we need? #657

nicowilliams opened this issue Dec 27, 2014 · 32 comments

Comments

@nicowilliams
Copy link
Contributor

Besides the issues and PRs marked for 1.5, what else must we add?

I'd really like to have some built-in modules that can be imported but which are compiled into the executable. In particular I'd like to have a) all of the normal jq builtins available as a module named "jq" that can be imported (and even versioned!) by calling modules, and b) builtin modules to start putting new builtins into and stop polluting the global namespace.

Ideally a program could even ask not to get the <1.4 pollution, except for very few, very special builtins, namely: all the byte-coded ones, like empty, not, path, and range/2, all the private C-coded builtins, and all the ones that parser.y codes or depends on, like recurse.

But I don't think I'll have time to put that together for 1.5. Instead I'd like to fix all the outstanding issues and make 1.6 a vehicle for module/namespace awesomeness.

So, what must still be done for 1.5?

@nicowilliams
Copy link
Contributor Author

Well, for one there's a PR from @pkoppstein I need to finish merging...

@nicowilliams
Copy link
Contributor Author

And, of course, I hope you're all having very happy holidays. Don't spend much time thinking about jq 1.5.

@nicowilliams
Copy link
Contributor Author

I think it's ready for rc1.

@pkoppstein
Copy link
Contributor

I seem constantly to be defining:

def count(stream): reduce stream as $i (0; . + 1);

Less often but nicely compact:

def count(stream; until):
  count( label | stream | if until then break else . end );

A random number generator would be very nice as well :-)

@nicowilliams
Copy link
Contributor Author

On Thu, Jan 01, 2015 at 10:26:43PM -0800, pkoppstein wrote:

I seem constantly to be defining:

def count(stream): reduce stream as $i (0; . + 1);

Yes, that's a good one. We already have length, why not length(exp)?

Though count is a good name for it too.

Less often but nicely compact:

def count(generator; until):
  count( label | generator | if until then break else . end );

Interesting. Could we do something like:

def upto(exp; upto): label | exp | if upto then break else . end;

then you could say:

length(upto(exp; ...))

Not every idiom needs a builtin, but many do!

@pkoppstein
Copy link
Contributor

why not length(exp)?

A little oblique.

def upto(;)

Yes, that would be handy, but then again, (1) it doesn't do much more than break; (2) "upto" isn't a word; and (3) "upto(x;y)" could reasonably be interpreted both as "up to and including y" and as "up to but excluding y". However, these are just small points to consider. Whatever you decide is fine by me, but I like the consistency and utility of count/1 and count/2.

@nicowilliams
Copy link
Contributor Author

@pkoppstein

Yes, length(exp) is oblique, but sticks to the existing convention. count(exp) is better, so I might add count/0 as an alias of length/0 too. I'll think about it.

I also thought of break_at(exp; cond), but it can't work because breaks are lexical now, not dynamic :)

Another possible name for upto (the fact that it isn't a word isn't worrisome to me) would be all_up_to, or truncate_at(exp; cond). I think that's going to be a common pattern, so if we can give it a name, we should (if we can't we shouldn't?). What if instead of counting the truncated pattern you wanted to collect all of the values in the truncated stream?

I've been writing a lot of if ... then break else . end lately, so I'm tempted to revive the optional-else discussion and allow if ... then ... end == if ... then ... else . end, which is different from what select(exp) does (which is if exp then . else empty end).

@pkoppstein
Copy link
Contributor

@nicowilliams wrote:

Another possible name for upto ...

Another thing I don't like so much about upto(stream; quit) is that the ordering of the arguments is grammatically awkward. (Contrast: until(cond; next).) How about trim(stream;quit) or a synonym such as truncate, prune, snip, cut, lop ... ? Or read on...

What if instead of counting the truncated pattern you wanted to collect all of the values in the truncated stream?

def tail(stream; when):
  foreach stream as $item
    (false;
     if ($item|when) then true else . end;
     if . then $item else empty end);

As for length/1 -- although I could easily get used to it, there's something to be said for choosing one name for arrays (length/0) and another for streams (count/1). It helps keep the two concepts (arrays and streams) well-separated psychologically; and linguistically, we count sheep entering a pen, as opposed to measuring the length of the pen.

@nicowilliams
Copy link
Contributor Author

OK, fair point.

@pkoppstein
Copy link
Contributor

Regarding count/1 and count/2, I'd like to propose an enhancement to count/2 as follows:

def count(stream): reduce stream as $i (0; .+1);

def count(stream; until):
  if (until|type) == "number" then count(limit(until; stream))
  else count( label | stream | if until then break else . end )
  end;

E.g. count( range(1;10000); 1) => 1

In other words, a little more mileage.

@pkoppstein
Copy link
Contributor

There is a tiny error in the NEWS file, which seems too small to warrant an entire "Issue" unto itself, but which maybe could be corrected even before 1.5 is released: jq --arg-file variable file should be jq --argfile variable file

@apennebaker-ni
Copy link

I just want sub/gsub, so a faster 1.5 release with fewer features would be great for my purposes.

@dtolnay dtolnay removed the misc label Jul 25, 2015
@dtolnay
Copy link
Member

dtolnay commented Jul 26, 2015

@nicowilliams gets back from a month of travel tomorrow. Let's give him a few days to catch up, then put out 1.5rc2. No later than next Saturday! Then the real 1.5 release no later than 2 weeks after rc2 unless we discover major issues that take longer to fix.

@nicowilliams
Copy link
Contributor Author

We should do a 1.5rc2 release ASAP. I think 8295baf is a good candidate. I can build executables for Linux and Windows, and can probably build for OS X or get Homebrew to do it. I'm sure so can @dtolnay and @wtlangford. I wonder if we could get Travis-CI to build Windows executables... It would really be best to let Travis do all the building of release executables.

@dtolnay
Copy link
Member

dtolnay commented Jul 27, 2015

I agree, let's do rc2 ASAP. Either 8295baf or 21309bc is fine.

I intend to look into using Vagrant or some other way to do reproducible builds for 1.6 (#410).

@nicowilliams
Copy link
Contributor Author

OK, I'll do the builds later tonight (dinner time now).

@dtolnay
Copy link
Member

dtolnay commented Jul 27, 2015

Thanks Nico!

@nicowilliams
Copy link
Contributor Author

Habemus 1.5rc2. (Though I need to re-update the executables and tarballs after pushing a fix for #870, but none of them should change at all.)

@dtolnay
Copy link
Member

dtolnay commented Jul 27, 2015

What does the development process look like in between 1.5rc2 and 1.5? There are some bugs I want to fix that should not be in 1.5 at this point.

I propose creating a release branch "release/jq-1.5" starting at the commit currently tagged as the RC. Critical fixes that need to be in 1.5 can be done on the release branch, and less critical things can be done on master for 1.6. We merge the release branch into master as needed. At the time of the 1.5 release, the head of the release branch is tagged as jq-1.5 and the release branch is deleted. In the ideal case (no critical changes required), the release branch never moves and jq-1.5rc2 and jq-1.5 are the same commit.

@nicowilliams
Copy link
Contributor Author

I was hoping we could keep linear development. I agree that if we need to port anything from master into 1.5 and if master has things we don't want in 1.5, then we should create a 1.5 branch. Since you have such commits, I've pushed a jq-1.5-branch branch that we can use to cherry-pick commits onto.

@nicowilliams
Copy link
Contributor Author

Also, we should push to master first, then cherry-pick onto jq-1.5-branch as needed.

@nicowilliams
Copy link
Contributor Author

The person pushing to master should decide whether to also cherry-pick onto and push to jq-1.5-branch.

@dtolnay
Copy link
Member

dtolnay commented Jul 27, 2015

Okay, sounds good.

@nicowilliams
Copy link
Contributor Author

FYI: Homebrew/legacy-homebrew#42152

@nicowilliams
Copy link
Contributor Author

Tomorrow I'll edit gh-pages to point to the 1.5rc2 release.

@nicowilliams
Copy link
Contributor Author

@dtolnay Do please review the 1.5rc2 release text and add anything I may have missed.

@dtolnay
Copy link
Member

dtolnay commented Aug 13, 2015

At this point we are good to go for 1.5, right? @nicowilliams do you have time and energy this week or this weekend to do the release?

@nicowilliams
Copy link
Contributor Author

I'll have time and energy late next week and weekend, and hopefully a bit this Saturday. I think I still need to check on the module system parts for @joelpurra.

@dtolnay
Copy link
Member

dtolnay commented Aug 14, 2015

Per this comment I think that is okay to do next milestone. The existing PR is a little too scary to put in at the last second.

@nicowilliams
Copy link
Contributor Author

So, the only part of @joelpurra's patches that I think we should consider for 1.5 at this point is the patch to search for a .jq directory, but even that seems a bit risky at this point, so I agree, let's punt that till 1.6.

That means we need only consider whether to fix the bugs found by @DRMacIver using AFL. Otherwise I can make a release this weekend.

@dtolnay
Copy link
Member

dtolnay commented Aug 14, 2015

The scariest one you already fixed and cherry-picked across (#896). I have a PR waiting for code review that fixes 2 more, and there is 1 left to fix. I think all 3 remaining ones should wait for 1.6.

@nicowilliams
Copy link
Contributor Author

Yeah, speaking of which, I need to review it!

@dtolnay dtolnay closed this as completed Aug 16, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants