grep 2.27 broken

[fpqc]

I've been using an arch userspace for a while, and I had been able to build tmux and weechat from git in the past, but it seems like the new versions of grep (and maybe cmake) are causing failures while building tmux and weechat from git source.

Both tmux and weechat are failing to build from git, because cmake is spitting out garbage that looks like:

-- Found GCRYPT: grep: (standard input): Operation not permitted

Moreover, when I try to strace cmake, I get the following error

strace: PTRACE_SETOPTIONS: Invalid argument

which then locks up and can't be killed with ctrl-c (not because of 15002, it also happened in 14986), so
I'm having trouble pinning down exactly what operation is not permitted.

I made a big tarball of my arch rootfs, and I'm willing to host it on my own server for the devs, but I would like permission first, and instructions on who to e-mail a password and username to grab a copy. You should be able to just un-tar and chroot into it to test building weechat from git (git clone the repo, cd into the cloned repo, mkdir build, cd build, cmake ..).

EDIT: Most recent post has narrowed this down to just two things. Looks like it's not a glibc problem

`

[therealkenc]

A whole Arch chroot is a pretty big repro nut. Can you get grep to fail in your Arch environment with Operation not permitted on its lonesome? Like say: echo hello | grep hello?

Also what version does strace -V give you? That might be notable, because if Arch strace is using a new PTRACE_SETOPTIONS it'll eventually hit a Ubuntu release too.

[therealkenc]

Okay; best I can tell, strace broke with this change here on Nov 26, 2016, which first appeared in version 4.15 on Dec 14.

That PTRACE_O_TRACEEXIT ptrace(2) option was introduced in Linux 2.5.60 in 2003, but strace didn't use it (AFAICT) until a couple of months ago. There could be secondary effects causing the eventual fail, but that's the crux. Not related to Arch of course. 4.15 fails on WSL Xenial all the same.

4.14 (Oct 4, 2016) is good though. You can track down your cmake/grep (or possibly neither) related issue by running that version. On Arch do:

git clone https://github.com/strace/strace.git
cd strace
git checkout v4.14
./bootstrap
./configure
make
sudo make install

...then /usr/local/bin/strace -f <whatever>. Though, if Arch bleeds so bad as to be using a December strace, you could be hitting any number of things there (like glibc).

[fpqc]

Yep. I filed precisely because these changes came from up-upstream and will be making it into ubuntu one day or another. Going to test this all in a couple of hours, just got up to get some water 😴 😪

Edit: Evidently I didn't go back to sleep. Anyway, I was offering the whole chroot bc installing pritunl/archlinux from the docker hub uses a fixed snapshot of the arch repos and will never update.

[fpqc]

@therealkenc Yep, it's strace 4.15. I rolled back to the 4.14 I had in the pacman cache and now am able to get a nice big strace off. Also going to see what happens after doing the trace if I roll back to the previous version of grep as well.

strace of cmake for weechat:
strace.txt

Edit: Also, I just rolled grep back to 2.26 from 2.27, and now cmake is working again.

@russalex Looks like you're going to hit problems with grep 2.27 and strace 4.15. Both 2.26 and 4.14 worked fine.

[therealkenc]

[pid  1513] fstat(0, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid  1513] lseek(0, 0, SEEK_CUR)       = -1 ESPIPE (Illegal seek)
[pid  1513] read(0, "\n", 32768)        = 1
[pid  1513] splice(0, NULL, 1, NULL, 32768, SPLICE_F_MOVE) = -1 EPERM (Operation not permitted)

Doubtful that is coming from grep proper. Probably gnulib (sic), but that's only an educated guess.

[fpqc]

@therealkenc I rolled grep back to 2.26 and it works fine again, so it's definitely one of the files in the grep pkg.tar.xz stuff, not gnulib or whatever. Maybe it's making use of a new feature in gnulib, or is gnulib statically linked?

Those packages only seem to contain grep and its executable symlink aliases fgrep and egrep, some manpages, and grep.info

greps.zip

So if it's a library thing, it seems weird that it would work with 2.26 but not 2.27 (I did an in-place rollback without fixing any dependencies). I mean, unless you're saying that grep is statically linking gnulib or making use of a versioned API or something?

[therealkenc]

Dunno. I just grabbed 2.27.36 from the source. It seems to work in isolation. Academic. The problem is that splice(2) from your dump, which I don't think has been reported before.

[fpqc]

@therealkenc Hmm. Curiouser and curiouser. Well, does the 2.27 binary I just posted work for you?

Also, when I tried compiling it now, make worked alright, but make check failed (using the arch PKGBUILD from their repos).

[therealkenc]

Any binaries from Arch are going to use shared libs I don't have. They don't run.

Anyway, turns out it is grep proper after all. Change on Nov 19th here. Because grep, a program whose functionality hasn't materially changed since the 1980s, needs a Linux-specific system call in 2016. For performance. Or something.

[fpqc]

@therealkenc Also, when I tried building grep from the source, it seems that m4/autom4ke is giving the same weird kind of error (when I ran autoconf):

/usr/bin/m4: m4_esyscmd subprocess failed: Operation not permitted
/usr/bin/m4:configure.ac:22: cannot run command `build-aux/git-version-gen .tarball-version': Operation not permitted

autoconf.strace.txt

This looks like it is giving that clone() issue you reported a while ago (in the strace).

[pid  3245] clone(child_stack=0x7f2506d80ff0, flags=CLONE_VM|CLONE_VFORK|SIGCHLD) = -1 EINVAL (Invalid argument)

Also, another one of these things:

[pid  3245] lseek(0, 0, SEEK_CUR)       = -1 ESPIPE (Illegal seek)

It looks like the GNU project did a big update that broke a lot of stuff (for WSL) in December.

[therealkenc]

Yeah looks like it might be the lseek(), not necessarily (or possibly in addition to) the splice(). I've seen that lots of times in WSL before; for example (random pick) here. Usually it's a soft fail though. And yes, everyone on the bleeding edge is going to faceplant into #1005, which is why I speculated glibc up-thread.

So you have three "actionable and reproducible issues" here.

• ptrace() PTRACE_O_TRACEEXIT fails with EINVAL (edit: this is PTRACE_O_TRACEEXIT, PTRACE_O_TRACEEXEC #555)
• splice(0, ..., SPLICE_F_MOVE) fails with EPERM
• lseek(0, 0, SEEK_CUR) fails with ESPIPE

Vote for me on User Voice...

[fpqc]

@therealkenc Sure, could you link the uservoice things?

[therealkenc]

Don't know if they exist (likely not). Or you can re-task this issue to "Upgrade Yakety to ZestyZapus" similar to #482.

[fpqc]

I'll ping @stehufntdev and @benhillis since these are all bugs/missing features that will be coming up specifically in developer tools (m4, grep, and strace) in the next few months.

(Devs, my offer of a rootfs tarball ready to go is still available!)

[benhillis]

Thanks for bringing this to our attention. I know @dethoma is aware of this and will assign somebody to investigate. I agree with @fpqc and @therealkenc that this is should be a high-priority issue since we do not have control over when Ubuntu may decide to update these packages.

[therealkenc]

Not having control over Ubuntu releases is a tough situation; though, not really different than Ubuntu having no control over their upstream. Xenial won't end-of-life until 2019, and you'd be hard pressed to find a specific credible "development use case" that requires Yakety or ZestyZapus right now or in the next few months. Plenty of "high priority" stuff (for some value of "high") still broken in Trusty/Precise/Wily on WSL too (where grep works fine). OTOH, heading off the shower of duplicate issues and commensurate telemetry hits as people do-release-upgrade -d definitely has its own appeal.

[stehufntdev]

@fpqc thanks for the offer of the tarball, appreciate the help here. We should be able to repro this locally but will hit you up if we end up needing the tarball.

[fpqc]

@stehufntdev Also one last program to look at is the newest version of bison. I also noticed that was breaking some of the things. No trace for it though.

[JasonLinMS]

@therealkenc @fpqc Thank you for the investigations, they made our job much easier!
For the grep issue, we have fixed the issue with splice() failing. The only test case that I saw failing was piping the output to /dev/null: ls | ./src/grep a > /dev/null, which now works after the fix. If you guys encounter any other failing test cases please let us know and we can try them out.

[fpqc]

@benhillis So since the grep thing (with splice()) is fixed now as of 15025, and the other issues are being tracked ( #555 and #1005 ), should I close this?

[benhillis]

@fpqc - sounds good to me, thank you!

[fpqc]

I'm going to title change and close then. I know you're working on #1005 but has anyone taken a look at #555 yet (no comments on it).