GNU tests `factor` regressions #3171

tertsdiepraam · 2022-02-21T09:41:07Z

On a recent PR unrelated to factor, a lot of factor tests are failing

Error: GNU test failed: tests/factor/t01. tests/factor/t01 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t03. tests/factor/t03 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t04. tests/factor/t04 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t06. tests/factor/t06 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t08. tests/factor/t08 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t11. tests/factor/t11 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t15. tests/factor/t15 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t16. tests/factor/t16 is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/factor/t20. tests/factor/t20 is passing on 'main'. Maybe you have to rebase?

Same happened here.

The text was updated successfully, but these errors were encountered:

sylvestre · 2022-02-21T09:50:08Z

The question is also if they were working before or not ? :)
cc @nbraud

tertsdiepraam · 2022-02-21T10:17:39Z

They seem to pass on main: https://uutils.github.io/coreutils-docs/user/test_coverage.html Is there something else going on?

rivy · 2022-02-21T14:10:00Z

Maybe related to recent CI changes that I made to support "debug" builds for GnuTest coverage?

I'll try to take a closer look at these logs.

Intermittent CI bugs, ah joy... 😕

rivy · 2022-02-21T22:10:28Z

Yep, the factor tests are failing with...

2022-02-21T04:40:23.6941999Z FAIL: tests/factor/t11
2022-02-21T04:40:23.6942064Z ======================
2022-02-21T04:40:23.6942069Z 
2022-02-21T04:40:23.6942196Z sha1sum: failed to open file: No such file or directory
2022-02-21T04:40:23.6942319Z factor: warning: 18446744073708551700: Broken pipe (os error 32)
... literally 100000+ lines of "Broken pipe (os error 32)" ...

My recent changes caused some sort of breakage within the tests themselves.
It looks like they are all broken, some were just broken for other reasons prior...
I'm digging further into it.

rivy · 2022-02-22T22:50:13Z

So, after looking at this, I think we should just disable all the dynamically generated factor tests within GNU.
They are just testing for factor output over a given range of inputs. I don't think they add anything we can't or aren't already doing with in-house tests. We already test in-house (admittedly, for smaller ranges) with much better precision and robustness.

And we've already cut out half of the dynamic GNU tests for taking too long to test; and, now, in "debug" builds, the rest are taking even more annoyingly long amounts of time to complete. That has lead to partitioning some tests for "release" only, complicating the build script for the GNU tests. And hacking the test scripts directly with the build script is proving too fragile.

Disabling them should stabilize the test oscillations. 🤞🏻

I've got a commit set with the change running in my local fork which looks more stable.

It will cut the TOTAL number to 590 and PASS to 215 (to GNU tests summary = TOTAL: 590 / PASS: 215 / FAIL: 291 / ERROR: 10 from approximately GNU tests summary = TOTAL: 611 / PASS: 234 / FAIL: 293 / ERROR: 10).

We can judiciously move them back in at a later date if/when desired (although I'm not sure that we need them).

@sylvestre , @tertsdiepraam , thoughts?

tertsdiepraam · 2022-02-22T22:58:37Z

Do you know why they fail exactly? I'm fine with disabling them if the failures don't indicate any regressions, because that makes them pretty much useless anyway.

rivy · 2022-02-22T23:15:42Z

From what I see in the logs, the tests are failing due to broken pipes to a failed sha1sum (sha1sum: failed to open file: No such file or directory) not regressions of factor, although I'm not sure why the sha1sum is failing. I'm trying to fully duplicate on an Ubuntu VM, but it's not proceeding quickly.

I'm almost 100% certain that it's not a factor regression because the same tests all pass in the first non-debug iteration of GnuTests. The small remaining doubt is that maybe the "debug" build is introducing or uncovering some flaw that's otherwise hidden, but, again, it looks like a sha1sum problem not a factor problem.

tertsdiepraam · 2022-02-22T23:36:16Z

Interesting, I've been seeing more broken pipe errors in the CI lately, for example with those flaky ls tests recently. I wonder if that's related.

jfinkels · 2022-03-24T00:53:55Z

This seems to still be happening frequently. It's kind of annoying because it creates a lot of spurious build failure notifications. See https://github.com/uutils/coreutils/runs/5637336986?check_suite_focus=true for a recent example.

sylvestre · 2022-03-24T07:18:02Z

So, after looking at this, I think we should just disable all the

dynamically generated factor tests within GNU.

I've got a commit set with the change running in my local fork which looks more stable.

Agreed do you still have this patch ?:) thanks S

…

Message ID: ***@***.***>

rivy · 2022-03-24T10:32:12Z

Sure.
I'll add a PR this weekend.

rivy · 2022-03-27T22:04:15Z

I've got a PR almost done. I'm looking at two errors before I post it...

Error: GNU test failed: tests/df/df-symlink. tests/df/df-symlink is passing on 'master'. Maybe you have to rebase?
Error: GNU test failed: tests/ls/readdir-mountpoint-inode. tests/ls/readdir-mountpoint-inode is passing on 'master'. Maybe you have to rebase?
Error: Process completed with exit code 255.

rivy · 2022-03-27T22:48:39Z

df looks like a real problem.

From my Ubuntu server:

$ git log -1 | cat
commit d5d6bc30504b3161a932e8372d8c5508661ebad8
Merge: c93223682 ab717ce37
Author: Sylvestre Ledru <sylvestre@debian.org>
Date:   Sun Mar 27 23:41:50 2022 +0200

    Merge pull request #3294 from jfinkels/df-file-column

    df: implement the File column
$ cargo build --release --features unix
 ...
    Finished release [optimized] target(s) in 49.25s
$ ./target/release/coreutils df
Filesystem          1K-blocks         Used    Available  Use% Mounted on
udev                  1966968            0      1966968    0% /dev
tmpfs                  402616         1144       401472    1% /run
/dev/vda1            81120644     55683576     25420684   69% /
tmpfs                 2013076            0      2013076    0% /dev/shm
tmpfs                    5120            0         5120    0% /run/lock
tmpfs                 2013076            0      2013076    0% /sys/fs/cgroup
/dev/sdb             51475068       957772     47879472    2% /mnt/volume_nyc1_01
/dev/vda15             106858         6443       100414    7% /boot/efi
/dev/sda            259946948    175562520     71260844   68% /mnt/volume_nyc1_02
/dev/loop2              56960        56960            0  100% /snap/core18/2284
/dev/loop5              96896        96896            0  100% /snap/go/9028
tmpfs                  402612            0       402612    0% /run/user/0
tmpfs                  402612            0       402612    0% /run/user/1000
/dev/loop9             113536       113536            0  100% /snap/core/12725
/dev/loop0              63488        63488            0  100% /snap/core20/1361
/dev/loop7              69632        69632            0  100% /snap/lxd/22526
/dev/loop1              63488        63488            0  100% /snap/core20/1376
/dev/loop3              96896        96896            0  100% /snap/go/9360
/dev/loop4             113408       113408            0  100% /snap/core/12821
/dev/loop10             56960        56960            0  100% /snap/core18/2344
/dev/loop6              69504        69504            0  100% /snap/lxd/22753
$ ./target/release/coreutils df /dev/vda1
Filesystem          1K-blocks         Used    Available  Use% Mounted on
udev                  1966968            0      1966968    0% /dev
$ df
Filesystem                              Size  Used Avail Use% Mounted on
udev                                    1.9G     0  1.9G   0% /dev
tmpfs                                   394M  1.2M  393M   1% /run
/dev/vda1                                78G   54G   25G  69% /
tmpfs                                   2.0G     0  2.0G   0% /dev/shm
tmpfs                                   5.0M     0  5.0M   0% /run/lock
tmpfs                                   2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sdb                                 50G  936M   46G   2% /mnt/volume_nyc1_01
/dev/vda15                              105M  6.3M   99M   7% /boot/efi
/dev/sda                                248G  168G   68G  72% /mnt/volume_nyc1_02
/dev/loop2                               56M   56M     0 100% /snap/core18/2284
/dev/loop5                               95M   95M     0 100% /snap/go/9028
tmpfs                                   394M     0  394M   0% /run/user/0
tmpfs                                   394M     0  394M   0% /run/user/1000
/dev/loop9                              111M  111M     0 100% /snap/core/12725
/dev/loop0                               62M   62M     0 100% /snap/core20/1361
/dev/loop7                               68M   68M     0 100% /snap/lxd/22526
/dev/loop1                               62M   62M     0 100% /snap/core20/1376
/dev/loop3                               95M   95M     0 100% /snap/go/9360
/dev/loop4                              111M  111M     0 100% /snap/core/12821
/dev/loop10                              56M   56M     0 100% /snap/core18/2344
/dev/loop6                               68M   68M     0 100% /snap/lxd/22753
$ df /dev/vda1
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        78G   54G   25G  69% /
$

jfinkels · 2022-03-27T23:01:49Z

Can you say more about what the problem is? I see a couple of things:

whitespace issues: df: incorrect whitespace between columns in output table #3194
human-readable size versus block counts; maybe your df is aliased to df -h?
wrong filesystem when passing a device as an argument: df: df doesn't check if the input path is a filesystem #3246

Here are some other known issues: https://github.com/uutils/coreutils/issues?q=is%3Aopen+is%3Aissue+label%3A%22U+-+df%22

The tests/df/df-symlink.sh test case should have been resolved in #3161

rivy · 2022-03-27T23:37:31Z

[ Whoops! I initially edited your message instead of quoting it. 😨 ]

wrong filesystem when passing a device as an argument: df: df doesn't check if the input path is a filesystem #3246

I think this might be issue causing failure for tests/df/df-symlink.sh. Specifically, this...

$ file_system=$(df --out=source . | tail -n1)
$ echo $file_system
/dev/vda1
$ df --out=source,target $file_system
Filesystem     Mounted on
/dev/vda1      /
$ target/release/coreutils df --out=source,target $file_system
Filesystem       Mounted on
udev             /dev

because tests/df/df-symlink.sh is using...

file_system=$(df --out=source '.' | tail -n1) ||
  skip_ "cannot determine '.' file system"

ln -s "$file_system" symlink || framework_failure_

df --out=source,target "$file_system" > exp ||
  skip_ "cannot get info for $file_system"
df --out=source,target symlink > out || fail=1
compare exp out || fail=1

which causes a failure.

The tests/df/df-symlink.sh test case should have been resolved in #3161

Hmm, I'm using the most recent commit (d5d6bc3) which includes that merged commit.

rivy · 2022-03-27T23:42:30Z

I think the gnu/tests/ls/readdir-mountpoint-inode.sh failure is also a PR-unrelated true test failure.

When I dig into the script...

# dir='/dev' ; readdir_inode=12 (via `ls ...`); stat_inode=2
$ ls -i -I '.?*' -I 'dev?*' -I 'de[^v]*' -I '??' -I 'd[^e]*' -I '?' -I '[^d]*' '/'
2 dev/
$ target/release/coreutils ls -i -I '.?*' -I 'dev?*' -I 'de[^v]*' -I '??' -I 'd[^e]*' -I '?' -I '[^d]*' '/'
     12 bin      1619 home      3809 lib64          3811 media        1 proc     3821 sbin        1 sys    60476 var
    180 boot     1622 lib    6709249 libx32         3812 mnt       3815 root     4041 snap     4044 tmp
    206 etc   6193153 lib32       11 lost+found  9805825 opt          2 run      4042 srv      4045 usr

jfinkels · 2022-03-28T02:07:02Z

The tests/df/df-symlink.sh test case should have been resolved in #3161

Hmm, I'm using the most recent commit (d5d6bc3) which includes that merged commit.

I must have been mistaken. Anyway, yes it looks like #3246 is the issue to fix.

rivy · 2022-03-30T14:09:43Z

I've got a working commit change set with 'GnuTests' changes including disabling the t* factor tests.
I'll post the PR later today assuming that the CI run passes/fails correctly.

jfinkels added the U - factor label Feb 26, 2022

sylvestre added the gnu testsuite label Mar 11, 2022

tertsdiepraam mentioned this issue Mar 23, 2022

tests for factor are failing frequently in CI build on unrelated changes #3298

Closed

rivy self-assigned this Mar 27, 2022

sylvestre mentioned this issue Mar 29, 2022

[Meta bug] GNU testsuite #3336

Closed

10 tasks

rivy closed this as completed Apr 2, 2022

rivy removed their assignment Apr 2, 2022

sylvestre added this to GNU Compatibility Aug 5, 2022

sylvestre moved this to Done in GNU Compatibility Aug 18, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GNU tests `factor` regressions #3171

GNU tests `factor` regressions #3171

tertsdiepraam commented Feb 21, 2022 •

edited

Loading

sylvestre commented Feb 21, 2022

tertsdiepraam commented Feb 21, 2022

rivy commented Feb 21, 2022 •

edited

Loading

rivy commented Feb 21, 2022

rivy commented Feb 22, 2022 •

edited

Loading

tertsdiepraam commented Feb 22, 2022

rivy commented Feb 22, 2022 •

edited

Loading

tertsdiepraam commented Feb 22, 2022 •

edited

Loading

jfinkels commented Mar 24, 2022

sylvestre commented Mar 24, 2022 via email

rivy commented Mar 24, 2022

rivy commented Mar 27, 2022 •

edited

Loading

rivy commented Mar 27, 2022

jfinkels commented Mar 27, 2022 •

edited by rivy

Loading

rivy commented Mar 27, 2022 •

edited

Loading

rivy commented Mar 27, 2022 •

edited

Loading

jfinkels commented Mar 28, 2022

rivy commented Mar 30, 2022

GNU tests factor regressions #3171

GNU tests factor regressions #3171

Comments

tertsdiepraam commented Feb 21, 2022 • edited Loading

sylvestre commented Feb 21, 2022

tertsdiepraam commented Feb 21, 2022

rivy commented Feb 21, 2022 • edited Loading

rivy commented Feb 21, 2022

rivy commented Feb 22, 2022 • edited Loading

tertsdiepraam commented Feb 22, 2022

rivy commented Feb 22, 2022 • edited Loading

tertsdiepraam commented Feb 22, 2022 • edited Loading

jfinkels commented Mar 24, 2022

sylvestre commented Mar 24, 2022 via email

rivy commented Mar 24, 2022

rivy commented Mar 27, 2022 • edited Loading

rivy commented Mar 27, 2022

jfinkels commented Mar 27, 2022 • edited by rivy Loading

rivy commented Mar 27, 2022 • edited Loading

rivy commented Mar 27, 2022 • edited Loading

jfinkels commented Mar 28, 2022

rivy commented Mar 30, 2022

GNU tests `factor` regressions #3171

GNU tests `factor` regressions #3171

tertsdiepraam commented Feb 21, 2022 •

edited

Loading

rivy commented Feb 21, 2022 •

edited

Loading

rivy commented Feb 22, 2022 •

edited

Loading

rivy commented Feb 22, 2022 •

edited

Loading

tertsdiepraam commented Feb 22, 2022 •

edited

Loading

rivy commented Mar 27, 2022 •

edited

Loading

jfinkels commented Mar 27, 2022 •

edited by rivy

Loading

rivy commented Mar 27, 2022 •

edited

Loading

rivy commented Mar 27, 2022 •

edited

Loading