Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

split: implement round-robin arg to --number #3281

Merged
merged 1 commit into from
Oct 23, 2022

Conversation

jfinkels
Copy link
Collaborator

@jfinkels jfinkels commented Mar 20, 2022

Help wanted! This change was previously merged in pull request #3205. Unfortunately, it resulted in an issue with the GNU test suite; see issue #3268. The change was subsequently reverted in pull request #3269. I am re-creating this pull request hoping that someone can diagnose the issue with the GNU test case tests/split/filter.sh as described in issue #3269. That issue no longer seems to be present on this branch after I rebased.


Implement distributing lines of a file in a round-robin manner to a
specified number of chunks. For example,

$ (seq 1 10 | split -n r/3) && head -v xa[abc]
==> xaa <==
1
4
7
10

==> xab <==
2
5
8

==> xac <==
3
6
9

@jfinkels jfinkels changed the title split: implement round-robin arg to --number [do not merge] split: implement round-robin arg to --number Mar 20, 2022
@jfinkels jfinkels force-pushed the split-round-robin branch from 24a3056 to 82ed863 Compare May 5, 2022 22:14
@jfinkels
Copy link
Collaborator Author

jfinkels commented May 6, 2022

I rebased this branch to take another look at this issue.

The summary from the GNU test workflow is misleading. It says:

Error: GNU test failed: tests/split/filter. tests/split/filter is passing on 'main'. Maybe you have to rebase?
Error: GNU test error: tests/split/filter. tests/split/filter is passing on 'main'. Maybe you have to rebase?

but tests/split/filter.sh is not passing on the main branch, it is exiting with an error:

2022-05-05T23:38:24.9436982Z filter.sh: set-up failure: 
2022-05-05T23:38:24.9437113Z ERROR tests/split/filter.sh (exit status: 99)

(from recent build on main branch: https://github.com/uutils/coreutils/runs/6314524034?check_suite_focus=true )

I believe that this is another case where implementing a feature has caused one of the GNU tests to be able to run further than it had previously been able to run. Specifically, I believe that if we merge this, then there will be one less ERROR and one more FAIL in the GNU test suite. The issue we were previously seeing no longer seems to be present, so I think we should re-consider merging this new feature.

@jfinkels jfinkels changed the title [do not merge] split: implement round-robin arg to --number split: implement round-robin arg to --number May 6, 2022
@tertsdiepraam
Copy link
Member

tertsdiepraam commented May 6, 2022

If I understand correctly, we should change how we report the GNU changes? ERROR -> FAIL should probably be considered an improvement, instead of being reported as a problem.

@jfinkels
Copy link
Collaborator Author

jfinkels commented May 7, 2022

If I understand correctly, we should change how we report the GNU changes? ERROR -> FAIL should probably be considered an improvement, instead of being reported as a problem.

Yes, I believe that is correct. But I haven't looked at the code that summarizes the GNU test results so I'm not totally positive about that.

@jfinkels jfinkels force-pushed the split-round-robin branch 2 times, most recently from 3788401 to d118373 Compare May 18, 2022 22:06
@sylvestre sylvestre force-pushed the split-round-robin branch from d118373 to f033b25 Compare May 20, 2022 09:54
@jfinkels jfinkels force-pushed the split-round-robin branch 2 times, most recently from 26e1583 to ebe200b Compare May 26, 2022 23:45
@jfinkels
Copy link
Collaborator Author

I rebased this pull request.

Error: GNU test failed: tests/misc/timeout. tests/misc/timeout is passing on 'main'. Maybe you have to rebase?
Error: GNU test failed: tests/split/filter. tests/split/filter is passing on 'main'. Maybe you have to rebase?
Warning: Congrats! The gnu test tests/split/filter is no longer ERROR!
Error: Process completed with exit code 255.
  • As I mentioned in a previous comment, the tests/split/filter.sh test is in the ERROR state on main so the "is passing on 'main'" message is incorrect.
  • The tests/split/filter.sh test is still producing about one million lines of shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
  • For tests/misc/timeout.sh test, this could be failing because tests/split/filter.sh is spending about one minute printing those error messages? I'm not sure.

@sylvestre sylvestre force-pushed the split-round-robin branch from ebe200b to 2e50456 Compare June 1, 2022 21:13
@sylvestre sylvestre force-pushed the split-round-robin branch from 2e50456 to 4e7bca2 Compare June 16, 2022 11:20
@jfinkels jfinkels force-pushed the split-round-robin branch 2 times, most recently from c7f719c to 1df52a3 Compare September 6, 2022 04:25
@sylvestre
Copy link
Contributor

Fails with:


error[E0061]: this function takes 5 arguments but 4 arguments were supplied
Error:     --> src/uu/split/src/split.rs:1233:33
     |
1233 |       let mut filename_iterator = FilenameIterator::new(
     |  _________________________________^^^^^^^^^^^^^^^^^^^^^-
1234 | |         &settings.prefix,
1235 | |         &settings.additional_suffix,
1236 | |         settings.suffix_length,
1237 | |         settings.suffix_type,
1238 | |     );
     | |_____- an argument of type `usize` is missing

Copy link
Contributor

@sylvestre sylvestre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI broken, could you please have a look? sorry

Implement distributing lines of a file in a round-robin manner to a
specified number of chunks. For example,

    $ (seq 1 10 | split -n r/3) && head -v xa[abc]
    ==> xaa <==
    1
    4
    7
    10

    ==> xab <==
    2
    5
    8

    ==> xac <==
    3
    6
    9
@github-actions
Copy link

GNU testsuite comparison:

GNU test failed: tests/split/filter. tests/split/filter is passing on 'main'. Maybe you have to rebase?
Congrats! The gnu test tests/split/filter is no longer ERROR!

@sylvestre sylvestre merged commit c766726 into uutils:main Oct 23, 2022
@jfinkels
Copy link
Collaborator Author

@sylvestre The tests/split/filter.sh test is still producing about one million lines of shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory. So, maybe we should skip that part of the test for the moment? I believe it is this part:

# Ensure that "endless" input _is_ processed for unbounded number of filters    
for buf in 1000 1000000; do
  returns_ 124 timeout .5 sh -c \
    "yes | split --filter='head -c1 >/dev/null' -b $buf" || fail=1
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants