Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move ffcv up in test order #1953

Merged
merged 1 commit into from
Feb 9, 2023
Merged

Conversation

dskhudia
Copy link
Contributor

@dskhudia dskhudia commented Feb 7, 2023

What does this PR do?

Randomize the order of pytest to catch errors that may work if the tests are run in the same order.

Move the test up in order so that it passes. Don't ask me why!!

@dskhudia dskhudia requested a review from a team as a code owner February 7, 2023 23:00
@mvpatel2000
Copy link
Contributor

Can we fix the ones failing rn that are deterministically failing first?

@dskhudia
Copy link
Contributor Author

dskhudia commented Feb 7, 2023

ffcv test is not deterministically failing for me. It passes on local run.

@dakinggg
Copy link
Contributor

dakinggg commented Feb 7, 2023

So, I agree with the sentiment of this change, but I'm a little worried that we may have a number of non deterministic issues and will spend a significant amount of time debugging them on unrelated PRs. Is there any way we can de risk that? Like, should someone spend a few days just trying different random orders and fixing anything that pops up?

@dskhudia
Copy link
Contributor Author

dskhudia commented Feb 7, 2023

@dakinggg we don't have to merge this now but eventually do something like this to make testing more robust. Currently I want to see if the test_ffcv failure goes away with this.

@dakinggg
Copy link
Contributor

dakinggg commented Feb 7, 2023

Sounds good 👍

@dskhudia
Copy link
Contributor Author

dskhudia commented Feb 8, 2023

non deterministic issues and will spend a significant amount of time debugging them on unrelated PRs.

BTW this is more one time effort than ongoing effort.

@dakinggg
Copy link
Contributor

dakinggg commented Feb 8, 2023

non deterministic issues and will spend a significant amount of time debugging them on unrelated PRs.

BTW this is more one time effort than ongoing effort.

Absolutely :)

@dskhudia
Copy link
Contributor Author

dskhudia commented Feb 8, 2023

Ignore any updates to this PR. This is just to fix the failing test.

@dskhudia dskhudia force-pushed the randomize_tests branch 3 times, most recently from e137b00 to 40c477f Compare February 9, 2023 06:33
@dskhudia dskhudia changed the title randomize order of pytests Move test up in order. Feb 9, 2023
@bandish-shah bandish-shah changed the title Move test up in order. Move failing ffcv up in test order Feb 9, 2023
@bandish-shah bandish-shah changed the title Move failing ffcv up in test order Move ffcv up in test order Feb 9, 2023
Copy link
Contributor

@bandish-shah bandish-shah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Super bizarre but LGTM.

@bandish-shah bandish-shah merged commit b577b7e into mosaicml:dev Feb 9, 2023
@dskhudia dskhudia deleted the randomize_tests branch February 9, 2023 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants