Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: disable flaky tests on windows #4101

Merged
merged 1 commit into from
Jan 21, 2022

Conversation

lukasholzer
Copy link
Collaborator

@lukasholzer lukasholzer commented Jan 21, 2022

🎉 Thanks for submitting a pull request! 🎉

Summary

Disable flaky tests on windows:

These two where always failing and blocking
https://github.com/netlify/cli/runs/4896037137?check_suite_focus=true

I curated a list of flaky tests and those two are failing more often
https://www.notion.so/netlify/Flaky-CLI-Test-1d125bd140da4d1dbdb05ca02113d46d (internal link)

For us to review and ship your PR efficiently, please perform the following steps:

  • Open a bug/issue before writing your code 🧑‍💻. This ensures we can discuss the changes and get feedback from everyone that should be involved. If you`re fixing a typo or something that`s on fire 🔥 (e.g. incident related), you can skip this step.
  • Read the contribution guidelines 📖. This ensures your code follows our style guide and
    passes our tests.
  • Update or add tests (if any source code was changed or added) 🧪
  • Update or add documentation (if features were changed or added) 📝
  • Make sure the status checks below are successful ✅

A picture of a cute animal (not mandatory, but encouraged)

@lukasholzer lukasholzer added the type: chore work needed to keep the product and development running smoothly label Jan 21, 2022
@github-actions
Copy link

📊 Benchmark results

Comparing with 5ba83d8

Package size: 357 MB

⬇️ 0.00% decrease vs. 5ba83d8

^  355 MB  355 MB  355 MB  355 MB  355 MB  355 MB  355 MB  354 MB  354 MB  354 MB  354 MB  354 MB  357 MB 
│   ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐    ┌──┐  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
│   |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |  |    |▒▒|  
└───┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴────┴──┴──>
    T-12    T-11    T-10    T-9     T-8     T-7     T-6     T-5     T-4     T-3     T-2     T-1      T    
Legend

@lukasholzer
Copy link
Collaborator Author

FYI @XhmikosR

@lukasholzer lukasholzer merged commit 2c28f21 into main Jan 21, 2022
@lukasholzer lukasholzer deleted the disable-flaky-tests-on-windows branch January 21, 2022 13:11
@XhmikosR
Copy link
Contributor

Doesn't seem to fix all failures :/ https://github.com/netlify/cli/runs/4898013668?check_suite_focus=true

There must be a way to circumvent the affected tests script BTW. In this PR only a few tests ran :)

@lukasholzer
Copy link
Collaborator Author

I forgot this one updated already a pr: #4102

@lukasholzer
Copy link
Collaborator Author

@XhmikosR why should we find a way around the affected test script? this was good that only this test was running as no others where affected by the changset.

The problem is when the lockfile or the package.json is touched all tests are run as it can potentially influence all (which causes often troubles as running only few of them is not causing issues only the majority of running in paralell is causing fs issues especially on windows)

@XhmikosR
Copy link
Contributor

Because in this case you couldn't test everything due to the nature of that script :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: chore work needed to keep the product and development running smoothly
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants