Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixing the occasional errors in tests #63

Merged
merged 3 commits into from
Oct 27, 2023

Conversation

evalott100
Copy link
Contributor

@evalott100 evalott100 commented Oct 24, 2023

A couple of tests fail occasionally:

  • test_create_softioc_system sometimes cagets SEQ.TABLE fields part way through the panda-side update of SEQ.TABLE. To fix this, the caget has been moved to the start of the test and more time has been added before SEQ.TABLE is updated in the mocked panda.
  • test_table_column_info would fail because
    • the test process was too fast: pva would try to fetch values before panda introspection
    • the mocked panda process was too fast: by the time the pva context had been updated the mocked panda would have altered its table
    • To fix the former we used a pva monitor instead of a caget, timing out at TIMEOUT, to fix the latter we used a different mocked panda which doesn't update its tables. We've also changed to the asyncio p4p client.
  • During testing for the above two tests, another test failed: test_non_defined_seq_table_can_be_added_to_panda_side this was because very rarely the panda subprocess was too fast and had already changed SEQ3 to have values.
    • To fix this we'll give the panda 50 updates before it changes the value of SEQ3 rather than 10.

Tested locally with

for i in {1..50}; do echo $i; pytest | grep failed; done

to ensure no failures,

tested on the CI 10 times.

@evalott100 evalott100 self-assigned this Oct 24, 2023
@evalott100 evalott100 marked this pull request as draft October 24, 2023 11:48
@codecov
Copy link

codecov bot commented Oct 24, 2023

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (eba34d5) 86.29% compared to head (532d964) 86.29%.

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #63   +/-   ##
=======================================
  Coverage   86.29%   86.29%           
=======================================
  Files           7        7           
  Lines        1138     1138           
  Branches      187      187           
=======================================
  Hits          982      982           
  Misses        121      121           
  Partials       35       35           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@evalott100 evalott100 marked this pull request as ready for review October 24, 2023 12:58
Copy link
Contributor

@AlexanderWells-diamond AlexanderWells-diamond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there's better ways to do these fixes

tests/test_pvaccess.py Outdated Show resolved Hide resolved
tests/test_ioc_system.py Outdated Show resolved Hide resolved
@evalott100
Copy link
Contributor Author

evalott100 commented Oct 24, 2023

Strange, somehow the table packing and unpacking have found their way back into this repo... I will look into this.

@evalott100
Copy link
Contributor Author

For some reason TablePacking had been reintroduced. My last push removed it again (we moved them to the client pandablocks.utils which is where all the ioc code currently gets table_to_words and words_to_table from. I'm unsure of where they were reintroduced from.

@AlexanderWells-diamond
Copy link
Contributor

As best as I can work out, TablePacking was never deleted in any PR that was merged.

We stopped using it in #15 , which was merged.

However, there also exists both #14 and #17 , both of which did delete the code but neither of them were merged. There's no notes to say why that may be.

It would be clearer if we don't delete that code in this PR - it's an entirely unrelated change to fixing the tests.
(And while we're on the subject of neatness, can you please rewrite your commits to conform to the standard 50-character header line and 72 character body line. There's a decent commit message guide here)

@evalott100 evalott100 force-pushed the make_tests_more_robust branch 3 times, most recently from dcb689a to a9736d4 Compare October 25, 2023 07:46
@evalott100
Copy link
Contributor Author

evalott100 commented Oct 25, 2023

@AlexanderWells-diamond

I've cherry-picked the TablePacking to another branch and formatted the git commits.

Docs is failing due to the warning

WARNING: The default value for `navigation_with_keys` will change to `False` in the next release. If you wish to preserve the old behavior for your site, set `navigation_with_keys=True` in the `html_theme_options` dict in your `conf.py` file.Be aware that `navigation_with_keys = True` has negative accessibility implications:https://github.com/pydata/pydata-sphinx-theme/issues/1492

this is occurring on main too. We should fix this on another branch (pr here).

Increased the amount of time in before SEQ.TABLE is updated panda-side
in a test, changed test_create_softioc_system so that it checks the
SEQ.TABLE fields first.
Changed tests to use mocked pandas which don't internally change
table values throughout the tests.
…side

test_non_defined_seq_table_can_be_added_to_panda_side will rarely update the
panda in the subprocess before the test process has started. To fix this
the panda will wait 50 updates before it changes SEQ3 rather than 10.
Copy link
Contributor

@AlexanderWells-diamond AlexanderWells-diamond left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@evalott100 evalott100 merged commit bdca62d into PandABlocks:main Oct 27, 2023
10 checks passed
@evalott100 evalott100 deleted the make_tests_more_robust branch October 27, 2023 10:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants