Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Acceptance test time improvement #142

Closed
4 of 6 tasks
jantman opened this issue Oct 29, 2017 · 3 comments
Closed
4 of 6 tasks

Acceptance test time improvement #142

jantman opened this issue Oct 29, 2017 · 3 comments
Assignees
@jantman

Comments

@jantman
Copy link
Owner

jantman commented Oct 29, 2017 •
edited
Loading

With #141 and the move from PhantomJS to Chrome, acceptance test runtime has gone up from 17-18 minutes to 19-21 minutes. Look into what we can do to speed up acceptance test runs.

While the pytest --durations=10 report shows actual tests - usually either data creation or web interaction - taking the longest, there are probably a number of things we can do to try and improve overall performance:

  • Many of the test classes utilize the class_refresh_db and/or refreshdb fixtures; some of them even empty the database at the beginning via reflect(), drop_all() and create_all(). See if we can generate raw SQL to do this (empty db, create models, load test data) and if that would be faster.
  • See if we can modify acceptance tests to either not require refreshing the DB, or at least add data instead of emptying the DB and then re-adding.
  • make sure there aren't any uses of time.sleep()
  • investigate all uses of webdriver waits
  • use incremental testing to fail subsequent steps in an acceptance class if a previous step failed
  • ???
jantman added a commit that referenced this issue Oct 31, 2017
@jantman
issue #142 - WIP restore DB via mysqldump instead of SQLAlchemy
91999a5
@jantman jantman self-assigned this Oct 31, 2017
jantman added a commit that referenced this issue Oct 31, 2017
@jantman
issue #142 - simplify session-scope refreshdb fixture for fewer conne…
5745afb 
…ctions and faster setup
jantman added a commit that referenced this issue Oct 31, 2017
@jantman
issue #142 - we should never need metadata.reflect()
3c9b380
jantman added a commit that referenced this issue Oct 31, 2017
@jantman
issue #142 - record top 50 step durations instead of top 10
405acb7
jantman added a commit that referenced this issue Oct 31, 2017
@jantman
issue #142 - instead of using ORM to drop & recreate, use SQL dump
fd39789
jantman added a commit that referenced this issue Nov 1, 2017
@jantman
issue #142 - for no-data dumps, need to remove the AUTO_INCREMENT set…
98163e8 
…ters
jantman added a commit that referenced this issue Nov 1, 2017
@jantman
issue #142 - log all acceptance test durations to JSON
bc07efb
@jantman
Copy link
Owner Author

jantman commented Nov 2, 2017 •
edited
Loading

WIP acceptance test cleanup:

  • Next, pull the test timing and show totals by class and module; figure out which are the worst
  • test_transactions
    • TestTransModalCantEditReconciled - combine with another class
    • TestTransModalByURL - doesnt need class refresh
    • TestTransReconciledModal - doesnt need class refresh
    • TestTransAddModal - combine
    • TestTransAddStandingBudgetModal - combine?
  • test_scheduled
    • TestSchedTransModalPerPeriod - combine, no class refresh
    • TestSchedTransMonthlyURL - combine, no class refresh
    • TestSchedTransDateInactive - combine?
    • TestSchedTransAddDate - combine?
    • TestSchedTransAddMonthly - combine?
    • TestSchedTransAddPerPeriod - combine?
    • TestSchedTransAddIncome - combine?
  • test_reconcile - LOTS of clean DB and then recreate data...
  • test_payperiods
    • TestPayPeriodsIndex - clean and then recreate
    • TestPayPeriodOtherPeriodInfo - clean and recreate
    • TestCurrentPayPeriod - clean and recreate
  • test_ofx
    • TestOfxApi - combine
    • TestTransReconciledModal - combine; doesnt need class refresh
    • Many others - should be able to combine all of the classes that get the same page, and use an initial method to do the GET
  • test_fuel
    • TestVehicleModal and TestFuelLogModal - combine
  • are the TravisCI caches not working? Or not working in tox? Seems like install takes a long time...

jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - use @pytest.mark.incremental to short-circuit long-runni…
fbdd00b 
…ng acceptance tests with failed steps
jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - combine some test classes that could be combined; remove…
0619bb5 
… class_refresh_db from some classes that don't alter DB data
jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - track how long we spend in class_refresh_db() mysql calls
c62c4df
jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - fix some acceptance tests broken in last commit
aff74ee
jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - fix acceptance tests broken by previous commit
f023f0a
jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - add dev/test_timings.py to analyze acceptance test durat…
1fa9f90 
…ions
@jantman
Copy link
Owner Author

jantman commented Nov 2, 2017

1fa9f90 added dev/test_timings.py to analyze the captured durations from pytest. Example output:

$ dev/test_timings.py -v
2017-11-02 17:46:28,140 INFO:root:Found latest finished build: 400 (296454724)
2017-11-02 17:46:28,952 INFO:root:Found acceptance test job: 400.7
=> Build job 296454731 ran in 0:18:13
=> pytest durations total: 0:15:56.019711
=> class_refresh_db fixture total: 0:00:25.989817
        Called 43 times
        Mean runtime: 0.6044143410616143
        Median runtime: 0.21680808067321777
        Variance: 1.4477528706136569
=> pytest durations by setup/call/teardown
        call: 0:08:20.547312
        setup: 0:06:48.979753
        teardown: 0:00:46.492646
=> pytest durations by module
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py: 0:02:37.849538
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py: 0:01:42.681520
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py: 0:01:35.554050
        biweeklybudget/tests/acceptance/flaskapp/views/test_budgets.py: 0:01:33.447763
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py: 0:01:33.118256
        biweeklybudget/tests/acceptance/flaskapp/views/test_accounts.py: 0:01:25.085695
        biweeklybudget/tests/acceptance/flaskapp/views/test_fuel.py: 0:01:24.917764
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py: 0:01:12.771286
        biweeklybudget/tests/acceptance/flaskapp/views/test_credit_payoffs.py: 0:00:58.276282
        biweeklybudget/tests/acceptance/flaskapp/views/test_projects.py: 0:00:47.839118
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py: 0:00:30.744728
        biweeklybudget/tests/acceptance/flaskapp/views/test_index.py: 0:00:26.778105
        biweeklybudget/tests/acceptance/test_0sane_test_settings.py: 0:00:02.906490
        biweeklybudget/tests/acceptance/test_biweeklypayperiod.py: 0:00:02.618173
        biweeklybudget/tests/acceptance/test_prime_rate.py: 0:00:00.463998
        biweeklybudget/tests/acceptance/test_db_event_handlers.py: 0:00:00.388745
        biweeklybudget/tests/acceptance/models/test_ofx_transaction.py: 0:00:00.296082
        biweeklybudget/tests/acceptance/test_interest.py: 0:00:00.282117
=> pytest durations by class
        biweeklybudget/tests/acceptance/flaskapp/views/test_accounts.py::TestAccountModal: 0:01:09.357752
        biweeklybudget/tests/acceptance/flaskapp/views/test_budgets.py::TestBudgetModals: 0:01:04.068929
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestOFXDefault: 0:00:46.348180
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestBudgetTransfer: 0:00:36.902245
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestCurrentPayPeriod: 0:00:35.142606
        biweeklybudget/tests/acceptance/flaskapp/views/test_credit_payoffs.py::TestSettings: 0:00:34.636633
        biweeklybudget/tests/acceptance/flaskapp/views/test_fuel.py::TestFuelLogModal: 0:00:32.939244
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransactionsDefault: 0:00:32.054667
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestSkipScheduled: 0:00:28.372039
        biweeklybudget/tests/acceptance/flaskapp/views/test_fuel.py::TestFuelLogView: 0:00:26.999992
        biweeklybudget/tests/acceptance/flaskapp/views/test_projects.py::TestOneProjectView: 0:00:22.871223
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransDefault: 0:00:20.520253
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestDragLimitations: 0:00:19.826154
        biweeklybudget/tests/acceptance/flaskapp/views/test_projects.py::TestProjectsView: 0:00:18.520719
        biweeklybudget/tests/acceptance/flaskapp/views/test_credit_payoffs.py::TestNoSettings: 0:00:16.847542
        biweeklybudget/tests/acceptance/flaskapp/views/test_fuel.py::TestVehicleModal: 0:00:16.074324
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransAddStandingBudgetModal: 0:00:15.801562
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransDateInactive: 0:00:15.271971
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransModalPerPeriod: 0:00:14.939062
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestTransReconcileNoOfx: 0:00:14.754323
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestAccountReconcileFalse: 0:00:14.165782
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransAddIncome: 0:00:14.069301
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestFindPayPeriod: 0:00:12.868361
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestMakeTransModal: 0:00:12.479706
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransModal: 0:00:12.420546
        biweeklybudget/tests/acceptance/flaskapp/views/test_index.py::TestIndexAccounts: 0:00:11.879935
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransAddDate: 0:00:11.567405
        biweeklybudget/tests/acceptance/flaskapp/views/test_budgets.py::TestBudgets: 0:00:10.734269
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py::TestPPOverBalanceNotification: 0:00:10.268394
        biweeklybudget/tests/acceptance/flaskapp/views/test_accounts.py::TestAccountsMainPage: 0:00:09.721784
        biweeklybudget/tests/acceptance/flaskapp/views/test_budgets.py::TestBudgetTransfer: 0:00:09.431924
        biweeklybudget/tests/acceptance/flaskapp/views/test_budgets.py::TestBudgetTransferStoP: 0:00:09.212641
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestPayPeriodsIndex: 0:00:08.956760
        biweeklybudget/tests/acceptance/flaskapp/views/test_fuel.py::TestFuel: 0:00:08.904205
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestDragAndDropReconcile: 0:00:08.625677
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransAddPerPeriod: 0:00:08.577186
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestUIReconcileMulti: 0:00:08.512365
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransModalCantEditReconciled: 0:00:08.350971
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestTransReconciledModal: 0:00:08.069399
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransAddModal: 0:00:07.935376
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransAddMonthly: 0:00:07.663389
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py::TestBaseTemplateNavigation: 0:00:07.552445
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransReconciledModal: 0:00:07.499367
        biweeklybudget/tests/acceptance/flaskapp/views/test_index.py::TestIndexNavigation: 0:00:07.325367
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransactions: 0:00:07.281564
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestPayPeriods: 0:00:06.869724
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestOFXMakeTrans: 0:00:06.848360
        biweeklybudget/tests/acceptance/flaskapp/views/test_credit_payoffs.py::TestCreditPayoffs: 0:00:06.792108
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestOFXTransModal: 0:00:06.670011
        biweeklybudget/tests/acceptance/flaskapp/views/test_projects.py::TestProjects: 0:00:06.447177
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTrans: 0:00:06.401202
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestOFX: 0:00:06.208883
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestPayPeriodFor: 0:00:06.099276
        biweeklybudget/tests/acceptance/flaskapp/views/test_accounts.py::TestAccountsNavigation: 0:00:06.006159
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestPayPeriod: 0:00:05.997827
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestReconcile: 0:00:05.802304
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestColumns: 0:00:05.659051
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestOFXTransURL: 0:00:05.167574
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py::TestBaseTemplateNotifications: 0:00:05.095544
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestTransactionEditModal: 0:00:04.981933
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py::TestBaseTmplUnreconciledNotification: 0:00:04.947049
        biweeklybudget/tests/acceptance/flaskapp/views/test_index.py::TestIndexPayPeriods: 0:00:04.830656
        biweeklybudget/tests/acceptance/flaskapp/views/test_transactions.py::TestTransModalByURL: 0:00:04.209997
        biweeklybudget/tests/acceptance/flaskapp/views/test_payperiods.py::TestPayPeriodOtherPeriodInfo: 0:00:04.160994
        biweeklybudget/tests/acceptance/flaskapp/views/test_reconcile.py::TestReconcileBackend: 0:00:03.942308
        biweeklybudget/tests/acceptance/flaskapp/views/test_scheduled.py::TestSchedTransMonthlyURL: 0:00:03.671751
        biweeklybudget/tests/acceptance/test_0sane_test_settings.py::Test00AcceptanceSaneSettings: 0:00:02.906490
        biweeklybudget/tests/acceptance/flaskapp/views/test_base_template.py::TestBudgetOverBalanceNotification: 0:00:02.881295
        biweeklybudget/tests/acceptance/flaskapp/views/test_index.py::TestIndexBudgets: 0:00:02.742147
        biweeklybudget/tests/acceptance/test_biweeklypayperiod.py::TestSchedTransOrderingAndPeriodAssignment: 0:00:01.637532
        biweeklybudget/tests/acceptance/test_biweeklypayperiod.py::TestSums: 0:00:00.627913
        biweeklybudget/tests/acceptance/test_prime_rate.py::TestPrimeRateCalculator: 0:00:00.463998
        biweeklybudget/tests/acceptance/test_db_event_handlers.py::TestTransStandingBudgetBalanceUpdate: 0:00:00.388745
        biweeklybudget/tests/acceptance/test_biweeklypayperiod.py::TestTransFromSchedTrans: 0:00:00.352728
        biweeklybudget/tests/acceptance/flaskapp/views/test_ofx.py::TestOfxApi: 0:00:00.307239
        biweeklybudget/tests/acceptance/models/test_ofx_transaction.py::TestFirstStatementByDate: 0:00:00.296082
        biweeklybudget/tests/acceptance/test_interest.py::TestInterestHelper: 0:00:00.282117

jantman added a commit that referenced this issue Nov 2, 2017
@jantman
issue #142 - more acceptance test optimization
3772dd9
jantman added a commit that referenced this issue Nov 3, 2017
@jantman
issue #142 - changelog
f3aea66
@jantman jantman mentioned this issue Nov 3, 2017
Merged
@jantman
Copy link
Owner Author

jantman commented Nov 3, 2017 •
edited
Loading

I've done as much as I reasonably can for this issue; test time is down to ~15-17 minutes. The big win would be speeding up the selenium tests, as they're currently standing up Chrome/selenium, GETing a page, and tearing it down, for every test method that does browser tests.

I only see 3 ways to do that, none of which sound terribly good:

  1. Figure out a way to run selenium/chromedriver/chrome as a persistent service during the test session, which comes with its own problems and flakiness, and would either really complicate the test infrastructure or require migrating TravisCI back off the fast containerized infrastructure to the VMs.
  2. Fork pytest-selenium into a version which uses class-scoped fixtures instead of module-scoped (which makes more sense the way the tests for this project are anyway). Or use something like this simple example.
  3. Look into using Sauce Labs (free for open source, up to 5 parallel VMs), BrowserStack (free for open source, but need to contact a human), TestingBot (same deal as SauceLabs), or CrossBrowserTesting (need to contact). But all of those are VM-based and geared towards many browsers, so they'd likely be slower, and would also require tunneling.

jantman added a commit that referenced this issue Nov 3, 2017
@jantman
Merge pull request #146 from /issues/142
7f5edba 
Issues/142 - Acceptance test time improvements
@jantman jantman closed this as completed Nov 12, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jantman jantman

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jantman