Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Test reform post and results #627

Closed
wants to merge 13 commits into from

Conversation

hdoupe
Copy link
Collaborator

@hdoupe hdoupe commented Aug 18, 2017

This PR mainly addresses issue #615 but was brought up in #602 and #606. It adds a series of scripts that posts a reform file or a reform through the Taxbrain GUI on a local Taxbrain instance using the requests library and simultaneously runs the same reform directly using taxcalc.dropq.run_nth_year_tax_calc_model. The results from TaxBrain are retrieved from the local sqlite3 table taxbrain_taxsaveinputs and the results from submitting directly to taxcalc are formatted into the same structure as the TaxBrain results. These two results are then compared.

This effectively tests two parts of TaxBrain:

  1. The reform data is processed as expected by TaxBrain. If it is not, then the end results will not be the same.
  2. The process of sending the processed parameters and receiving the correct results from taxcalc works as expected.

Currently, it takes about 37 minutes to run the 7 tests (each runs for about 5 minutes). I think that the number of tests could be decreased. For example, I don't think the tests involving the Trump and RyanBrady plans test anything that is not tested using r1.json and the reform which posts all of the taxbrain parameters.

The TaxBrain GUI parameters are changed in unexpected ways such as being rounded, casted from floats to integers (1.0 to 1), and from boolean to binary (0-1). Since the dropq algorithm generates a random seed directly from the reform data, this unexpected data manipulation results in using different data sets to estimate the results from using the same reform. Thus, to test the TaxBrain GUI, I tried to compare the reform results within a 2% tolerance.

Further, it does not work when the entire test suite is run. I think that at some point in the test suite the puf file and maybe the requests functions are mocked. To run these tests run py.test webapp/apps/taxbrain/tests/reform_input_processing/ and make sure the server and supervisord are running in other terminals. To run the test suite without these tests run py.test webapp/apps/ -m "not requires_pufcsv

Initially, I hoped that this PR would be ready by the 1.0.2 release (#624). However, I don't think it will be ready in time. I think it could still be cleaned up a good bit and maybe some of the functions could be mocked.

cc @martinholmer @brittainhard

@hdoupe
Copy link
Collaborator Author

hdoupe commented Jan 17, 2018

Closing PR #627. See rationale in #615

@hdoupe hdoupe closed this Jan 17, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant