TaxBrain and inctax.py/Tax-Calculator results are not same in 0.7.3

[martinholmer]

After TaxBrain 0.7.3 started using an up-to-date version of the puf.csv file, issue #1119 was resolved and closed. Since then there has been more testing of the TaxBrain file upload capability, most of which has been positive.

For example, static analysis of the reform in file0.json --- which is complicated enough that it cannot be represented using the parameter-value-interface to TaxBrain --- produces the same 2022 results when file0.json is uploaded to TaxBrain running in the cloud and when file0.json is supplied to the inctax.py command-line interface to Tax-Calculator on a local computer.

And when the policy reform in file0.json is analyzed assuming a behavioral response, the TaxBrain and inctax.py results are the same as shown in file1.json.

And when the policy reform in file0.json is analyzed assuming a consumption response (but no behavioral response), the TaxBrain and inctax.py results are the same as shown in file2.json. Notice that the results generated by file2.json and the static file0.json are exactly the same. This makes sense because the consumption response affects only the marginal tax rates which in turn affect behavioral responses, which are not present in file2.json.

Full agreement thus far.

But when we combine the behavioral response in file1.json with the consumption response in file2.json, we have file3.json, which does not produce the same results when it is uploaded to TaxBrain and when it is used by inctax.py.

The differences in aggregate tax revenue results shown in file3.json suggest that TaxBrain and inctax.py are not using the Consumption object in the same way. The only place where the Consumption class is used by inctax.py is in these few lines of the IncomeTaxIO class constructor. I've looked briefly at the dropq.py code but cannot figure out exactly what is being done there. @talumbau can you see a difference in how the Consumption class is being used by TaxBrain and by the IncomeTaxIO class?

@MattHJensen @feenberg @Amy-Xu @andersonfrailey @GoFroggyRun @PeterDSteinberg

[talumbau]

I am looking at this now. I tried some naïve scripting that does what I think is done in inctax.py but I got the same answer as in dropq. Can you tell me the exact way to run the IncomeTaxIO code with file3.json so I can track down the difference?

[martinholmer]

@talumbau, Thanks for looking into issue #1133.

Can you tell me the exact way to run the IncomeTaxIO code with file3.json so I can track down the difference?

Well, I guess, you could do this two ways:

(1) From the system command line along the lines of the comment at the top of file3.json. If you are in the top-level tax-calculator directory with the puf.csv and file3.json files copied there, you can do this:

python inctax.py puf.csv 2022 --blowup --weights --reform file3.json

(2) From a custom Python script that uses the IncomeTaxIO class like it is used in the inctax.py file.

[talumbau]

OK, very good. After I execute the line python inctax.py puf.csv 2022 --blowup --weights --reform file3.json I get a 39 MB file named puf-22.out-inctax-file3 in my current working directory. What do I do to compare this output to running file3.json with TaxBrain?

[martinholmer]

@talumbau said:

OK, very good. After I execute the line python inctax.py puf.csv 2022 --blowup --weights --reform file3.json I get a 39 MB file named puf-22.out-inctax-file3 in my current working directory. What do I do to compare this output to running file3.json with TaxBrain?

Read the comments at the top of file3.json where it gives the following replication instructions:

// 2022 RESULTS from taxcalc-inctax and taxbrain-upload version 0.7.3
// INCOME TAX ($B)   1659.23            1,650.9        <-- diff
// PAYROLL TAX ($B)  1478.04            1,475.2        <-- diff
// taxcalc-inctax results from:
//   python ../../inctax.py puf.csv 2022 --blowup --weights --reform file3.json
//   awk '{r+=$4*$29}END{print r*1e-9}' puf-22.out-inctax-file3
//   awk '{r+=$6*$29}END{print r*1e-9}' puf-22.out-inctax-file3
// taxbrain-upload results from:
//   http://www.ospc.org/taxbrain/9467/

[talumbau]

@martinholmer thanks for that info. I don't have a definite answer, but could you look at this gist:

https://gist.github.com/talumbau/cdf100eb84a8be8c57ce5789be6de9ab

This is a script that exactly matches what TaxBrain does for the year 2022 when given a start year of 2017 and the reform file3.json. If there is a mistake in that script, then it means we must fix something in dropq. To me it looks like it's doing the same thing as what happens in the IncomeTaxIO code.

[talumbau]

@martinholmer I have discovered the issue: it is whether or not you apply the Consumption parameters to the baseline Calculator. In dropq/TaxBrain, we do apply the Consumption parameters. In IncomeTaxIO, you do not apply the Consumption parameters to the baseline calculation. In the gist I shared in the link above, if you comment out line 25, you will get the IncomeTaxIO values of:

ITAXES($b)= 1659.213
PTAXES($b)= 1478.032

if you run the script as-is, you get the TaxBrain values of:

ITAXES($b)= 1650.894
PTAXES($b)= 1475.236

The reason I applied the Consumption parameters to the baseline is due to the answer I got to this question:

ospc-org/ospc.org#433

So, is there a disagreement on how to handle the Consumption parameters?

cc @MattHJensen

[martinholmer]

@talumbau said:

I have discovered the issue: it is whether or not you apply the Consumption parameters to the baseline Calculator.

Yes that is the source of the differences. Thanks for pointing this out.

[talumbau]

Sure. Keeping this open until PR is merged that unifies handling of Consumption parameters in dropq and in IncomeTaxIO.

[martinholmer]

Issue #1133 has been resolved by pull request #1136, which fixes a bug in the IncomeTaxIO class.
@talumbau, Thanks so much for all your help in diagnosing this issue.