"Buffer dtype mismatch" when calling difference_table()

[MaxGhenis]

Copying from PSLmodels/taxdata#141:

The below error arises when calling difference_table, but the code still seems to run. Could be related to the pandas update.

For example, this happens in cell 7 of this recipe01 notebook:

diff_table = calc1.difference_table(calc2, tax_to_diff='iitax')

---

ValueError Traceback (most recent call last)
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'

Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored

[MaxGhenis]

I also get this when calling distribution_tables. I'm in Python 2.7.

[martinholmer]

@MaxGhenis said in Tax-Calculator issue #1799:

The below error arises when calling difference_table <in my personal notebook version of Cookbook recipe01.py](https://github.com/MaxGhenis/taxcalc-notebooks/blob/master/recipes/recipe01.ipynb)>, but the code still seems to run.

@MaxGhenis, Tax-Calculator is open-source so you can use the source code as you please, like taking the Tax-Calculator cookbook recipe01.py script and making it into your own personal notebook.
But when you did that, apparently you didn't read recipe01, which says this:

This is an advanced recipe that should be followed only after mastering the basic recipe.

And the basic recipe says this:

When following the recipe, you will get several instances of the same 
ignored error message from deep inside the Pandas library that is being
used by Tax-Calculator. After conferring with the Pandas developers, 
our expectation is these error messages will go away when we upgrade
to Pandas version 0.22.0, which is scheduled to be released in January
2018, and which fixes a bug in the Pandas library. Meanwhile, the error
messages are annoying but harmless.

We are doing our best to document Tax-Calculator, but must rely on users reading the documentation. If you did read this documentation about this ignored Pandas error, please offer some suggestions about how we can make it clearer.

[MaxGhenis]

Yep, just saw that when making the recipe00 notebook. Was this a recent change though? It didn't show up when I first made the recipe01 notebook.

Would it make sense to keep this as a tracking issue open until the upstream pandas issue is resolved? My first instinct when seeing errors like these is to search open issues for an explanation, especially if it's been a bit since reading the documentation (and if it's changed).

[MaxGhenis]

Is there an open issue on the pandas side for this? I only see closed ones regarding this error message in https://github.com/pandas-dev/pandas.

[martinholmer]

@MaxGhenis asked:

Is there an open issue on the pandas side for this? I only see closed ones regarding this error message in https://github.com/pandas-dev/pandas.

I think the Pandas developers have closed this issue because the fix is scheduled for inclusion in Pandas version 0.22.0.

[martinholmer]

This issue did not get fixed with the upgrade to pandas 0.22.0, so I have raised an issue (link to 19037 is above) with the pandas developers. Perhaps I'm using the pandas library incorrectly, but if so I don't understand why pandas is ignoring the error message it is generating.

Now the pandas developers are saying that this pandas bug fix was not included in 0.22.0 as originally planned, but will be in included in release 0.23.0, which is scheduled to be available early in February 2018.

[martinholmer]

@martinholmer said in issue #1779 on January 2, 2018:

Now the pandas developers are saying that this pandas bug fix was not included in 0.22.0 as originally planned, but will be in included in release 0.23.0, which is scheduled to be available early in February 2018.

Now the pandas developers seem to be planning release 0.23.0 for early April 2018.

[martinholmer]

@martinholmer said in issue #1779 on January 2, 2018:

Now the pandas developers seem to be planning release 0.23.0 for early April 2018.

Now the pandas developer say they are planning release 0.23.0 for early May 2018.

[martinholmer]

Issue #1779 has been resolved by the refactoring of the logic that creates decile details in the distribution and difference tables. This refactoring was done in pull request #1972 for the distribution table and in pull request #1973 for the difference table.