Example of test error when using numpy 1.13.1

[martinholmer]

Below I show how updating from numpy 1.12.1 to numpy 1.13.1 causes an error in the decorator code. It does seem to be all about the code in decorators.py because when I turn off the decorator logic the error goes away. I don't know whether the decorator code needs to be changed or whether there is a bug in the new numpy 1.13.1 package or in another package (like numba) that relies on the new numpy 1.13.1.

Here is the detailed information where @pytest.mark.one is used to execute just the third test in the test_behavior.py file. First, I show the marked test passing when using numpy 1.12.1:

iMac2:tax-calculator mrh$ conda list | grep -e numpy -e pandas -e numba
numba                     0.34.0              np112py27_0  
numpy                     1.12.1                   py27_0  
numpydoc                  0.6.0                    py27_0  
pandas                    0.20.2              np112py27_0  

iMac2:tax-calculator mrh$ pushd taxcalc ; py.test -m one ; popd
============================= test session starts ==============================
platform darwin -- Python 2.7.13, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /Users/mrh/work/OSPC/tax-calculator, inifile: setup.cfg
collected 303 items 

tests/test_behavior.py .

============================= 302 tests deselected =============================
================== 1 passed, 302 deselected in 17.17 seconds ===================

Then I update to the newest numpy version and run the same test again.

iMac2:tax-calculator mrh$ conda update numpy
Fetching package metadata .........
Solving package specifications: .

Package plan for installation in environment /Users/mrh/anaconda:

The following packages will be UPDATED:

    astropy:      2.0-np112py27_0    --> 2.0-np113py27_0   
    bottleneck:   1.2.1-np112py27_0  --> 1.2.1-np113py27_0 
    h5py:         2.7.0-np112py27_0  --> 2.7.0-np113py27_0 
    matplotlib:   2.0.2-np112py27_0  --> 2.0.2-np113py27_0 
    numba:        0.34.0-np112py27_0 --> 0.34.0-np113py27_0
    numexpr:      2.6.2-np112py27_0  --> 2.6.2-np113py27_0 
    numpy:        1.12.1-py27_0      --> 1.13.1-py27_0     
    pandas:       0.20.2-np112py27_0 --> 0.20.2-np113py27_0
    pytables:     3.3.0-np112py27_0  --> 3.3.0-np113py27_0 
    pywavelets:   0.5.2-np112py27_0  --> 0.5.2-np113py27_0 
    scikit-image: 0.13.0-np112py27_0 --> 0.13.0-np113py27_0
    scikit-learn: 0.18.2-np112py27_0 --> 0.18.2-np113py27_0
    scipy:        0.19.1-np112py27_0 --> 0.19.1-np113py27_0
    statsmodels:  0.8.0-np112py27_0  --> 0.8.0-np113py27_0 

iMac2:tax-calculator mrh$ conda list | grep -e numpy -e pandas -e numba
numba                     0.34.0              np113py27_0  
numpy                     1.13.1                   py27_0  
numpydoc                  0.6.0                    py27_0  
pandas                    0.20.2              np113py27_0  

iMac2:tax-calculator mrh$ pushd taxcalc ; py.test -m one ; popd
============================= test session starts ==============================
platform darwin -- Python 2.7.13, pytest-3.0.5, py-1.4.32, pluggy-0.4.0
rootdir: /Users/mrh/work/OSPC/tax-calculator, inifile: setup.cfg
collected 303 items 

tests/test_behavior.py F

=================================== FAILURES ===================================
_____________________ test_behavioral_response_Calculator ______________________

puf_1991 =       age_head  DSI  EIC  FLPDYR  f2441  f6251  MARS  XTOT   e00200   e00300  ...8798           0    10170         0        0   8798  

[8799 rows x 73 columns]
weights_1991 =        WT2008       WT2009       WT2010       WT2011       WT2012  \
0     335...11201   446.629696   443.611201   446.629696  32095  

[8799 rows x 20 columns]

    @pytest.mark.one
    def test_behavioral_response_Calculator(puf_1991, weights_1991):
        # create Records objects
        records_x = Records(data=puf_1991, weights=weights_1991, start_year=2009)
        records_y = Records(data=puf_1991, weights=weights_1991, start_year=2009)
        # create Policy objects
        policy_x = Policy()
        policy_y = Policy()
        # implement policy_y reform
        reform = {2013: {'_II_rt7': [0.496],
                         '_PT_rt7': [0.496]}}
        policy_y.implement_reform(reform)
        # create two Calculator objects
        behavior_y = Behavior()
        calc_x = Calculator(policy=policy_x, records=records_x)
        calc_y = Calculator(policy=policy_y, records=records_y,
                            behavior=behavior_y)
        # test incorrect use of Behavior._mtr_xy method
        with pytest.raises(ValueError):
            behv = Behavior._mtr_xy(calc_x, calc_y,
                                    mtr_of='e00200p',
>                                   tax_type='nonsense')

tests/test_behavior.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
behavior.py:348: in _mtr_xy
    _, iitax_x, combined_x = calc_x.mtr(mtr_of, wrt_full_compensation=True)
calculate.py:306: in mtr
    self.calc_all(zero_out_calc_vars=zero_out_calculated_vars)
calculate.py:146: in calc_all
    self._calc_one_year(zero_out_calc_vars)
calculate.py:442: in _calc_one_year
    ALD_InvInc_ec_base(self.policy, self.records)
decorators.py:307: in wrapper
    ans = high_level_fn(*args, **kwargs)
<string>:6: in hl_func
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = CPUDispatcher(<function ap_func at 0x117193b90>)
args = (array([ 0.,  0.,  0., ...,  0.,  0.,  0.]), array([ -3.44786334e+07,  -2.36157869e+07,  -1.53949175e+07, ...,
       ....64555302,  101070.91385835,  139925.6231185 , ...,
          1733.18501293,       0.        ,    2287.47975185]), ...)
kws = {}
argtypes = [array(float64, 1d, C), array(float64, 1d, C), array(float64, 1d, C), array(int64, 1d, C), pyobject, array(float64, 1d, C), ...]
a = array([ 0.,  0.,  0., ...,  0.,  0.,  0.])
e = TypingError('Caused By:\nTraceback (most recent call last):\n  File "/Users/mr...nt 4: cannot determine Numba type of <class \'pandas.core.series.Series\'>\n',)
failed_args = [(4, "cannot determine Numba type of <class 'pandas.core.series.Series'>")]
i = 8, arg = array([ 0.,  0.,  0., ...,  0.,  0.,  0.])
val = array([ 0.,  0.,  0., ...,  0.,  0.,  0.]), tp = array(float64, 1d, C)
typeof_exc = ValueError("cannot determine Numba type of <class 'pandas.core.series.Series'>",)
msg = 'Caused By:\nTraceback (most recent call last):\n  File "/Users/mrh/anaconda/lib/python2.7/site-packages/numba/compile...ed by the following argument(s):\n- argument 4: cannot determine Numba type of <class \'pandas.core.series.Series\'>\n'

    def _compile_for_args(self, *args, **kws):
        """
            For internal use.  Compile a specialized version of the function
            for the given *args* and *kws*, and return the resulting callable.
            """
        assert not kws
        argtypes = []
        for a in args:
            if isinstance(a, OmittedArg):
                argtypes.append(types.Omitted(a.value))
            else:
                argtypes.append(self.typeof_pyval(a))
        try:
            return self.compile(tuple(argtypes))
        except errors.TypingError as e:
            # Intercept typing error that may be due to an argument
            # that failed inferencing as a Numba type
            failed_args = []
            for i, arg in enumerate(args):
                val = arg.value if isinstance(arg, OmittedArg) else arg
                try:
                    tp = typeof(val, Purpose.argument)
                except ValueError as typeof_exc:
                    failed_args.append((i, str(typeof_exc)))
                else:
                    if tp is None:
                        failed_args.append(
                            (i,
                             "cannot determine Numba type of value %r" % (val,)))
            if failed_args:
                # Patch error message to ease debugging
                msg = str(e).rstrip() + (
                    "\n\nThis error may have been caused by the following argument(s):\n%s\n"
                    % "\n".join("- argument %d: %s" % (i, err)
                                for i, err in failed_args))
                e.patch_message(msg)
>           raise e
E           TypingError: Caused By:
E           Traceback (most recent call last):
E             File "/Users/mrh/anaconda/lib/python2.7/site-packages/numba/compiler.py", line 238, in run
E               stage()
E             File "/Users/mrh/anaconda/lib/python2.7/site-packages/numba/compiler.py", line 452, in stage_nopython_frontend
E               self.locals)
E             File "/Users/mrh/anaconda/lib/python2.7/site-packages/numba/compiler.py", line 841, in type_inference_stage
E               infer.propagate()
E             File "/Users/mrh/anaconda/lib/python2.7/site-packages/numba/typeinfer.py", line 773, in propagate
E               raise errors[0]
E           TypingError: Invalid usage of getitem with parameters (pyobject, int64)
E            * parameterized
E           File "<string>", line 3
E           [1] During: typing of intrinsic-call at <string> (3)
E           
E           Failed at nopython (nopython frontend)
E           Invalid usage of getitem with parameters (pyobject, int64)
E            * parameterized
E           File "<string>", line 3
E           [1] During: typing of intrinsic-call at <string> (3)
E           
E           This error may have been caused by the following argument(s):
E           - argument 4: cannot determine Numba type of <class 'pandas.core.series.Series'>

/Users/mrh/anaconda/lib/python2.7/site-packages/numba/dispatcher.py:330: TypingError
----------------------------- Captured stdout call -----------------------------
You loaded data for 2009.
Your data include the following unused variables that will be ignored:
  e20500
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Your data include the following unused variables that will be ignored:
  e20500
Tax-Calculator startup automatically extrapolated your data to 2013.
============================= 302 tests deselected =============================
=================== 1 failed, 302 deselected in 2.77 seconds ===================

When using the numpy 1.13 packages, I can pass this one marked test by uncommenting two lines in the decorators.py file in order to "turn off" the decorator logic. The test takes much longer to run, but it does pass. The decorator logic is turned off by changing these lines:

try:
    import numba
    jit = numba.jit  # pylint: disable=invalid-name
    DO_JIT = True
except (ImportError, AttributeError):
    jit = id_wrapper  # pylint: disable=invalid-name
    DO_JIT = False
# One way to use the Python debugger is to do these two things:
#    (a) uncomment the two lines below item (b) in this comment, and
#    (b) import pdb package and call pdb.set_trace() in calculator.py
# jit = id_wrapper
# DO_JIT = False

to these lines:

try:
    import numba
    jit = numba.jit  # pylint: disable=invalid-name
    DO_JIT = True
except (ImportError, AttributeError):
    jit = id_wrapper  # pylint: disable=invalid-name
    DO_JIT = False
# One way to use the Python debugger is to do these two things:
#    (a) uncomment the two lines below item (b) in this comment, and
#    (b) import pdb package and call pdb.set_trace() in calculator.py
jit = id_wrapper
DO_JIT = False

The decorator logic was written before I started working on Tax-Calculator, so I don't know much about it. Any thoughts or suggestions?

@MattHJensen @PeterDSteinberg @talumbau