Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"RecursionError: maximum recursion depth exceeded while calling a Python object" when using groupby, resample, and agg #23568

Closed
trianta2 opened this issue Nov 8, 2018 · 4 comments
Closed

"RecursionError: maximum recursion depth exceeded while calling a Python object" when using groupby, resample, and agg #23568

trianta2 opened this issue Nov 8, 2018 · 4 comments
Labels
Apply Apply, Aggregate, Transform, Map Bug Closing Candidate May be closeable, needs more eyeballs Groupby Resample resample method

Comments

@trianta2
Copy link

trianta2 commented Nov 8, 2018

Code Sample, a copy-pastable example if possible

from datetime import datetime, timedelta
import pandas as pd

times = [datetime.now() + timedelta(hours=i) for i in range(10)] * 2
ids = ['foo', 'bar'] * 10
df = pd.DataFrame({'time': times, 'id': ids, 'value': range(20)})

# raises RecursionError: maximum recursion depth exceeded while calling a Python object
df.set_index('time').groupby('id').resample('H').agg(['mean', 'std'])

Problem description

This issue is derived from #23531 and affects the current master branch.

Using agg raises a RecursionError while using methods mean and std work as expected, i.e.:

df.set_index('time').groupby('id').resample('H').mean()
df.set_index('time').groupby('id').resample('H').std()

Here's a chunk of stack trace:

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 608, in _aggregate_multiple_funcs
    results.append(colg.aggregate(arg))

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/resample.py", line 258, in aggregate
    result, how = self._aggregate(func, *args, **kwargs)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 562, in _aggregate
    _axis=_axis), None

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 608, in _aggregate_multiple_funcs
    results.append(colg.aggregate(arg))

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/resample.py", line 258, in aggregate
    result, how = self._aggregate(func, *args, **kwargs)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 562, in _aggregate
    _axis=_axis), None

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 608, in _aggregate_multiple_funcs
    results.append(colg.aggregate(arg))

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/resample.py", line 258, in aggregate
    result, how = self._aggregate(func, *args, **kwargs)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 562, in _aggregate
    _axis=_axis), None

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/base.py", line 607, in _aggregate_multiple_funcs
    subset=obj.iloc[:, index])

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/indexing.py", line 1498, in __getitem__
    return self._getitem_tuple(key)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/indexing.py", line 2149, in _getitem_tuple
    return self._getitem_lowerdim(tup)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/indexing.py", line 990, in _getitem_lowerdim
    section = self._getitem_axis(key, axis=i)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/indexing.py", line 2236, in _getitem_axis
    return self._get_loc(key, axis=axis)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/indexing.py", line 153, in _get_loc
    return self.obj._ixs(key, axis=axis)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/frame.py", line 2764, in _ixs
    result = self._box_col_values(values, label)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/frame.py", line 3229, in _box_col_values
    return klass(values, index=self.index, name=items, fastpath=True)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/series.py", line 260, in __init__
    self.name = name

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/generic.py", line 4742, in __setattr__
    object.__getattribute__(self, name)

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/series.py", line 382, in name
    return self._name

  File "REDACTED/envs/py36/lib/python3.6/site-packages/pandas-0.24.0.dev0+936.gcf4c0b616-py3.6-macosx-10.7-x86_64.egg/pandas/core/generic.py", line 4726, in __getattr__
    return object.__getattribute__(self, name)

RecursionError: maximum recursion depth exceeded while calling a Python object

Expected Output

A dataframe with the combined outputs of:

>>> df.set_index('time').groupby('id').resample('H').mean()
                         value
id  time                      
bar 2018-11-06 12:00:00    6.0
    2018-11-06 13:00:00    NaN
    2018-11-06 14:00:00    8.0
    2018-11-06 15:00:00    NaN
    2018-11-06 16:00:00   10.0
    2018-11-06 17:00:00    NaN
    2018-11-06 18:00:00   12.0
    2018-11-06 19:00:00    NaN
    2018-11-06 20:00:00   14.0
foo 2018-11-06 11:00:00    5.0
    2018-11-06 12:00:00    NaN
    2018-11-06 13:00:00    7.0
    2018-11-06 14:00:00    NaN
    2018-11-06 15:00:00    9.0
    2018-11-06 16:00:00    NaN
    2018-11-06 17:00:00   11.0
    2018-11-06 18:00:00    NaN
    2018-11-06 19:00:00   13.0

>>> df.set_index('time').groupby('id').resample('H').std()
                            value
id  time                         
bar 2018-11-06 12:00:00  7.071068
    2018-11-06 13:00:00       NaN
    2018-11-06 14:00:00  7.071068
    2018-11-06 15:00:00       NaN
    2018-11-06 16:00:00  7.071068
    2018-11-06 17:00:00       NaN
    2018-11-06 18:00:00  7.071068
    2018-11-06 19:00:00       NaN
    2018-11-06 20:00:00  7.071068
foo 2018-11-06 11:00:00  7.071068
    2018-11-06 12:00:00       NaN
    2018-11-06 13:00:00  7.071068
    2018-11-06 14:00:00       NaN
    2018-11-06 15:00:00  7.071068
    2018-11-06 16:00:00       NaN
    2018-11-06 17:00:00  7.071068
    2018-11-06 18:00:00       NaN
    2018-11-06 19:00:00  7.071068

Output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Darwin
OS-release: 16.7.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.0.dev0+936.gcf4c0b616
pytest: 3.7.1
pip: 18.0
setuptools: 39.1.0
Cython: 0.29
numpy: 1.14.5
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.7.5
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.12
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None
@trianta2 trianta2 mentioned this issue Nov 8, 2018
Closed
@WillAyd WillAyd added Bug Groupby Resample resample method labels Nov 8, 2018
@WillAyd WillAyd added this to the Contributions Welcome milestone Nov 12, 2018
@moonshoes87 moonshoes87 mentioned this issue Feb 6, 2019
Closed
@mroeschke mroeschke added the Apply Apply, Aggregate, Transform, Map label May 11, 2020
@Tekaichi
Copy link

Hello,
Is anyone working on a fix for this issue?

@elicutler
Copy link

I am running into this issue on pandas version 1.3.2

@eymerich92
Copy link

Hi!
I am also experiencing this issue on pandas 1.4.1.
Perhaps it may be useful to know that RecursionError is only thrown when using agg with a list (even with a single element):
df.set_index('time').groupby('id').resample('H').aggregate(['std'])
but not for single strings:
df.set_index('time').groupby('id').resample('H').agg('std') # works fine

@mroeschke mroeschke removed this from the Contributions Welcome milestone Oct 13, 2022
@jbrockmendel
Copy link
Member

This correctly raises on main bc we no longer ignore nuisance columns. Is there a reproducer that does not involve nuisance columns?

@mroeschke mroeschke added the Closing Candidate May be closeable, needs more eyeballs label May 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Apply Apply, Aggregate, Transform, Map Bug Closing Candidate May be closeable, needs more eyeballs Groupby Resample resample method
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@WillAyd @trianta2 @jbrockmendel @mroeschke @Tekaichi @elicutler @eymerich92