DEPR: Silent dropping of nuisance columns in agg_list_like

[rhshadrach]

Part of #43740

• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[jbrockmendel]

are all warnings that happen here going to represent a failure?

[rhshadrach]

Woah - thanks. No, probably not. Will inspect the warnings themselves.

[rhshadrach]

This got a bit hairy, I wasn't able to find a good way to capture and suppress only our warnings while passing through any other warnings and being able to determine if we raised. Supplying a filter within the catch_warnings context manager means we can't fell if we raised; using record=True there means all user warnings would also be suppressed.

If there are any suggestions for a better implementation, it would be appreciated.

[jbrockmendel]

what happens if you dont catch warnings at all here?

[rhshadrach]

Running

def foo(x):
   return x.prod()
df = pd.DataFrame({'a': [1, 2], 'b': ['x', 'y'], 'c': ['z', 'w'], 'd': ['a', 'b']})
df.agg([foo, lambda x: x.sum()])

gives

FutureWarning: ['foo'] did not aggregate successfully.
FutureWarning: ['foo'] did not aggregate successfully.
FutureWarning: ['foo'] did not aggregate successfully.

[jbrockmendel]

i think there's something like a filter="once" to prevent exactly that (and i think its the default)

[rhshadrach]

I see - thanks. Digging into this, the only way I can see why I'm getting multiple warnings is because of https://bugs.python.org/issue29672. I haven't confirmed yet if the code within agg ever hits a separate catch_warnings; if it does that will cause repeat messages.

[rhshadrach]

I'm still going to verify that the use of another catch_warnings (in a different part of the code) is the cause for the multiple messages I am seeing without the special handling in this PR.

[rhshadrach]

I've confirmed it is because of the Python bug mentioned above. The code in the sample hits pandas.core.indexes.base.Index_with_infer which uses the catch_warnings context manager. Here is a MWE to demonstrate the bug (mirroring the structure of agg_list_like):

def foo():
    with warnings.catch_warnings():
        warnings.filterwarnings("ignore", ".*the Index constructor", FutureWarning)


def bar(ndim):
    if ndim == 1:
        pass
    else:
        for _ in range(3):
            bar(1)

    warnings.warn('test warning', FutureWarning)
    foo()

bar(2)

With the call to foo from within bar, three warning messages are printed. When the call to foo is removed, only one is printed.

With this, are we okay with catching the warnings as was merged @jbrockmendel?

[jbrockmendel]

im fine with it. a comment to this effect might be worthwhile

[jreback]

can you rebase

[jreback]

thanks @rhshadrach very nice.