Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REGR: fix isin for large series with nan and mixed object dtype (causing regression in read_csv) #37499

Merged
merged 4 commits into from
Oct 30, 2020
Merged

REGR: fix isin for large series with nan and mixed object dtype (causing regression in read_csv) #37499

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
a62e4c0
REGR: fix isin for large series with nan and mixed object dtype (caus…
jorisvandenbossche Oct 29, 2020
d516b43
add whatsnew
jorisvandenbossche Oct 29, 2020
520cf10
also add csv test
jorisvandenbossche Oct 29, 2020
93c79cc
use pd.isna instead of np.isnan
jorisvandenbossche Oct 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.1.4.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ including other versions of pandas.
Fixed regressions
~~~~~~~~~~~~~~~~~
- Fixed regression in :func:`read_csv` raising a ``ValueError`` when ``names`` was of type ``dict_keys`` (:issue:`36928`)
- Fixed regression in :func:`read_csv` with more than 1M rows and specifying a ``index_col`` argument (:issue:`37094`)
- Fixed regression where attempting to mutate a :class:`DateOffset` object would no longer raise an ``AttributeError`` (:issue:`36940`)
- Fixed regression where :meth:`DataFrame.agg` would fail with :exc:`TypeError` when passed positional arguments to be passed on to the aggregation function (:issue:`36948`).
- Fixed regression in :class:`RollingGroupby` with ``sort=False`` not being respected (:issue:`36889`)
Expand Down
2 changes: 1 addition & 1 deletion pandas/core/algorithms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ def isin(comps: AnyArrayLike, values: AnyArrayLike) -> np.ndarray:
if len(comps) > 1_000_000 and not is_object_dtype(comps):
# If the the values include nan we need to check for nan explicitly
# since np.nan it not equal to np.nan
if np.isnan(values).any():
if isna(values).any():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How bad is it if you skip the check and always use the logical_or? Is it possible that the isna is more expensive than the logical_or and isnan?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

From a quick check with the simple example from the test I don't really see much difference.
But, for backporting this, I would personally prefer to keep the change as minimal as possible. For master / 1.2, we should maybe re-evaluate this full branch, but see also #36611 which is basically already doing this (but more drastically, by potentially removing the use of np.in1d entirely)

Note that in the if clause here it is isna(values) (so on the values passed to the isin() method), which avoids isnan(comps) (the values of the Series), where values is typically much smaller than comps.

f = lambda c, v: np.logical_or(np.in1d(c, v), np.isnan(c))
else:
f = np.in1d
Expand Down
15 changes: 15 additions & 0 deletions pandas/tests/io/parser/test_index_col.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -207,3 +207,18 @@ def test_header_with_index_col(all_parsers):

result = parser.read_csv(StringIO(data), index_col="I11", header=0)
tm.assert_frame_equal(result, expected)


@pytest.mark.slow
def test_index_col_large_csv(all_parsers):
# https://github.com/pandas-dev/pandas/issues/37094
parser = all_parsers

N = 1_000_001
df = DataFrame({"a": range(N), "b": np.random.randn(N)})

with tm.ensure_clean() as path:
df.to_csv(path, index=False)
result = parser.read_csv(path, index_col=[0])

tm.assert_frame_equal(result, df.set_index("a"))
10 changes: 10 additions & 0 deletions pandas/tests/series/methods/test_isin.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,3 +89,13 @@ def test_isin_read_only(self):
result = s.isin(arr)
expected = Series([True, True, True])
tm.assert_series_equal(result, expected)


@pytest.mark.slow
def test_isin_large_series_mixed_dtypes_and_nan():
# https://github.com/pandas-dev/pandas/issues/37094
# combination of object dtype for the values and > 1_000_000 elements
ser = Series([1, 2, np.nan] * 1_000_000)
result = ser.isin({"foo", "bar"})
expected = Series([False] * 3 * 1_000_000)
tm.assert_series_equal(result, expected)