Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Pandas 2.0 support #5662

Merged
merged 37 commits into from
Apr 21, 2023
Merged

Add Pandas 2.0 support #5662

merged 37 commits into from
Apr 21, 2023

Conversation

hoxbro
Copy link
Member

@hoxbro hoxbro commented Mar 14, 2023

This PR adds support for Pandas 2.0 and fixes a bug and an inconsistency.

Changes made in this PR:

  • Internally holoviews/operation/datashader now uses datetime[ns] instead of datetime[us], for reasons see the comment below.
  • The changes in the notebooks are simple changes where an argument has been changed to a keyword-only argument.
  • Fixes Rasterize not working in combination with multiple rectangles with DateTime coordinates #5239.
  • Fixes an inconsistency where dt_to_int could return a float in certain cases.
  • Update setup.py/setup.cfg to work with Pandas 2.
  • Add testing to both Pandas 2.0 and Pandas 1 in test CI.
  • Move the logic of examples to ignore into examples/conftest,py.

Reference:

@codecov-commenter
Copy link

codecov-commenter commented Mar 14, 2023

Codecov Report

Merging #5662 (d2ec764) into main (04ba54a) will increase coverage by 0.00%.
The diff coverage is 68.57%.

@@           Coverage Diff           @@
##             main    #5662   +/-   ##
=======================================
  Coverage   88.29%   88.29%           
=======================================
  Files         302      302           
  Lines       62576    62586   +10     
=======================================
+ Hits        55253    55263   +10     
  Misses       7323     7323           
Impacted Files Coverage Δ
holoviews/operation/datashader.py 83.81% <52.17%> (+0.02%) ⬆️
holoviews/core/util.py 86.60% <100.00%> (ø)
holoviews/element/util.py 82.21% <100.00%> (ø)
holoviews/tests/core/data/test_daskinterface.py 88.09% <100.00%> (+1.42%) ⬆️
holoviews/tests/plotting/bokeh/test_callbacks.py 98.65% <100.00%> (-0.01%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@hoxbro hoxbro changed the title Initial Pandas 2 support Add Pandas 2.0 support Mar 14, 2023
@hoxbro
Copy link
Member Author

hoxbro commented Mar 14, 2023

The datetime change is because Pandas is now beginning to support non-nanosecond precision values:

import pandas as pd
from datetime import datetime

s0 = pd.Series([datetime(2020, 1, 1)])
s1 = s0.astype("datetime64[us]")
s2 = s0.astype("int64")
s3 = s1.astype("int64")


print(pd.__version__)

for s in [s0, s1, s2, s3]:
    print(f"{s[0]}, {s.dtype}")
    print(f"{s.values[0]}, {s.values.dtype}")
    print()

Outputs:

1.5.3
2020-01-01 00:00:00, datetime64[ns]
2020-01-01T00:00:00.000000000, datetime64[ns]

2020-01-01 00:00:00, datetime64[ns]
2020-01-01T00:00:00.000000000, datetime64[ns]

1577836800000000000, int64
1577836800000000000, int64

1577836800000000000, int64
1577836800000000000, int64
2.0.0rc0
2020-01-01 00:00:00, datetime64[ns]
2020-01-01T00:00:00.000000000, datetime64[ns]

2020-01-01 00:00:00, datetime64[us]
2020-01-01T00:00:00.000000, datetime64[us]

1577836800000000000, int64
1577836800000000000, int64

1577836800000000, int64
1577836800000000, int64

Also beginning to get this warning from xarray because of this:

/home/shh/Repos/holoviz/holoviews/holoviews/operation/datashader.py:848: UserWarning: Converting non-nanosecond precision datetime values to nanosecond precision. This behavior can eventually be relaxed in xarray, as it is an artifact from pandas which is now beginning to support non-nanosecond precision values. This warning is caused by passing non-nanosecond np.datetime64 or np.timedelta64 values to the DataArray or Variable constructor; it can be silenced by converting the values to nanosecond precision ahead of time.
  agg[ydim] = (agg[ydim]/1e3).astype('datetime64[us]')

Can be recreated with:

import datetime as dt

import holoviews as hv
from holoviews.operation.datashader import rasterize

rects = hv.Rectangles([(0, dt.datetime(2016, 1, 2), 4, dt.datetime(2016, 1, 3))])
agg = rasterize(rects, width=4, height=4, dynamic=False)

@hoxbro
Copy link
Member Author

hoxbro commented Apr 6, 2023

xarray does not yet support pandas 2.0: pydata/xarray#7716

@hoxbro hoxbro force-pushed the pandas_2 branch 2 times, most recently from 626933a to 6c6e9ac Compare April 6, 2023 08:02
@hoxbro hoxbro added this to the 1.16.0 milestone Apr 11, 2023
@hoxbro hoxbro merged commit 38a4350 into main Apr 21, 2023
@hoxbro hoxbro deleted the pandas_2 branch April 21, 2023 08:54
Copy link

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Rasterize not working in combination with multiple rectangles with DateTime coordinates
3 participants