You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The reason why the dtype is being loaded as a int64 in python 3 is because pandas is not reading correctly the value_type attribute from the hdf file. The current implementation does not explicitly decodes to "UTF-8" the attribute value when its type is np.bytes_, which causes it to read the dtype as b'datetime64 instead of the correct datetime64 value. That, in turn, causes it to treat the raw value 1581012000000000000 with an inferred dtype of int64 instead of coercing it to a M8[ns] dtype that would give the correct 2020-02-06 18:00:00 datetime value.
This is a similar problem to the one from issue #15725, but that one was when reading the index from the hdf but this one is when reading the data.
The fix for this seems to be really straight forward, you just need to ensure the value of thevalue_type attribute is decoded to UTF-8 when reading it from the hdf file. More specifically, in this line of pytables.py
diff --git a/pytables.py b/pytables.py
index 06c9aa1..9d074d5 100644
--- a/pytables.py+++ b/pytables.py@@ -2724,7 +2724,7 @@ class GenericFixed(Fixed):
if isinstance(node, tables.VLArray):
ret = node[0][start:stop]
else:
- dtype = getattr(attrs, "value_type", None)+ dtype = _ensure_decoded(getattr(attrs, "value_type", None))
shape = getattr(attrs, "shape", None)
I can make the change and open a PR, as it seems to be really straight forward, just let me know what you prefer.
Note: this is an issue only when the ".hdf" file is created in python 2 using the fixed format. If it is created using the table format it works as expected.
Thanks!
Expected Output
The expected output is that in python 3 it will parse the dtype of the columns correctly, so in this case it should be:
The text was updated successfully, but these errors were encountered:
pedroreys
changed the title
Deconding issue when reading in py3 a datetime64 hdf data that was created in py2
Decoding issue when reading in py3 a datetime64 hdf data that was created in py2
Feb 6, 2020
Code Sample, a copy-pastable example if possible
In Python 2, create the
.hdf
file with adatetime64[ns]
columnReading the hdf in python 2 works just fine
Reading the hdf in Python 3 the dtype of the data column is set to
int64
instead ofdatatime64
Problem description
The reason why the dtype is being loaded as a
int64
in python 3 is because pandas is not reading correctly thevalue_type
attribute from the hdf file. The current implementation does not explicitly decodes to "UTF-8" the attribute value when its type isnp.bytes_
, which causes it to read the dtype asb'datetime64
instead of the correctdatetime64
value. That, in turn, causes it to treat the raw value1581012000000000000
with an inferred dtype ofint64
instead of coercing it to aM8[ns]
dtype that would give the correct2020-02-06 18:00:00
datetime value.This is a similar problem to the one from issue #15725, but that one was when reading the index from the hdf but this one is when reading the data.
The fix for this seems to be really straight forward, you just need to ensure the value of the
value_type
attribute is decoded to UTF-8 when reading it from the hdf file. More specifically, in this line of pytables.pyI can make the change and open a PR, as it seems to be really straight forward, just let me know what you prefer.
Note: this is an issue only when the ".hdf" file is created in python 2 using the fixed format. If it is created using the table format it works as expected.
Thanks!
Expected Output
The expected output is that in python 3 it will parse the dtype of the columns correctly, so in this case it should be:
Output of
pd.show_versions()
pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.1.0.post20200127
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : 3.6.1
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None
The text was updated successfully, but these errors were encountered: