Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: HDFStore fixed format not decoding meta-data in PY3 #15725

Open
toobaz opened this issue Mar 17, 2017 · 3 comments
Open

BUG: HDFStore fixed format not decoding meta-data in PY3 #15725

toobaz opened this issue Mar 17, 2017 · 3 comments
Labels
Bug Compat pandas objects compatability with Numpy or Python functions IO HDF5 read_hdf, HDFStore

Comments

@toobaz
Copy link
Member

toobaz commented Mar 17, 2017

Code Sample, a copy-pastable example if possible

In Python 2:

df = pd.DataFrame([[1, 2], [3, 4]])
df.index.name = "date"
df.to_hdf('/tmp/test.hdf', 'test')

In Python 3:

In [2]: pd.read_hdf('/tmp/test.hdf', 'test')
Out[2]: 
         0  1
b'date'      
0        1  2
1        3  4

Problem description

b'date' should actually be just date

Expected Output

Out[2]: 
         0  1
date      
0        1  2
1        3  4

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.7.0-1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.utf8
LOCALE: it_IT.UTF-8

pandas: 0.19.0+603.g2cad4dd0b
pytest: 3.0.6
pip: 9.0.1
setuptools: 33.1.1
Cython: 0.25.2
numpy: 1.12.0
scipy: 0.18.1
xarray: 0.9.1
IPython: 5.1.0.dev
sphinx: 1.4.9
patsy: 0.3.0-dev
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: 1.2.0
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.0
openpyxl: 2.3.0
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: 3.7.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
pandas_gbq: None
pandas_datareader: 0.2.1

@jreback
Copy link
Contributor

jreback commented Mar 17, 2017

fixed stores don't handle encoding very well (or the meta data rather). table do just fine.

In [2]: df = pd.DataFrame([[1, 2], [3, 4]])
   ...: df.index.name = "date"
   ...: df.to_hdf('py2_test.hdf', 'test', format='table')
   ...: df.to_hdf('py2_test.hdf', 'test2', format='fixed')
   ...: 

In [3]: quit()
(py2.7) bash-3.2$ . deactivate
bash-3.2$ cd

bash-3.2$ . activate pandas
(pandas) bash-3.2$ ipython
Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul  2 2016, 17:52:12) 
Type "copyright", "credits" or "license" for more information.

IIn [1]: pd.read_hdf('py2_test.hdf', 'test')
Out[1]: 
      0  1
date      
0     1  2
1     3  4

In [2]: pd.read_hdf('py2_test.hdf', 'test2')
Out[2]: 
         0  1
b'date'      
0        1  2
1        3  4

@jreback
Copy link
Contributor

jreback commented Mar 17, 2017

its a little related to this: #11126

@jreback jreback changed the title Misinterpreted strings when reading with Python 3 some hdf generated with Python 2 BUG: HDFStore fixed format not decoding meta-data in PY3 Mar 17, 2017
@jreback jreback added 2/3 Compat Bug IO HDF5 read_hdf, HDFStore labels Mar 17, 2017
@jreback jreback added this to the Next Major Release milestone Mar 17, 2017
@jreback
Copy link
Contributor

jreback commented Feb 4, 2019

xref #25058, @rbenes if you want to look.

@mroeschke mroeschke added Compat pandas objects compatability with Numpy or Python functions and removed 2/3 Compat labels May 8, 2021
@mroeschke mroeschke removed this from the Contributions Welcome milestone Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Compat pandas objects compatability with Numpy or Python functions IO HDF5 read_hdf, HDFStore
Projects
None yet
Development

No branches or pull requests

4 participants