BUG: Switched shapes in ValueError msg in DataFrame construct (#20742) #24725

meiermark · 2019-01-11T15:13:07Z

closes DataFrame creation incorrect error message #20742
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…-dev#20742)

gfyoung · 2019-01-11T21:33:53Z

doc/source/whatsnew/v0.24.0.rst

@@ -1852,6 +1852,7 @@ Other
 ^^^^^

 - Bug where C variables were declared with external linkage causing import errors if certain other C libraries were imported before Pandas. (:issue:`24113`)
+- Switched shape in ``ValueError`` message when constructiong a :class:`DataFrame` with parameters ``columns`` and ``index`` not matching the shape of the input data. (:issue:`20742`)


Let's be explicit as to how the dimensions are reported (i.e. their order) after the switch.

TST: adjusted panel test for change in value error msg in block manager TST: adjusted dataframe init test from json for change in value error msg in block manager

codecov · 2019-01-12T01:40:41Z

Codecov Report

Merging #24725 into master will decrease coverage by 49.32%.
The diff coverage is 0%.

@@             Coverage Diff             @@
##           master   #24725       +/-   ##
===========================================
- Coverage   92.39%   43.07%   -49.33%     
===========================================
  Files         166      166               
  Lines       52358    52358               
===========================================
- Hits        48374    22551    -25823     
- Misses       3984    29807    +25823

Flag	Coverage Δ
#multiple	`?`
#single	`43.07% <0%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/internals/managers.py	`66.49% <0%> (-29.56%)`	⬇️
pandas/io/formats/latex.py	`0% <0%> (-100%)`	⬇️
pandas/core/categorical.py	`0% <0%> (-100%)`	⬇️
pandas/io/sas/sas_constants.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/plotting.py	`0% <0%> (-100%)`	⬇️
pandas/tseries/converter.py	`0% <0%> (-100%)`	⬇️
pandas/io/formats/html.py	`0% <0%> (-99.35%)`	⬇️
pandas/core/groupby/categorical.py	`0% <0%> (-95.46%)`	⬇️
pandas/io/sas/sas7bdat.py	`0% <0%> (-91.17%)`	⬇️
pandas/io/sas/sas_xport.py	`0% <0%> (-90.15%)`	⬇️
... and 124 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1a81fe...46d4d55. Read the comment docs.

codecov · 2019-01-12T01:40:41Z

Codecov Report

Merging #24725 into master will decrease coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24725      +/-   ##
==========================================
- Coverage   92.38%   92.38%   -0.01%     
==========================================
  Files         166      166              
  Lines       52382    52383       +1     
==========================================
  Hits        48395    48395              
- Misses       3987     3988       +1

Flag	Coverage Δ
#multiple	`90.81% <100%> (ø)`	⬆️
#single	`42.91% <0%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/internals/managers.py	`96.06% <100%> (+0.01%)`	⬆️
pandas/util/testing.py	`88.04% <0%> (-0.1%)`	⬇️
pandas/core/indexes/timedeltas.py	`90.22% <0%> (-0.03%)`	⬇️
pandas/core/computation/pytables.py	`92.35% <0%> (-0.03%)`	⬇️
pandas/core/dtypes/cast.py	`88.7% <0%> (-0.02%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 17a6bc5...cb282dc. Read the comment docs.

doc/source/whatsnew/v0.24.0.rst

jreback · 2019-01-13T21:15:27Z

pandas/core/internals/managers.py

@@ -1673,8 +1673,10 @@ def create_block_manager_from_arrays(arrays, names, axes):

 def construction_error(tot_items, block_shape, axes, e=None):
    """ raise a helpful message about our construction """
-    passed = tuple(map(int, [tot_items] + list(block_shape)))
-    implied = tuple(map(int, [len(ax) for ax in axes]))
+    passed = tuple(map(int, list(block_shape) + [tot_items]))


just do a reverse if ndim == 2 here

Probably not a good idea for panels with 3 dimensions (items, major, minor)
with reversed for ndim == 2: (items, major, minor)
current: (major, minor, items)

The current version is the intended behavior, right?

I don't care about Panels, they are being removed shortly.

this only affect ndim==2

pandas/tests/frame/test_constructors.py

pandas/tests/test_panel.py

CLN: removed issue number above new tests

jreback · 2019-01-16T01:58:22Z

doc/source/whatsnew/v0.24.0.rst

@@ -1852,6 +1852,8 @@ Other
 ^^^^^

 - Bug where C variables were declared with external linkage causing import errors if certain other C libraries were imported before Pandas. (:issue:`24113`)
+- Cleanup ``ValueError`` message thrown in creation of block manager: list block shape before number of total items in passed input shape and implied shape. (:issue:`20742`)


move to Reshaping section.

Reversed shapes in ValueError message when data shape does not match index shape in construction

jreback · 2019-01-16T01:59:01Z

pandas/core/internals/managers.py

@@ -1673,8 +1673,10 @@ def create_block_manager_from_arrays(arrays, names, axes):

 def construction_error(tot_items, block_shape, axes, e=None):
    """ raise a helpful message about our construction """
-    passed = tuple(map(int, [tot_items] + list(block_shape)))
-    implied = tuple(map(int, [len(ax) for ax in axes]))
+    passed = tuple(map(int, list(block_shape) + [tot_items]))


this only affect ndim==2

jreback · 2019-01-16T02:00:18Z

pandas/tests/test_panel.py

@@ -1143,20 +1143,20 @@ def test_from_dict_mixed_orient(self):
        assert panel['A'].values.dtype == np.float64

    def test_constructor_error_msgs(self):
-        msg = (r"Shape of passed values is \(3, 4, 5\), "
-               r"indices imply \(4, 5, 5\)")
+        msg = (r"Shape of passed values is \(4, 5, 3\), "


leave panel entirely.

TomAugspurger

Need to fix a merge conflict.

TomAugspurger · 2019-01-16T20:29:47Z

doc/source/whatsnew/v0.24.0.rst

@@ -1108,6 +1108,7 @@ update the ``ExtensionDtype._metadata`` tuple to match the signature of your
 - :meth:`~pandas.api.types.ExtensionArray.searchsorted` has been added (:issue:`24350`)
 - Support for reduction operations such as ``sum``, ``mean`` via opt-in base class method override (:issue:`22762`)
 - :func:`ExtensionArray.isna` is allowed to return an ``ExtensionArray`` (:issue:`22325`).
+- Cleanup ``ValueError`` message thrown in creation of block manager: list block shape before number of total items in passed input shape and implied shape. (:issue:`20742`)


We probably shouldn't mention block manager in the release notes. Maybe just something about "fixed (or improved?) error message when ...)" where ... is a user action that would lead to the exception.

jreback · 2019-01-17T12:50:04Z

pandas/core/internals/managers.py

@@ -1674,7 +1674,13 @@ def create_block_manager_from_arrays(arrays, names, axes):
 def construction_error(tot_items, block_shape, axes, e=None):
    """ raise a helpful message about our construction """
    passed = tuple(map(int, [tot_items] + list(block_shape)))
-    implied = tuple(map(int, [len(ax) for ax in axes]))
+    if len(passed) == 2:


can you add a comment on the reversing. (e.g. this is a user facing error message and this is a DataFrame construction)

you can do len(passed) <= 2

jreback · 2019-01-17T12:50:53Z

doc/source/whatsnew/v0.24.0.rst

@@ -1816,6 +1816,7 @@ Reshaping
 - Bug in :func:`DataFrame.unstack` where a ``ValueError`` was raised when unstacking timezone aware values (:issue:`18338`)
 - Bug in :func:`DataFrame.stack` where timezone aware values were converted to timezone naive values (:issue:`19420`)
 - Bug in :func:`merge_asof` where a ``TypeError`` was raised when ``by_col`` were timezone aware values (:issue:`21184`)
+- Reversed shapes in ValueError message when data shape does not match index shape in construction. (:issue:`20742`)


say Incorrect shape in DataFrame construction error message.

jreback

whatsnew correction. ping on green.

jreback · 2019-01-18T12:27:43Z

doc/source/whatsnew/v0.24.0.rst

@@ -1816,6 +1816,7 @@ Reshaping
 - Bug in :func:`DataFrame.unstack` where a ``ValueError`` was raised when unstacking timezone aware values (:issue:`18338`)
 - Bug in :func:`DataFrame.stack` where timezone aware values were converted to timezone naive values (:issue:`19420`)
 - Bug in :func:`merge_asof` where a ``TypeError`` was raised when ``by_col`` were timezone aware values (:issue:`21184`)
+- Bug incorrect shape in ``DataFrame`` construction ValueError message. (:issue:`20742`)


Bug showing an incorrect shape when erroring during DataFrame construction

TomAugspurger · 2019-01-18T16:53:41Z

Thanks @meiermark.

…-dev#20742) (pandas-dev#24725) * BUG: Switched shapes in ValueError msg in DataFrame construct (pandas-dev#20742) * DOC: improved docu of the value error msg changes TST: adjusted panel test for change in value error msg in block manager TST: adjusted dataframe init test from json for change in value error msg in block manager * BUG: printed a list instead of a tuple in the ValueError msg * CLN: removed unnecessary list comprehension * DOC: Improved whatsnew message * CLN: readded deleted tests CLN: removed issue number above new tests * Bug: No changes for panel * DOC: improved whatsnew message for reversed shapes * DOC: improved comments and whatsnew message for df construction error * DOC: improved whatsnew message

BUG: Switched shapes in ValueError msg in DataFrame construct (pandas…

92b0bb3

…-dev#20742)

gfyoung added Error Reporting Incorrect or improved errors from pandas DataFrame DataFrame data structure labels Jan 11, 2019

gfyoung reviewed Jan 11, 2019

View reviewed changes

DOC: improved docu of the value error msg changes

46d4d55

TST: adjusted panel test for change in value error msg in block manager TST: adjusted dataframe init test from json for change in value error msg in block manager

meiermark added 2 commits January 12, 2019 04:19

BUG: printed a list instead of a tuple in the ValueError msg

aa3c5d5

CLN: removed unnecessary list comprehension

35acafd

jreback requested changes Jan 13, 2019

View reviewed changes

meiermark added 2 commits January 15, 2019 14:53

DOC: Improved whatsnew message

cdeeacc

CLN: readded deleted tests

31cf4f6

CLN: removed issue number above new tests

jreback requested changes Jan 16, 2019

View reviewed changes

Bug: No changes for panel

39b8ca9

TomAugspurger reviewed Jan 16, 2019

View reviewed changes

meiermark added 2 commits January 17, 2019 10:09

DOC: improved whatsnew message for reversed shapes

244621b

Merge branch 'master' into dataframe_creation_error_reversed_shape

4cfae58

jreback requested changes Jan 17, 2019

View reviewed changes

jreback added this to the 0.24.0 milestone Jan 17, 2019

DOC: improved comments and whatsnew message for df construction error

7a6cd18

jreback requested changes Jan 18, 2019

View reviewed changes

DOC: improved whatsnew message

cb282dc

TomAugspurger merged commit 08f92c4 into pandas-dev:master Jan 18, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Switched shapes in ValueError msg in DataFrame construct (#20742) #24725

BUG: Switched shapes in ValueError msg in DataFrame construct (#20742) #24725

meiermark commented Jan 11, 2019 •

edited

Loading

gfyoung Jan 11, 2019

codecov bot commented Jan 12, 2019

codecov bot commented Jan 12, 2019 •

edited

Loading

jreback Jan 13, 2019

meiermark Jan 15, 2019

jreback Jan 16, 2019

jreback Jan 16, 2019

jreback Jan 16, 2019

jreback Jan 16, 2019

jreback Jan 16, 2019

TomAugspurger left a comment

TomAugspurger Jan 16, 2019

jreback Jan 17, 2019

jreback Jan 17, 2019

jreback left a comment

jreback Jan 18, 2019

TomAugspurger commented Jan 18, 2019

BUG: Switched shapes in ValueError msg in DataFrame construct (#20742) #24725

BUG: Switched shapes in ValueError msg in DataFrame construct (#20742) #24725

Conversation

meiermark commented Jan 11, 2019 • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jan 12, 2019

Codecov Report

codecov bot commented Jan 12, 2019 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TomAugspurger commented Jan 18, 2019

meiermark commented Jan 11, 2019 •

edited

Loading

codecov bot commented Jan 12, 2019 •

edited

Loading