DEPR: list-likes of list-likes in str.cat #22264

h-vetinari · 2018-08-09T18:14:53Z

closes DEPR: list of lists in Series.str.cat #21950
tests modified / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

As mentioned in #21950, my suggestion is to modify the allowed combinations (as of v0.23) as follows:

Type of "others"                        |  action  |  comment
---------------------------------------------------------------------
list-like of strings                    |   keep   |  as before; mimics behavior elsewhere,
                                                      cf.: pd.Series(range(3)) + [2,4,6]
Series                                  |   keep   |
np.ndarray (1-dim)                      |   keep   |
DataFrame                               |   keep   |  sequential concatenation
np.ndarray (2-dim)                      |   keep   |  sequential concatenation
list-like of
    Series/Index/np.ndarray (1-dim)     |   keep   |  sequential concatenation
list-like containing list-likes (1-dim)
    other than Series/Index/np.ndarray  |   DEPR   |  sequential concatenation

In other words, if the user wants sequential concatenation, there are many possibilities available, and list-of-lists does not have to be one of them, IMO. This would substantially simplify (post-deprecation) the code for str.cat._get_series_list:

def _get_series_list(self, others):
    """
    Auxiliary function for :meth:`str.cat`. Turn potentially mixed input
    into a list of Series (elements without an index must match the length
    of the calling Series/Index).

    Parameters
    ----------
    others : Series, DataFrame, np.ndarray, list-like or list-like of
        objects that are either Series, Index or np.ndarray (1-dim)

    Returns
    -------
    list : others transformed into list of Series
    """
    from pandas import Index, Series, DataFrame

    # self._orig is either Series or Index
    idx = self._orig if isinstance(self._orig, Index) else self._orig.index

    # Generally speaking, all objects without an index inherit the index
    # `idx` of the calling Series/Index - i.e. must have matching length.
    # Objects with an index (i.e. Series/Index/DataFrame) keep their own.
    if isinstance(others, Series):
        return [others]
    elif isinstance(others, Index):
        return [Series(others.values, index=others)]
    elif isinstance(others, DataFrame):
        return [others[x] for x in others]
    elif isinstance(others, np.ndarray) and others.ndim == 2:
        others = DataFrame(others, index=idx)
        return [others[x] for x in others]
    elif is_list_like(others):
        others = list(others)  # ensure iterators do not get read twice etc

        # in case of list-like `others`, all elements must be
        # either Series/Index/np.ndarray (1-dim)...
        if all(isinstance(x, (Series, Index))
               or (isinstance(x, np.ndarray) and x.ndim == 1) for x in others):
            los = []
            while others:  # iterate through list and append each element
                los = los + self._get_series_list(others.pop(0))
            return los
        # ... or just strings
        elif all(not is_list_like(x) for x in others):
            return [Series(others, index=idx)]
    raise TypeError('others must be Series, Index, DataFrame, np.ndarrary or '
                    'list-like (either containing only strings or containing '
                    'only objects of type Series/Index/np.ndarray[1-dim])')

gfyoung · 2018-08-09T20:02:26Z

doc/source/text.rst

@@ -306,20 +306,20 @@ The same alignment can be used when ``others`` is a ``DataFrame``:
 Concatenating a Series and many objects into a Series
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

-All one-dimensional list-likes can be arbitrarily combined in a list-like container (including iterators, ``dict``-views, etc.):
+Several objects of type ``Series``, ``Index`` or ``np.ndarray`` can be arbitrarily combined in a list-like container (including iterators, ``dict``-views, etc.):


"Several objects of type Series, Index, or np.ndarray" - are you sure that's what you meant to say? I think that should be reworded.

gfyoung · 2018-08-09T20:03:28Z

pandas/core/strings.py

+                    join_warn = join_warn or wnx
+
+                    if depr_warn:
+                        warnings.warn('list-likes other than Series, Index or '


Nit: add a comma after "Index"

pep8speaks · 2018-08-09T21:27:16Z

Hello @h-vetinari! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on August 28, 2018 at 17:09 Hours UTC

codecov · 2018-08-09T21:27:52Z

Codecov Report

❗ No coverage uploaded for pull request base (master@fa47b8d). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #22264   +/-   ##
=========================================
  Coverage          ?   92.03%           
=========================================
  Files             ?      169           
  Lines             ?    50785           
  Branches          ?        0           
=========================================
  Hits              ?    46742           
  Misses            ?     4043           
  Partials          ?        0

Flag	Coverage Δ
#multiple	`90.44% <100%> (?)`
#single	`42.22% <0%> (?)`

Impacted Files	Coverage Δ
pandas/core/strings.py	`98.63% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa47b8d...7e74d80. Read the comment docs.

jreback

looks fine, ex comments on the text.rst

jreback · 2018-08-09T22:35:35Z

doc/source/text.rst


 .. ipython:: python

    s
    u
-    s.str.cat([u.values, ['A', 'B', 'C', 'D'], map(str, u.index)], na_rep='-')
+    s.str.cat([u.values, u, u.index.astype(str)], na_rep='-')


can you save these in another variable, rather than doing the combination in-line. its easier to see it printed out (with comments)

What do you mean by "comments"?

jreback · 2018-08-09T22:35:41Z

doc/source/text.rst


 All elements must match in length to the calling ``Series`` (or ``Index``), except those having an index if ``join`` is not None:

 .. ipython:: python

    v
-    s.str.cat([u, v, ['A', 'B', 'C', 'D']], join='outer', na_rep='-')
+    s.str.cat([u, v, u.values], join='outer', na_rep='-')


In this case, u and u.values will have been shown 3 lines above, and v directly above. I think this should be plenty, no?

use w as you defined above

The point here is to show how join works with objects that have no index, so can't use w (which has an index).

jreback · 2018-08-10T10:06:59Z

doc/source/text.rst


 .. ipython:: python

    s
    u
-    s.str.cat([u.values, ['A', 'B', 'C', 'D'], map(str, u.index)], na_rep='-')


put some comments here, this is just too much code in a row

From my POV, it's just one line of code, while showing all the ingredients. But I'm gonna change it back to just using variants of u in different "boxes".

I think it's fair enough to show just u and not u.values and pd.Index(u) as well.

so pls show u.values, I have no idea what it actually is, and as I said, pls either add comments or split up showing s, u, u.values, this is just too long

@jreback u is shown directly above...?! Also in the screenshot I posted - in context it's IMO very clear (especially as it's at the end of the concatenation section, where the reader has seen the same objects 10 times already).

I can of course show both u and u.values, but I thought that understanding the relationship between a Series and its .values would be safe to expect of a users pandas-fu.

jreback · 2018-08-10T10:07:09Z

doc/source/text.rst


 All elements must match in length to the calling ``Series`` (or ``Index``), except those having an index if ``join`` is not None:

 .. ipython:: python

    v
-    s.str.cat([u, v, ['A', 'B', 'C', 'D']], join='outer', na_rep='-')
+    s.str.cat([u, v, u.values], join='outer', na_rep='-')


use w as you defined above

jreback · 2018-08-10T10:45:18Z

maybe would be helpful to show a rendered version here of text.rst

h-vetinari · 2018-08-10T15:13:38Z

@jreback here you go:

h-vetinari · 2018-08-20T03:45:15Z

@jreback @TomAugspurger ping :)

jreback · 2018-08-20T10:32:03Z

doc/source/text.rst


 .. ipython:: python

    s
    u
-    s.str.cat([u.values, ['A', 'B', 'C', 'D'], map(str, u.index)], na_rep='-')


so pls show u.values, I have no idea what it actually is, and as I said, pls either add comments or split up showing s, u, u.values, this is just too long

jreback · 2018-08-20T10:33:03Z

doc/source/whatsnew/v0.24.0.txt

@@ -474,12 +474,14 @@ Other API Changes
 Deprecations
 ~~~~~~~~~~~~

- :meth:`DataFrame.to_stata`, :meth:`read_stata`, :class:`StataReader` and :class:`StataWriter` have deprecated the ``encoding`` argument.  The encoding of a Stata dta file is determined by the file type and cannot be changed (:issue:`21244`).
- :meth:`MultiIndex.to_hierarchical` is deprecated and will be removed in a future version  (:issue:`21613`)
+- :meth:`DataFrame.to_stata`, :meth:`read_stata`, :class:`StataReader` and :class:`StataWriter` have deprecated the ``encoding`` argument. The encoding of a Stata dta file is determined by the file type and cannot be changed (:issue:`21244`)


pls don't edit other things. The longer the PR the more time things will take. I guess since its already done leave it.

jreback · 2018-08-20T10:33:26Z

doc/source/whatsnew/v0.24.0.txt

 - :meth:`Series.ptp` is deprecated. Use ``numpy.ptp`` instead (:issue:`21614`)
 - :meth:`Series.compress` is deprecated. Use ``Series[condition]`` instead (:issue:`18262`)
 - :meth:`Categorical.from_codes` has deprecated providing float values for the ``codes`` argument. (:issue:`21767`)
 - :func:`pandas.read_table` is deprecated. Instead, use :func:`pandas.read_csv` passing ``sep='\t'`` if necessary (:issue:`21948`)
+- :meth:`Series.str.cat` has deprecated using arbitrary list-likes *within* list-likes. A list-like container may still contain
+  arbitrarily many ``Series``, ``Index`` or 1-dimenstional ``np.ndarray``, or alternatively, only scalar values. (:issue:`21950`)


remove the word arbitrarily (and in text.rst)

jreback · 2018-08-20T10:33:52Z

pandas/core/strings.py

-        others : Series, DataFrame, np.ndarray, list-like or list-like of
-            objects that are either Series, np.ndarray (1-dim) or list-like
+        others : Series, Index, DataFrame, np.ndarray, list-like or list-like
+            of objects that are Series, Index, np.ndarray (1-dim)


or np.ndarray

h-vetinari · 2018-08-20T13:17:22Z

@jreback Incorporated feedback, except the u.values part, where my comment above doesn't show up (so copying it here):

so pls show u.values, I have no idea what it actually is, and as I said, pls either add comments or split up showing s, u, u.values, this is just too long

u is shown directly above...?! Also in the screenshot I posted. I can of course show both u and u.values, but I thought that understanding the relationship between a Series and its .values would be safe to expect of a users' pandas-fu.

In context it's IMO very clear (especially as it's at the end of the concatenation section, where the reader has seen the same objects 10 times already).

h-vetinari · 2018-08-21T06:15:08Z

@jreback Same screenshot as above. IMO the relationship u <--> u.values does not need to be explained - if you disagree with that, please let me know. (side note: the example in the current docs is even more complicated; but ultimately its easy to grok from the result what's the content of the inputs -- see [70] below)

jreback · 2018-08-22T12:42:35Z

ok this is fine. can you rebase once more and ping on green.

h-vetinari · 2018-08-23T05:47:13Z

@jreback All green.

h-vetinari · 2018-08-28T05:36:49Z

@jreback

ok this is fine. can you rebase once more and ping on green.

All green. Just fixed an indentation error on the way.

gfyoung

cc @jreback

jreback

tiny typo, ping on push.

jreback · 2018-08-28T12:19:36Z

doc/source/whatsnew/v0.24.0.txt

 - :meth:`Series.ptp` is deprecated. Use ``numpy.ptp`` instead (:issue:`21614`)
 - :meth:`Series.compress` is deprecated. Use ``Series[condition]`` instead (:issue:`18262`)
 - The signature of :meth:`Series.to_csv` has been uniformed to that of doc:meth:`DataFrame.to_csv`: the name of the first argument is now 'path_or_buf', the order of subsequent arguments has changed, the 'header' argument now defaults to True. (:issue:`19715`)
 - :meth:`Categorical.from_codes` has deprecated providing float values for the ``codes`` argument. (:issue:`21767`)
 - :func:`pandas.read_table` is deprecated. Instead, use :func:`pandas.read_csv` passing ``sep='\t'`` if necessary (:issue:`21948`)
+- :meth:`Series.str.cat` has deprecated using arbitrary list-likes *within* list-likes. A list-like container may still contain
+  many ``Series``, ``Index`` or 1-dimenstional ``np.ndarray``, or alternatively, only scalar values. (:issue:`21950`)


dimensional

h-vetinari · 2018-08-28T17:10:34Z

@jreback

tiny typo, ping on push.

Side note: [ci skip] does not seem to be respected by the new CircleCI.

h-vetinari · 2018-08-31T06:26:55Z

@jreback

This is all done, approved and green.

jreback · 2018-08-31T10:24:54Z

Side note: [ci skip] does not seem to be respected by the new CircleCI.

I don't like these skip's generally. they are not necessary as all of the CI's cancel prior builds anyhow.

jreback · 2018-08-31T10:25:44Z

thanks @h-vetinari

h-vetinari · 2018-09-04T23:11:55Z

@jreback

Side note: [ci skip] does not seem to be respected by the new CircleCI.

I don't like these skip's generally. they are not necessary as all of the CI's cancel prior builds anyhow.

This was not about cancelling prior builds but not having to run the full CI suite if (say) a single comment (like for the GH number) gets added. I view the CI as somewhat limited resources (appveyor recently having something like 12h waiting time), so [ci skip] can be useful for avoiding a CI run where it's pointless (any code/doc changes should of course run through).

h-vetinari force-pushed the depr_str_cat_LLoLL branch 2 times, most recently from 0eeff33 to 372cc24 Compare August 9, 2018 18:24

gfyoung added Strings String extension data type and string data Deprecate Functionality to remove in pandas labels Aug 9, 2018

gfyoung reviewed Aug 9, 2018

View reviewed changes

gfyoung requested review from TomAugspurger and WillAyd and removed request for WillAyd August 9, 2018 20:03

h-vetinari force-pushed the depr_str_cat_LLoLL branch from 16b0b03 to 1ab981d Compare August 9, 2018 21:28

jreback requested changes Aug 9, 2018

View reviewed changes

jreback added this to the 0.24.0 milestone Aug 9, 2018

jsexauer mentioned this pull request Aug 9, 2018

DEPR: Clean up list of deprecations from prior versions #6581

Closed

1 task

jreback requested changes Aug 10, 2018

View reviewed changes

jreback requested changes Aug 20, 2018

View reviewed changes

jreback approved these changes Aug 22, 2018

View reviewed changes

DEPR: list-likes of list-likes in str.cat

a3ea6fc

h-vetinari force-pushed the depr_str_cat_LLoLL branch from dd74c2e to a3ea6fc Compare August 22, 2018 15:32

h-vetinari force-pushed the depr_str_cat_LLoLL branch 2 times, most recently from 730ab79 to 80547eb Compare August 27, 2018 18:15

Move warning out of loop

d6b8a68

h-vetinari force-pushed the depr_str_cat_LLoLL branch from 80547eb to d6b8a68 Compare August 27, 2018 18:21

h-vetinari added 2 commits August 27, 2018 20:31

Merge remote-tracking branch 'upstream/master' into depr_str_cat_LLoLL

71179bb

Re-trigger CI after failed job

31fd9e4

gfyoung approved these changes Aug 28, 2018

View reviewed changes

jreback approved these changes Aug 28, 2018

View reviewed changes

Typo [ci skip]

7e74d80

h-vetinari force-pushed the depr_str_cat_LLoLL branch from bf66ce3 to 4a2b33b Compare August 28, 2018 17:10

h-vetinari force-pushed the depr_str_cat_LLoLL branch from 4a2b33b to 7e74d80 Compare August 28, 2018 17:10

jreback merged commit 98fb53c into pandas-dev:master Aug 31, 2018

h-vetinari deleted the depr_str_cat_LLoLL branch September 2, 2018 00:03

h-vetinari mentioned this pull request Sep 18, 2018

CI: Fail doc build on warning #22743

Closed

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

DEPR: list-likes of list-likes in str.cat (pandas-dev#22264)

d2d7b4a

h-vetinari mentioned this pull request Nov 15, 2018

DOC: update str.cat example #23723

Merged

h-vetinari mentioned this pull request Jan 9, 2019

API: capabilities of df.set_index #24046

Open

7 tasks

h-vetinari mentioned this pull request Mar 10, 2019

DEPR/API: disallow lists within list for set_index #24697

Closed

5 tasks

h-vetinari mentioned this pull request Jul 26, 2019

DEPR: execute deprecations for str.cat in v1.0 #27611

Merged

3 tasks

jreback mentioned this pull request Jul 27, 2019

DEPR: deprecations log for removed issues #13777

Closed

DEPR: list-likes of list-likes in str.cat #22264

DEPR: list-likes of list-likes in str.cat #22264

Conversation

h-vetinari commented Aug 9, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Aug 9, 2018 • edited Loading

Comment last updated on August 28, 2018 at 17:09 Hours UTC

codecov bot commented Aug 9, 2018 • edited Loading

Codecov Report

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 10, 2018

h-vetinari commented Aug 10, 2018

h-vetinari commented Aug 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari commented Aug 20, 2018

h-vetinari commented Aug 21, 2018

jreback commented Aug 22, 2018

h-vetinari commented Aug 23, 2018

h-vetinari commented Aug 28, 2018

gfyoung left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

h-vetinari commented Aug 28, 2018

h-vetinari commented Aug 31, 2018

jreback commented Aug 31, 2018

jreback commented Aug 31, 2018

h-vetinari commented Sep 4, 2018

pep8speaks commented Aug 9, 2018 •

edited

Loading

codecov bot commented Aug 9, 2018 •

edited

Loading