DOC: Reorganization of the documentation

[datapythonista]

The documentation currently has almost all the pages in the same level. Making it difficult to navigate.

Having few top-level sections that group the pages, and some hierarchy would make things simpler to find. Here there is a prototype of how having few top-level sections could look like: https://pandas-dev.github.io/pandas-sphinx-theme/pr-datapythonista_base/

Those are the pages we currently have:

10min
advanced
api
basics
categorical
comparison_with_r
comparison_with_sas
comparison_with_sql
comparison_with_stata
computation
contributing_docstring
contributing
cookbook
developer
dsintro
ecosystem
enhancingperf
extending
gotchas
groupby
indexing
index
install
internals
io
merging
missing_data
options
overview
releases
reshaping
r_interface
sparse
style
text
timedeltas
timeseries
tutorials
visualization
whatsnew/

An initial proposal of organization is:

index

install

getting_started (probably making this a 3 or 4 part tutorial is a good idea)
    overview
    10min
    basics
    dsintro
    tutorials (I'd remove this page)
    comparisons
        comparison_with_r
            r_interface (I'd remove this page)
        comparison_with_sas
        comparison_with_sql
        comparison_with_stata

user_guide
    io
    indexing
    advanced
    merging
    reshaping
    text
    missing_data
    categorical
    visualization
    computation
    groupby
    timeseries
    timedeltas
    style
    options
    enhancingperf
    sparse
    gotchas
    cookbook (not sure if user_guide is the right place, but I don't see a better one)
    ecosystem (not sure if user_guide is the right place, but I don't see a better one)

api
    splitted in #24462, names of secions and possible changes to structure to be discussed 

development
    contributing (I'd split this one)
        contributing_docstring
    extending
    developer (this could probably be merged with extending)
    internals

whatsnew/
    releases -> index

@pandas-dev/pandas-core thoughts?

[mroeschke]

+1 on top level section for the different pages.

For reference, here's scikit-learn's documentation structure which I think is very concise and similar to what you're proposing https://scikit-learn.org/stable/documentation.html

[jorisvandenbossche]

This is actually a duplicate of #6000 :-)

But it's more or less the same. I will try to take a closer look tomorrow.

Quick feedback:

• With the current content of 'basics' and 'dsintro', I think they fit better in the User Guide section (they are very long and detailed, not really "starting" material IMO)

• I think some pages in the User guide also need splitting, but I assume the above is mainly about the top-level division? (which is a good first step to discuss)

[jorisvandenbossche]

Also, I know it will be annoying to keep those PRs open for a while (and keeping up to date with changes in master), so it might be preferred to merge them rather quickly.
But, I think we should discuss whether we only want to do this after 0.24 is released, given how close we are to the release (those PRs will break links, and if we still do some further changes we might continue breaking some links, and I think it would be even more annoying to break links multiple times if that is avoidable).

But maybe we should have a chat about how to move forward with the whole doc revamp (both content, theme, integration with website, etc)?

[datapythonista]

Not sure about 0.24, you'll know better. We're already changing the links of all the api/ since api.rst was splitted, so I'd say let's try to make the structure as close as what we want for the future for 0.24 already.

Reorganizing the current pages should be quite fast, there are almost PRs for everything, except few pages that we'll require some discussion. But some changes will surely happen in the future, as it'll take much longer to split pages, and fine tune the structure.

I'd say for what we discussed during the sprint, we're all happy with having few top level sections (user guide, development, whatsnew...), and also with the theme change. My opinion is that we should implement those, and then it'll be easier to discuss about the more "advanced" things like the home page, the navigation, the integration with the website... We tried twice to work on a one-shot documentation migration, and we failed, I think making the changes with small PRs is working much better.

I'm flexible on the basics and dsintro, I'd probably keep them in the getting started for now, as there is not much more than the 10min there, and may be move them later. But also happy having them in the user guide, it's just that the getting started will look quite poor, and I think users would probably appreciate some more recommended readings to start with (and those are the best we have for now).

[mbirdi]

I'd like to help with this one if I can. I'm a regular pandas user, but I have not contributed on github much and and would need some guidance.

[datapythonista]

@mbirdi can you better look at issues tagged as "good first issue". This requires discussion, and is not the best for a first time contributor.

[mbirdi]

@datapythonista Thank you! I'll look for that "first issue" tag.

[jorisvandenbossche]

I'm flexible on the basics and dsintro, I'd probably keep them in the getting started for now, as there is not much more than the 10min there

Yeah, I certainly agree that some more content would be nice in the getting started section (shorter version of those basics, or a expanded version of the 10min).
It's just that the current content of those pages is IMO not really fitting for a getting started section. I would personally rather add a new page with proper content later to the getting started, but not feeling strongly about it.

[datapythonista]

What makes sense for me is a multipart tutorial, I'd say a 3 part, something like 10min, something like dsintro, and probably an intro to time series.

But I'd use a real dataset and not arbitrary data where possible, give an overview (without much details) of everything that pandas does, and following a logical sequence.

But that discussion is probably for later, this won't happen during the reorganization I'd say, it's a lot of work.

[neili02]

I use the online docs quite a bit and am wondering what is the rationale for the new format of the Table of Contents. The new format nicely compresses the main TOC tree into reasonable starting groups, however it detrimentally removes a third level of the tree that is very useful for quickly identifying and navigating to subsections of individual User Guide main topics.

For example the new format now only has User Guide>MultiIndex / Advanced Indexing

MultiIndex / Advanced Indexing itself is a very long page, as are most others, with many different sub-topics. Previously these subtopics were listed as part of the TOC tree. I hope this third level will return as one now has to scroll through lengthy pages to try and find desired content. This can be particularly painful depending on the device from which you access the docs.

Can't the third level be added as an expansion of the tree that occurs upon click? This would allow the new compact organization to be maintained.

Thanks for all the work being done to maintain and improve this great tool.

[datapythonista]

Can you open a new issue with your proposal (screenshots on where you'd like to expand/add those toc would be useful). This is still work in progress, and we're working on improving the navigation, but using sphinx is not trivial.

[neili02]

What I had in mind is just to add back the additional level that was previously present. For example:

User Guide now expands to reveal:

User Guide
      IO Tools (Text, CSV, HDF5, …)
      Indexing and Selecting Data
      MultiIndex / Advanced Indexing

Clicking on MultiIndex/ Advanced Indexing should show as previously

User Guide
      IO Tools (Text, CSV, HDF5, …)
      Indexing and Selecting Data
      MultiIndex / Advanced Indexing
            Hierarchical indexing (MultiIndex)
                ...
            Advanced indexing with hierarchical index
                ...

[jorisvandenbossche]

@datapythonista I think @neili02 is pointing out what I mentioned this weekend as well: for some reason the TOC now only shows two levels, instead of 3, which has the consequence you don't see anything of TOC from the page you are on.

Looking at it, I think this is due to the change here 013ae3d where we changed maxdepth from 4 to 2 for the main index.rst. So it seems this toctree depth is what determines the depth, whatever depth you put on eg user_guide/index.rst.

[datapythonista]

Commented on the PR. If the toctree in the index is affecting that, that's a bug from sphinx. I mentioned in the PR how I think could be fixed.

[mroeschke]

It appears our current https://pandas.pydata.org/pandas-docs/stable/index.html page more-or-less fits the hierarchy initially proposed. I think from that this issue can be closed, but if certain sections still should be moved we can open new issues for them individually.