Multi-level navigation #462

pdmosses · 2020-10-15T13:30:53Z

This PR is intended to supersede #192, and incorporates #448. It aims to fix #219, #447, and to respond to the discussion in #425 and in the comments on #192.

A summary of the background for this PR, and an overview of the code, are available here.

The rec-nav-2 branch used for this PR is based on ~~the latest~~ an outdated release of the theme (v0.3.3, October 2020).

To test rec-nav-2 locally as a remote theme:

Specify in _config.yml:

remote_theme: pdmosses/just-the-docs@rec-nav-2

Specify in Gemfile:

gem "just-the-docs", github: "pdmosses/just-the-docs", branch: "rec-nav-2"

Run:
```
bundle update
bundle exec jekyll serve
```

The proposed updated documentation pages for the theme navigation structure are available here.

To serve the rec-nav-2 documentation pages locally and display the results of the regression tests:

Download the rec-nav-2 branch

Run:

bundle update
bundle exec jekyll serve --config _config.yml,_tests.yml

- Update `_config.yml` to add plugin `jekyll-include-cache` and option `nav_cache`. - Update `just-the-docs.gemspec` to add `jekyll-include-cache`. - Add `_nav_cache_false.yml` to disable caching (for profiling). - Add `_include_tests.yml` to include `docs/tests/`. - Replace `_includes/nav.html` by `_includes/nav/*.html`. - Update `_layouts/default.html` to include `nav/{main,crumbs,toc}.html`. Run jekyll with option `--config _config.yml,_include_tests.yml` to check unchanged.

- Major update for recursive navigation. - Explain disambiguation by ancestor and section id. - Move notes for users of previous versions to footnotes.

- Move regression tests from `docs/tests` to `_tests`. - Update `_config.yml` and `_include_tests.yml`. - Configure front matter default layout for tests. - Simplify navigation for tests using recursive navigation. - Add tests for title disambiguation. Run jekyll with option `--config _config.yml,_include_tests.yml` to check.

The regression tests have been expanded, and the navigation hierarchy is deeper.

- Update the site title to show the fork branch name. - Needs to be reverted if merged.

- Update `url` to fix the internal links on the published site. - Needs to be reverted if merged.

- Rename `_include_tests.yml` to `_tests.yml`.

- Fix the check that the `ancesstor` title is in the path by enclosing it in a pair of "^"s, and changing the extension of the path to append the "^". - Add regression tests with overlapping titles to show that ancestors are now strictly checked using the separators. To do: - Titles are assumed not to contain "^", which is used to separate titles in paths. Replace "^" by newline, which cannot occur in a title.

- Check that the grandchildren with incorrect ancestor fields now appear.

- Change the separator used in navigiation paths from `^` to newline. - Add regression tests to check that ancestor titles do not match prefixes or suffixes.

pdmosses · 2020-10-20T12:21:28Z

The last three commits above remove the need for avoiding a special character (^) in titles, and add regression tests where a page title is a prefix or suffix of a potential ancestor title.

jtebert · 2020-11-19T20:36:28Z

From a quick test, I believe that this also fixes the issue I opened yesterday (#492) about incorrect breadcrumbs and table of contents in collections.

pdmosses · 2020-11-19T22:46:13Z

@jtebert thanks for testing this PR with multiple collections – I will add them to the regression tests.

pdmosses · 2020-11-20T15:33:36Z

~~N.B. If further collections are added, the TOC/breadcrumbs links in the Tests collection will be incorrect, as reported in #492.~~
The added tests for multiple collections confirm that this fixes #492.

- Add `_config_dev.yml` - Relax gem version constraints

- Invert a condition in the test for top-level pages This avoids the navigation for non-collection pages disappearing when there are no child pages.

- Add `test_collection_1` and `test_collection_2` - Update `_tests.yml` to include the extra collections TOC/breadcrumbs correctly link to the same collection.

pyorot · 2021-03-08T19:51:12Z

If I rocket-react then hopefully this gets reviewed sooner :D
I'd be interested to know the roadmap for the project, what the priorities are, and if it's still active; maybe I should @ the maintainer?
Also interested in if there's an example website based on this PR, and how ready for use it is. I was hoping it'd be merged before I needed it, but it's been 5 months since the last commit.

adfoster-r7 · 2023-03-07T12:29:56Z

I haven't investigated the specific performance bottlenecks; but I wondered if the performance could be improved with a custom opt-in _plugin. i.e. the slow parts could be handled directly in Ruby instead of liquid. It would have to be opt-in functionality though if the plugin worked - as I believe by default plugins aren't supported on Github's default gh-pages, but they're available in Github actions, folk's own CI pipelines, or local dev environments

pdmosses · 2023-03-08T17:03:47Z

@adfoster-r7 thanks for the suggestion. However, I'm not a Ruby programmer, so I wouldn't develop such a plugin myself.

I'd like to update this PR to use v0.4, then start investigating whether it still suffers from the reported performance issues. But that will have to wait until I get time for it (hopefully from late April).

mattxwang · 2023-04-20T06:23:06Z

Just circling back - slowly looking to devote more time on bigger JtD projects (instead of just maintenance). Anything I can do to help get this PR up to speed?

pdmosses · 2023-04-20T18:50:59Z

@mattxwang

Anything I can do to help get this PR up to speed?

I think it would be better to open a new PR for supporting multi-level navigation, rather than continuing this one. The conversation here has become too long for comfort. Having to scroll down through a lot of (mostly resolved) comments would surely discourage discussion and reviews.

Moreover, some sites may have been using this PR branch for the past year or two. To avoid the risk of breaking those sites, I suggest to leave the PR branch as it is now. We can also leave the PR itself open until a new PR for multi-level navigation has been approved and merged.

@peterchenadded and @adfoster-r7 have made some suggestions for reducing the build times for sites using my current implementation of multi-level navigation (which can be painfully long, even when using Jekyll 4). Version 0.4 of JtD significantly reduced the build times for some larger 3-level sites; it's conceivable that incorporating similar optimisations would also help with multi-level sites.

However, we might also consider a more radical redesign of how we specify navigation in JtD sites. I recently took a closer look at Hugo, which seems to be generally regarded as the fastest SSG currently available. My curiosity had been piqued by some (partly misleading) comments about slow builds with JtD on Jekyll Talk.

I also noticed a Hugo theme called Docsy, which supports several features that we have yet to implement in JtD – including multi-lingual sites, versioned docs, and multi-level navigation! Using the folder structure of the source files to specify the navigation structure (apart from the relative order of pages at the same level) might allow JtD to reduce multi-level build times significantly, as well as circumventing the current need of the present PR for specifying section or ancestor to disambiguate parent title resolution.

So instead of trying to "get this PR up to speed", I'd prefer us to open a fresh issue (or simply a new discussion) to decide how best to support multi-level navigation in JtD – paying special attention to build times – before opening a PR to implement that support. Would that be okay?

adfoster-r7 · 2023-04-20T22:48:55Z

@pdmosses Whatever you think is the best approach to move forwards with things works for me - I'm happy to test any new structure/format for bugs or performance comparisons as well 👍

Just as an extra datapoint; docusaurus also relies on the folder structure without needing extra frontmatter to be present when generating its sites:

Example page: https://docusaurus.io/docs/advanced/routing
Frontmatter: https://github.com/facebook/docusaurus/blob/41a52161fd3df5433ec2ed5e013743cec80a2286/website/docs/advanced/ssg.mdx

They do provide an escape hatch for specifying a slug to ignore the folder and generate a custom path though example

peterchenadded · 2023-04-21T00:57:25Z

If we are talking about performance, we need to also assess the underlying technology being used.

For example, are people actually using liquid in the markdown files?

If we instead use markdown-it an in browser js markdown converter and change to single page design, the pages will load much much faster.

We had actually gone down this path already and the build and page load performance is light and day for our growing 2.5k page site.

adfoster-r7 · 2023-04-21T08:52:49Z

are people actually using liquid in the markdown files?

I can't speak for the majority; For our docs we definitely use liquid for generating the links to other pages (example. I imagine folk are using different liquid filters. For our docs we also register a custom liquid filter to use.

If we instead use markdown-it an in browser js markdown converter and change to single page design, the pages will load much much faster.

I believe that might impact SEO and search engine indexing, as well as potentially screen readers. In my experience markdown parsers all have different semantics too, so the in-browser rendering might behave differently to Jekyll's parser - which would introduce regressions in behavior

I imagine that could probably be added on as a separate feature from multi-level navigation support either way, but I'll defer to the experts here

peterchenadded · 2023-04-21T09:53:36Z

I can't speak for the majority; For our docs we definitely use liquid for generating the links to other pages (example. I imagine folk are using different liquid filters. For our docs we also register a custom liquid filter to use.

I think we can divide parts of the page that need liquid and parts that do not need it. There is no way to remove liquid for layout files which includes for example header, nav, footer and aux links you mentioned. The actual content shown probably does not need it and pure markdown syntax is sufficient.

I believe that might impact SEO and search engine indexing, as well as potentially screen readers. In my experience markdown parsers all have different semantics too, so the in-browser rendering might behave differently to Jekyll's parser - which would introduce regressions in behavior

True, but for us our site is private and SEO was not important. Though, we still have the full.md files and should still work with SEO, if we really wanted it to.

Your right about the rendering being different. But actually markdown-it has good number of extensions. So for example if you wanted number lines for your code, you just use the extension.

Agree there will be regressions and some features will change but if build speed and page load performance is of concern, moving to SPA and not using liquid where it is not needed should be considered.

I imagine that could probably be added on as a separate feature from multi-level navigation support either way, but I'll defer to the experts here

Yes, nav is a different area and definitely needs to be cached for bigger sites. Just following the structure of the file system was good enough for us, but many will probably want front matter to define different titles. It was the worse performing when profiling but after it is fixed it is almost mandatory to fix the other two issues.

Forgot to mention search, which makes the page unresponsive for a second as it tries to load in search-data.json on every page you visit. We actually had to setTimeout and chunk the loading of that file to make the UI responsive.

mattxwang · 2023-04-24T22:25:51Z

Hi all, thanks for the lively discussion! Bigger picture, I think a new issue could be appropriate; I could also put some work in simply porting this PR over to main (on a new branch, to avoid disrupting this one).

Let me address a few quick conversation topics:

personally, I think rendering markdown client-side is significantly out of scope for this project. There are other solutions that work quite well already, and in those cases, you probably shouldn't be using Jekyll; it is intentionally a static-site generator. The "hybrid" approach that's being alluded to - code splitting and doing the equivalent of SSR/islands for SSG - is way out of scope of most Jekyll themes. I think that would be more appropriate for the modern JS stack for this, e.g. Astro/Next/11ty and associated tooling.
file-based navigation is certainly appealing! I will point out that we are somewhat limited by Jekyll itself and that our theme is intended to work out-of-the-box with GitHub Pages (not necessarily a bad thing); since file-based navigation is not first-class, we'd have to do some hackiness to get it to work. The tradeoff is that Hugo is arguably less portable (depends on who you ask). Docusaurus is becoming the gold standard for OSS docs (Stylellnt has been using it for a while, for example), so it definitely makes sense to look at what they're doing.
I do think an opt-in Ruby plugin is within the cards; pretty sure that we can have users opt-in with a _config.yml parameter and only load in the plugin/file at runtime (i.e. when compiling). This preserves backwards compatibility.

Of course, these are all my opinions, and I'm happy to hear out some other perspectives!

In terms of my personal roadmap as a maintainer, there are some prerequisites I'll do before I work on this:

merging in PRs Fix font-size scaling for text-related CSS properties by using rem instead of fixed px values; deprecate $root-font-size #1169 and Add visual tests with Storybook & Chromatic #1236
guiding in automatic theme switching (many open PRs for this, or I'll write my own)

After that, I'm happy to then dedicate my maintenance time to pushing this PR forward.

In the meantime, if someone would either like to:

fork this branch and open an up-to-date PR
or, experiment with any of the above suggestions (with the caveat that I may deem an approach out of the spirit of JtD)

feel free to do so - happy to review!

pdmosses · 2023-06-09T18:11:39Z

Bigger picture, I think a new issue could be appropriate; I could also put some work in simply porting this PR over to main (on a new branch, to avoid disrupting this one).

In any case, I think I first should move all the multi-level tests from this PR branch to (a branch of) just-the-doc-tests.

BTW, I'm currently using the combination-rec-nav branch of my fork of JtD to publish a draft proposal for multi-level navigation docs and some accompanying notes. The published website is clearly labeled as a fork of a particular branch.

Let me address a few quick conversation topics:

personally, I think rendering markdown client-side is significantly out of scope for this project.

I know nothing about it, and I wouldn't be able to contribute.

file-based navigation is certainly appealing! I will point out that we are somewhat limited by Jekyll itself and that our theme is intended to work out-of-the-box with GitHub Pages (not necessarily a bad thing); since file-based navigation is not first-class, we'd have to do some hackiness to get it to work. The tradeoff is that Hugo is arguably less portable (depends on who you ask). Docusaurus is becoming the gold standard for OSS docs (Stylellnt has been using it for a while, for example), so it definitely makes sense to look at what they're doing.

I've also looked at Material for MkDocs, which has already implemented many of the features that we intend to support in JtD (multi-level navigation, i18n, versioned docs, toggled between color schemes, …). It seems that the navigation hierarchy there can either be left implicit in the directory structure, using file names as titles – with optional front matter to override the URL and/or the title displayed in the navigation panel – or specified in a single YAML file that associates titles with file names in the intended order and nesting. Jekyll provides variables such as page.path, and I don't see why any hacking would be needed to implement the same idea in JtD. There's also Docsy Jekyll, a port of the Hugo theme Docsy.

I do think an opt-in Ruby plugin is within the cards; pretty sure that we can have users opt-in with a _config.yml parameter and only load in the plugin/file at runtime (i.e. when compiling). This preserves backwards compatibility.

I've never programmed in Ruby, and I wouldn't be able to contribute to developing such a plugin. If we'd need to maintain consistency between a plugin and a Liquid implementation of multi-level navigation, some other maintainers would need to take responsibility for that.

Of course, these are all my opinions, and I'm happy to hear out some other perspectives!

In terms of my personal roadmap as a maintainer, there are some prerequisites I'll do before I work on this:

merging in PRs Use rem instead of px for all text related CSS properties #1169 and Add visual tests with Storybook & Chromatic #1236

guiding in automatic theme switching (many open PRs for this, or I'll write my own)

After that, I'm happy to then dedicate my maintenance time to pushing this PR forward.

I have a pressing conference submission deadline at the end of June. After that, I should have time to participate in further discussion and potential implementation of multi-level navigation.

In the meantime, if someone would either like to:

fork this branch and open an up-to-date PR

I think we need an in-depth discussion of what we would like to implement before opening a new PR on this topic.

There's an open discussion #425 with the question "Navigation levels limitation?". My response is now outdated, but I think it would be easy enough to update it, as a starting point for a fresh discussion of our current ideas for multi-level navigation.

An alternative might be to revisit the open issue #219 on "Navigation doesn't work with grand-grand-children". However, the linear structure of an issue doesn't seem as convenient as a discussion with threads of comments on potential answers, and discussions have the possibility of sorting them by the date of latest activity.

ragalgut · 2023-12-02T23:44:25Z

It would be fantastic if this went ahead. A recursive navigation.

pdmosses · 2023-12-03T18:23:36Z

@ragalgut Thanks for your comment!

PR #1397 should make it much easier to add recursive navigation (by reducing dependence on the parent, grand_parent, and has_children relations).

I expect to have time this month to investigate whether the directory structure could determine the navigation hierarchy (apart from the title and nav_order, which would still be specified in front-matter). I think that would have some significant benefits, although there might also be some drawbacks.

I suggest to continue the discussion in #1277, to make it more visible.

didacrios · 2024-02-19T23:46:17Z

Hi @pdmosses is there any consensus achieved related to this point? Seen the branch from the proposal far from the current main branch of the project and conflicts raised related to this https://just-the-docs.com/migration/#moved-include

This PR is intended to supersede just-the-docs#462. The only user-level difference from just-the-docs#462 is that disambiguation of parent pages has to use either `grand_parent` or `ancestor` titles: the somewhat unnatural `section_id` and `in_section` fields are not supported. The implementation has been significantly simplified by the changes introduced in v0.7.0 of the theme. This initial draft has not yet been rigorously tested nor profiled.

pdmosses · 2024-02-24T21:55:33Z

This PR is to be superseded by #1431.

Users with websites built with the PR branch rec-nav-2 should consider migrating to the multi-level branch, as rec-nav-2 will not be updated, and may eventually be deleted.

pdmosses · 2024-03-13T22:07:48Z

Hi @pdmosses is there any consensus achieved related to this point?

@didacrios PR #1431 generalises the latest JtD release to allow any number of navigation levels. However, we cannot claim that a consensus has been achieved, as there has not been any further discussion of the adopted approach.

* Allow unlimited multi-level navigation This PR supersedes #462. The only user-level difference from #462 is that disambiguation of parent pages has to use either `grand_parent` or `ancestor` titles: the somewhat unnatural `section_id` and `in_section` fields are not supported. The implementation has been significantly simplified by the changes introduced in v0.7.0 of the theme. * Detect cyclic parenthood A page should not have a parent or ancestor with the same title. If it does, the location of the repeated link is marked by ∞, to facilitate debugging the navigation (and an unbounded loop leading to a build exception is avoided). * Add nav_error_report warning in main navigation When activated by `nav_error_report: true` in `_config.yml`, displays warnings about pages with the same title as their parent page or an ancestral page. * Cache site-nav with links to all pages The extra cached site-nav is used for determining breadcrumbs and children navigation, which may involve pages that are excluded from the main navigation. * Replace code for determining children by inclusion of components/nav/children.html * Update CHANGELOG.md --------- Co-authored-by: Matt Wang <matt@matthewwang.me>

pdmosses · 2024-08-20T20:54:42Z

Superseded by #1431

pdmosses added 7 commits October 15, 2020 12:24

Add profiles for nav-cache option

3a3c949

Update navigation-structure.md

2a64690

- Major update for recursive navigation. - Explain disambiguation by ancestor and section id. - Move notes for users of previous versions to footnotes.

Update profiles

5227847

The regression tests have been expanded, and the navigation hierarchy is deeper.

Update _config.yml

3feb543

- Update the site title to show the fork branch name. - Needs to be reverted if merged.

Update _config.yml

979f0fd

- Update `url` to fix the internal links on the published site. - Needs to be reverted if merged.

pdmosses mentioned this pull request Oct 15, 2020

recursive navigation #192

Closed

pdmosses added 5 commits October 15, 2020 17:00

Update about.md

99b4924

Rename tests config file

b9612a9

- Rename `_include_tests.yml` to `_tests.yml`.

Add a test with a title including the separator "^"

e33d834

- Check that the grandchildren with incorrect ancestor fields now appear.

Fix ancestor checks

e27e12e

- Change the separator used in navigiation paths from `^` to newline. - Add regression tests to check that ancestor titles do not match prefixes or suffixes.

pdmosses linked an issue Oct 24, 2020 that may be closed by this pull request

Allow pages together with collections #447

Closed

pdmosses linked an issue Nov 18, 2020 that may be closed by this pull request

Supports recursive display of all article lists #479

Closed

pdmosses mentioned this pull request Nov 19, 2020

TOC/breadcrumbs function incorrectly when using parent pages within collections #492

Closed

pdmosses requested a review from pmarsceill November 23, 2020 09:06

pdmosses added 2 commits December 22, 2020 08:47

Update for use with Jekyll 4.2

b75d82b

- Add `_config_dev.yml` - Relax gem version constraints

Fix navigation for sites with no child pages

9c52ed7

- Invert a condition in the test for top-level pages This avoids the navigation for non-collection pages disappearing when there are no child pages.

pdmosses mentioned this pull request Jan 4, 2021

Links to parent in 4th level child incorrectly link to the child itself instead of the parents #527

Closed

pdmosses linked an issue Jan 18, 2021 that may be closed by this pull request

Links to parent in 4th level child incorrectly link to the child itself instead of the parents #527

Closed

IamDavidovich mentioned this pull request Jan 21, 2021

Merge in branch for recursive navigation bigcommerce/just-the-docs#1

Merged

Add collections to test TOC/breadcrumbs links

e703d60

- Add `test_collection_1` and `test_collection_2` - Update `_tests.yml` to include the extra collections TOC/breadcrumbs correctly link to the same collection.

pdmosses linked an issue Feb 14, 2021 that may be closed by this pull request

TOC/breadcrumbs function incorrectly when using parent pages within collections #492

Closed

pdmosses mentioned this pull request Feb 25, 2021

Combination #578

Merged

simonebortolin added a commit to simonebortolin/just-the-docs that referenced this pull request Dec 24, 2022

Add Multi-level navigation (just-the-docs#462)

4b8d5df

simonebortolin added a commit to simonebortolin/just-the-docs that referenced this pull request Dec 27, 2022

Add Multi-level navigation (just-the-docs#462)

da2d4e3

captn3m0 mentioned this pull request Jan 4, 2023

[dep] Upgrade theme to 0.5.4 endoflife-date/endoflife.date#2243

Merged

pdmosses mentioned this pull request Feb 9, 2023

Create next_prev.html #394

Closed

pdmosses mentioned this pull request Mar 8, 2023

Add Prev and Next navigation buttons #1191

Open

mattxwang mentioned this pull request Apr 20, 2023

Resolve liquid variable leakage between components #1118

Closed

pdmosses mentioned this pull request Apr 30, 2023

Fix liquid variable leakage in navigation components #1243

Merged

pdmosses mentioned this pull request Jul 27, 2023

Parent page has_children should not require child pages parent setting #1302

Open

xinyuhoulucy mentioned this pull request Sep 1, 2023

Provena PR (Bug Fix): Fix Non-Functioning Links in Documentation provena/provena#27

Merged

6 tasks

VerzatileDev mentioned this pull request Dec 27, 2023

[ADDRESS] Child in Child in Child Content VerzatileDevOrg/Programming_HandBook#61

Open

pdmosses mentioned this pull request Feb 24, 2024

Feature: Allow unlimited multi-level navigation #1431

Merged

5 tasks

pdmosses closed this Aug 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Multi-level navigation #462

Multi-level navigation #462

pdmosses commented Oct 15, 2020 •

edited

Loading

pdmosses commented Oct 20, 2020

jtebert commented Nov 19, 2020

pdmosses commented Nov 19, 2020

pdmosses commented Nov 20, 2020 •

edited

Loading

pyorot commented Mar 8, 2021

adfoster-r7 commented Mar 7, 2023 •

edited

Loading

pdmosses commented Mar 8, 2023

mattxwang commented Apr 20, 2023

pdmosses commented Apr 20, 2023

adfoster-r7 commented Apr 20, 2023

peterchenadded commented Apr 21, 2023

adfoster-r7 commented Apr 21, 2023

peterchenadded commented Apr 21, 2023

mattxwang commented Apr 24, 2023

pdmosses commented Jun 9, 2023 •

edited

Loading

ragalgut commented Dec 2, 2023

pdmosses commented Dec 3, 2023

didacrios commented Feb 19, 2024

pdmosses commented Feb 24, 2024

pdmosses commented Mar 13, 2024

pdmosses commented Aug 20, 2024

Multi-level navigation #462

Multi-level navigation #462

Conversation

pdmosses commented Oct 15, 2020 • edited Loading

pdmosses commented Oct 20, 2020

jtebert commented Nov 19, 2020

pdmosses commented Nov 19, 2020

pdmosses commented Nov 20, 2020 • edited Loading

pyorot commented Mar 8, 2021

adfoster-r7 commented Mar 7, 2023 • edited Loading

pdmosses commented Mar 8, 2023

mattxwang commented Apr 20, 2023

pdmosses commented Apr 20, 2023

adfoster-r7 commented Apr 20, 2023

peterchenadded commented Apr 21, 2023

adfoster-r7 commented Apr 21, 2023

peterchenadded commented Apr 21, 2023

mattxwang commented Apr 24, 2023

pdmosses commented Jun 9, 2023 • edited Loading

ragalgut commented Dec 2, 2023

pdmosses commented Dec 3, 2023

didacrios commented Feb 19, 2024

pdmosses commented Feb 24, 2024

pdmosses commented Mar 13, 2024

pdmosses commented Aug 20, 2024

pdmosses commented Oct 15, 2020 •

edited

Loading

pdmosses commented Nov 20, 2020 •

edited

Loading

adfoster-r7 commented Mar 7, 2023 •

edited

Loading

pdmosses commented Jun 9, 2023 •

edited

Loading