Can we deactivate /page/ functionality from blog functionality?

[johnnyreilly]

The "Older Entries" / "Newer Entries" functionality at the bottom of blog posts leads to the creation of "pseudo" pages which allow you to step through your history of pages and duplicate blog content along the way.

Historically this was useful for SEO as it meant there was a simple way to crawl all content.

Given that we have a blog archive page in Docusaurus now, I wonder if it might be possible to turn off the generation of this. All the links that are required are covered by the sitemap and the blog archive, so is there a need for these pseudo pages?

Could we have a config option to turn them off perhaps?

Or do they cover a use case I've not considered?

Raised this on Canny also in case it belongs there: https://docusaurus.io/feature-requests/p/can-we-deactivate-page-functionality-from-blog-functionality

[slorber]

The problem is that most sites don't link to the archive page, and yet they want their older posts to remain available without having an unlimited sidebar nor a "blog archive" navbar item.

We should find a good story for those sites.

Maybe having a blogHome feature with the ability to select a list of featured blog posts could make sense? Many would prefer featuring best blog posts rather than the X last ones.

[slorber]

Hey

As far as I know the sitemap.xml is not like a RSS feed, and unless you submit it to Google manually or through an API there's no std way to tell Google with page meta what's the sitemap of your site.

I'm not even sure it's that useful, apart if you have unlinked pages that the crawler can't find naturally, or want to speed up the indexing of some new pages (for which I think you can submit URLs as JSON which is a better solution, see https://developers.google.com/search/apis/indexing-api/v3/quickstart#sitemaps).

Afaik excluding content from the sitemap is not a signal for Google to not index those pages. You also have to ensure these pages were not previously discovered, and can't be found be crawling links. Adding a noindex with swizzle is probably a better solution and automatically filters the URLs from the sitemap.

[johnnyreilly]

I didn't know about indexing-api - thanks for sharing! I explicitly removed my existing sitemap and resupplied afterwards to let Google know that things had significantly changed.

Part of the motivation was doing a search in Google and having a /tags/ page come up with the post I was after rather than the actual blog post.

My hope is that over time Google will stop favouring /tags/ and /pages/ pages now I've delisted them.

[slorber]

I'm not 100% sure removing sitemap + submitting new one is a signal for google to stop indexing those pages. They are in the Google crawler DB now, so Google doesn't need a sitemap anymore to discover them, and will just schedule new crawls and keep them in the index until they go down? 🤔 but they will eventually have a lower ranking due to removed internal links

[johnnyreilly]

Yeah - I've another trick up my sleeve. I'm going to add a noindex to all /page/ and /tag/ pages. That should give Google sufficient signal to stop listing them

https://developers.google.com/search/docs/crawling-indexing/block-indexing

[johnnyreilly]

you can see what I'm getting up to here: johnnyreilly/blog.johnnyreilly.com#350