Multiple URLs causes duplicated content

[redelschaap]

I noticed that categories, products, CMS pages and controllers load the same content with or without a trailing slash. Index controllers also load the same content with or without index folders in the URL.

This behaviour is not wanted from a SEO perspective. Multiple URLs causes duplicated pages. When search engines find two URLs for the same category, they will be treated as two URLs but marked as duplicates from each other. Canonicals can fix this problem, but even with the right canonicals it still costs your site's precious crawl budget.

The following URLs provide the same content.

Categories

• http://example.com/category
• http://example.com/category/

Products

• http://example.com/product.html
• http://example.com/product.html/

CMS

• http://example.com/page
• http://example.com/page/

Controllers

• http://example.com/contact
• http://example.com/contact/
• http://example.com/contact/index
• http://example.com/contact/index/
• http://example.com/contact/index/index
• http://example.com/contact/index/index/
• http://example.com/contact/index/index/any-random-path
• http://example.com/contact/index/index/any-random/set-of/paths

---

Ideally all pages have just one URL. All other variants must either 301 redirect to the first one or serve the 404 Not Found page.

There is also an issue with the way URLs are being generated. For example, URLs generated in the footer links have a trailing slash, while URLs generated by the Magento\Cms\Block\Widget\Page\Link widget don't. Ideally this is consistent throughout the whole website.

Maybe it's a good idea to have URL suffix config options for all types of URLs. Also make sure these config options are used correctly. I also think Issue #3872 is related to this behaviour.

[vzabaznov]

will be fixed in #4876

[magento-team]

Internal ticket to track issue progress: MAGETWO-70255

[magento-team]

Internal ticket to track issue progress: MAGETWO-70505