Provide a user friendly overview of everything which is wrong with url related data and a way to fix those

[hostep]

Hi

Magento has been struggling with many many url key, url path, url rewrite problems since Magento 2 was released. Most of these things happen when using translated url keys over multiple store views.
Many of these issues are still not properly fixed in 2.3.0. But people using Magento are struggling to find out what specific url-related issues they have which can cause all kinds of unexpected things to happen in Magento.

So next to fixing all those bugs, we also need to provide an overview of all incorrectly data which has accrued over the years in buggy Magento versions. Only fixing the bugs is not enough, as they won't fix the incorrect data.

Here are some suggestions:

1. Create an overview of url key's of products which are duplicated in the same storeview (example bug which can cause this: Duplicating product with translated url keys over multiple storeviews causes non-unique url keys to be generated magento2#21737)
2. Create an overview of products with empty url key fields (example bug which can cause this: We recently imported product prices using a csv file and the import functionality in the backend of Magento, and all of the sudden around 22% of the imported products had their url key field being set to an empty string, I don't have exact steps to reproduce unfortunately)
3. Create an overview of products which have a non-empty url path field (example bug which can cause this: Don't migrate the attribute url_path from product entities data-migration-tool#541)
4. Create an overview of categories having incorrect url paths (example bug which can cause this: Moving category in hierarchy causes url_path to be incorrect when using different url keys in multiple storeviews magento2#16202)
5. Create an overview of url rewrites which haven't been generated correctly (based on category/product data + certain configuration settings) (example bug which can cause this: URL Rewrites vs multiple storeviews - a never ending battle magento2#11615)
6. ...

There are tens/hundreds of bug reports on github touching many of these pain points, sorry for only linking the ones I created myself, but I haven't much time at the moment to make an overview of all these reported bugs.

Then, next to those overviews, we also need a user friendly way of fixing all this incorrect data somehow.

I'm opening this issue since I'm getting so tired of having to deal with all these bugs day in day out. It's a wonder if a week passes by where I'm nog hearing about some issue a client of ours has which is caused by all these bugs and it's very frustrating to see that there is no way for the client to be able to fix these things by themselves, usually you have to solve it by writing a bunch of SQL queries to fix stuff directly in the database, which is no fun at all.

Thanks for considering!

[hostep]

If somebody's interested, here's some open source module I've been working on for a while which implements most of the checks mentioned above: https://github.com/baldwin-agency/magento2-module-url-data-integrity-checker
It's still experimental, performance isn't very good, but it has already proven its worthy with a couple of our clients.

[sivaschenko]

@hostep thanks for the report! Can you please elaborate on what do you mean by "overview"? Would you like it to be in the form of documentation?

[hostep]

Hi @sivaschenko!
Basically doing something like the module I wrote. I got a bit fed up with the entire url bug disaster in Magento and wrote a module myself in the mean time since I assumed it would take years before Magento/Adobe would start to consider something like this.
Anyway the module exists now, it works more or less ok-ish, is pretty slow, uses a lot memory, so is not optimised yet. At the moment I no longer need this from Magento core since it already exists as an open source module, but if something like this would be considered to be added to the core, I certainly wouldn't be opposed to it 🙂

I had this other idea to extend this data integrity checking to other parts of Magento, not only url related.
For example, at one point Magento had a bug which allowed customers addresses to be saved with a country and a specific region from another country. When such a customer has this bug in his address data, he can't proceed in the checkout because the region doesn't match the country and the error he sees is very confusing.
It would be nice to have an overview of customer addresses in the backend of Magento where you can easily find such customers and fix them as a shop owner. This prevents sales los of customers who can't proceed in the checkout due to incorrect data stored in their account which was caused by a bug in Magento some years ago.

There are probably many things which can be checked in all kind of entities in Magento. Basically all kinds of checks to verify if the data in the database makes sense and didn't get corrupted due to various bugs.

Magento/Adobe fixes bugs to prevents stuff like this from happening, which is good, but they don't seem to consider already invalid data in the database. Somehow shop owners need to be made aware of all kind of things which might not be stored correctly in the database which can cause issues for their customers on the frontend.