Missing database columns after upgrading from CoreShop 3 to 4

[BlackbitDevs]

Q	A
Bug report?	yes
Feature request?	no
BC Break report?	yes
RFC?	yes

Assume you have a Pimcore with CoreShop 3. And let's assume that it is not the latest 3.* version. You want to upgrade to 4.*. In https://docs.coreshop.org/CoreShop/Getting_Started/Upgrade_Notes/index.html it seems easy:

CoreShop 4.0.0 is the same as 3.1.0, but with Pimcore 11 compatibility. Updating CoreShop therefore is quite easy. Since Symfony now doesn't have a full container anymore, we use Service Containers now for our Controllers. So your overwritten Controllers probably need changes.

The problem is that when you directly want to upgrade from 3.* to 4.* the migrations from current 3.* to latest 3.* are not available anymore (compare https://github.com/coreshop/CoreShop/tree/3.2/src/CoreShop/Bundle/CoreBundle/Migrations and https://github.com/coreshop/CoreShop/tree/4.0/src/CoreShop/Bundle/CoreBundle/Migrations). This means the existing database cannot be updated to the desired state.

We have now checked via bin/console coreshop:resources:create-tables coreshop --dump-sql and there are a lot of changes.

Questions:

1. Have we missed some upgrade notes which instruct to update to latest 3.* first, then run migrations and then update major version? Or does this not exist?
2. Why do migrations get deleted at all? Because Doctrine keeps track of the ones being executed anyway. So if we would keep them, above problem would not occur.
3. Is it save to execute bin/console coreshop:resources:create-tables coreshop now or how to migrate database to compatible state?

[dpfaffenbauer]

There is something wrong in the Docs, CoreShop 4.0 is the same as the latest CoreShop 3.2. If you update from 3.2 you have to be at the latest 3.2 and then go to 4.0 (and not 4.*, since 4.1 has new features already).

Have we missed some upgrade notes which instruct to update to latest 3.* first, then run migrations and then update major version? Or does this not exist?

Yes, go do latest 3.2 first

Why do migrations get deleted at all? Because Doctrine keeps track of the ones being executed anyway. So if we would keep them, above problem would not occur.

We follow the same principle as Pimcore does in these regards. We don't want to have like a 100 of migrations and we remove them therefore between major releases.

Is it save to execute bin/console coreshop:resources:create-tables coreshop now or how to migrate database to compatible state?

You can also run bin/console doctrine:schema:upgrade --dump-sql to see whats missing, as long as it doesn't DROP TABLES or COLUMNS, you should be fine. Manually review that please!

[BlackbitDevs]

Concerning Pimcore's approach I have already written somewhere that having separate install.sql and migrations does not make sense in my opinion. Because if there is a PR which adds a migration but forgets to add this to install.sql, you may have a problem on new installs. Not to speak of the problem when somebody forgets to update to latest version of the previous major release (as described above).
What is so bad about having hundreds of single migrations? Imho every database operation should be a migration. And a migration must never be deleted - Doctrine understood this because for this reason it complains if there is a migration in migration_versions table but the migration does not exist in the code anymore.

But whatever, thanks for mentioning bin/console doctrine:schema:upgrade --dump-sql, we will try.

[dpfaffenbauer]

We don't have a separate install.sql and migrations. we use Doctrine ORM for everything that is directly SQL related. We install the database directly based on the ORM configuration.

What is so bad about having hundreds of single migrations? Imho every database operation should be a migration. And a migration must never be deleted - Doctrine understood this because for this reason it complains if there is a migration in migration_versions table but the migration does not exist in the code anymore.

Its a decision we made to do it that way to have a clear history. once you follow our upgrade guide, you will not run into issues anyway.

[BlackbitDevs]

once you follow our upgrade guide, you will not run into issues anyway.

Where is this upgrade guide? This is what I asked in

Have we missed some upgrade notes which instruct to update to latest 3.* first, then run migrations and then update major version? Or does this not exist?

I only found https://docs.coreshop.org/CoreShop/Getting_Started/Upgrade_Notes/index.html and this does not tell anything that one has to upgrade to latest version of the old major version first.

[dpfaffenbauer]

@BlackbitDevs We added a migration guide: https://docs.coreshop.org/CoreShop/Getting_Started/Upgrade_Guide Anything in your opinion we should add there?

[BlackbitDevs]

Thanks for the upgrade guide, this helps a lot!

Have created #2745 to support people who have already upgraded major version without updating to latest minor version before.