User story: Why is upgrading such a chore?

[Kedarius]

I've been using InvenTree for about 4 years now, and I am quite happy with it, but for some reason the updating of it has been a chore every time I tried it. And yesterday it was even more frustrating. I've spent 5+ hours to upgrade the version from December 2023 to actual, but I am getting ahead of myself.
And btw: do not take this as a critique of Inventree, its developers or an angry rant. It is meant as just a user story....
First little background: I am using inventree on baremetal running Fedora since September 2020. The instalation was a little bit chaotic back then, and I did not have much of Django experience. But I pulled it from github, started somehow gunicorn in an virtual environment and via uswgi I managed to get it working through my nginx (that is doing the HTTP routing, TLS, etc...).
I updated it from time to time by just pulling the new version from github, invoke update, fix up the mess it made and restart the whole gunicorn/uswgi service. But more than once I had to deal with the database migration stuff not working. It was still something like:
Migration part.0104_alter_part_description is applied before its dependency part.0103_auto_20230317_0816, only the names were different. But I've always managed to get it working, the example can be found in issue #4582 .
I switched to postgres from sqlite because of some DB locking stuff ( #5936 ) and also because I hoped that it could solve this issue. This actually made the updating issue a little bit more problematic, because with sqlite I could just back up the whole Inventree directory to have a checkpoint to return to. With postgres I had to have database backups also - it is not an issue, just a thing to remember.
I also have regular backups via invoke backup and that made me feel safe that I will not lose my data, but little did I know that restoring it would be a little bit more difficult. But more about later.
And now what happen yesterday. I did last update sometimes in December 2023, so I decided that a new update would be great... So I did invoke backup, stopped the uswgi/gunicorn service, did git pull and invoke update.
And guess what, again the migration thing problems. I still do not understand how the migration stuff works exactly, but I know that this time it reported more problems than last time. I tried to reorder the files, so the dependencies were right, and I managed to get it happy and not complain about dependencies. However, that was a big mistake as the update process failed in the middle because of trying to remove some column that was not there.... And with that I made quite a mess in my DB, as some of the DB migration stuff was already done. After a few runs of the update and manually dropping columns so it can recreate them, I finally managed to get it done.
I have to say that the docs are not that useful in this state and this depth of problems (and I do not blame them for that), but one thing has been quite confusing for me. The docs mention invetree cli command, but I was not able to find it anywhere. This resulted in invoke commands complaining about not having the INVENTREE_STATIC_ROOT variable set (even though it is defined in the config.yaml). So I set it manually, and it started to complain about other INVENTREE.... variables. I've ended up with specifying them all by hand. While browsing the issues in here, I've discovered that this should be done by that mysterious inventree cli binary that I was not able to find. Also, why was it not needed in the past?
But after the database upgrade was done, the invoke update started to complain about missing yarn? So I installed yarn and it started to complain about node.js version? Well, I managed to upgrade the node.js version (there were other projects on that server using it). Somewhere in the process it also failed with some message that pointed to the fact, that I need to install jc (whatever that is).
But after some time I've managed to get it done, and it seemed that it is upgraded. The only issue was that the gunicorn/uswgi service did not start for some reason. It seems that the directory structure changed and the gunicorn must be run from somewhere else than before. I managed to get it running from the command line and when I tried to access the web UI, Django said, that I am using postgress 11 and it requires 13.... I have to say that at this point I had to take a break, because I was close to rm -rf .
With fresh head, I decided to install the new current production version and restore data from backup. That will get rid of all my weird setup from 4 years ago plagued with all my update troubles, and also make the installation a little bit more "standard". And as a bonus, maybe it will install the mysterious inventree cli binary :-) .
I decided to use the installer and see how it goes. But quite fast I've found that it supports only Ubuntu and Debian. So I was out of luck with my Fedora.
Fine, I will use docker - it will make it possible for Inventree to use whatever version of whatever it needs.... The docker installation is quite nicely described in the docs and all the docker containers' roles are described. So I've installed the docker containers via docker-compose, and started the whole thing. But for some reason, the caddy was not working for me. But I did not want to use it anyway (I already have nginx for that) so I managed to bypass it with nginx. And finally after cca 3 hours of work, I did have actual inventree back (without data). And I was not able to log in, because of django CORS setting, but that was a relatively simple thing to fix. I was feeling that I am close, all I needed is to restore the data. And I had two backups with the same data - one before the upgrade and one after the upgrade. Well, let's just say I was wrong.
I've tried the invoke restore with the backup, and it said something about not being able to drop an index. Fine, I've converted the postgre custom backup format to SQL to see what is going on. The funny thing is that in the SQL it tried to create that index not drop it. So I've dropped the database, recreated it and ran the import directly from the SQL via psql. And it worked! No dropping index error. I was feeling lucky, so I restarted the whole docker compose and logged into web UI. For a short glimpse I saw a few of my data, but after a few seconds it went to maintenance mode to do the database update. I did not think anything wrong about it, the backup I restored was taken before the upgrade. But to my surprise the update did not work, and I found a familiar "Migration part.0104_alter_part_description is applied before its dependency part.0103_auto_20230317_0816" message in the update log. I have to admit this was too much for me and I really thought about quitting.
But hey, I had the backup after the update, so there will be no need for this database migration shenanigans... So I dropped the database once more, recreated it, imported the data after upgrade via psql and restarted the whole inventree containers.
And I ended up with a log full of "Failed 'part.tasks.update_part_pricing' (west-stream-fish-william) - 'override_min'" messages. I had no idea why, the field was present in the database. I was quite frustrated, but I did not want to give up so easily. And I did get one last idea to try....
Even though the server was not starting, I was able to run docker-compose run inventree invoke commands. And I remembered that there is the export/import data to JSON feature. So I've run docker-compose .... invoke export (with the DB containing the data from the backup after the upgrade). Then I completely deleted all the postgresql data, invoke update and restarted the inventree to get it in actual but empty state. And then, finally, docker-compose ... invoke import and it imported all the data!
So for now it seems like all the data are in there (apart from access tokens) and it seems to be working. And I am looking forward to upgrading by just docker pull in the future, even though I fully expect my old friend "Migration something before something" to pop up....

I have no idea why updating inventree is always such a chore for me and why everything goes south every time. I've had fewer issues with more complicated systems. And also, would it be possible to add the json export to the invoke backup command? And maybe have an invoke restore flag to use it instead of the actual DB dump?

[matmair]

I can sympathize with trying everything that comes to mind but this sounds like a lot of issues could have been solved by reading the install/update docs fully or reaching out here instead of taking non-revocable steps like touching the database.

If your instance is complaining please fix that/reach out before upgrading, we are including warnings for that reason.
Touching migrations has a high likelihood of completely wrecking your database into an unrepairable state.
The CLI ships with the package installer - that is a different install method to bare metal and installs as a separate instance.

Docker is one of the easier deployment methods and should just work - but please read the docs before upgrading. You need to run the update command after pulling a new image version. Please feel free to reach out here when the instance throws errors/warnings.

[Kedarius]

I have read the docs as fully as I could but, I did not find anything related to the update failing. Or that for example postgres 13 is required now while it was not 5 months ago. Or that the paths changed or that now it requires node.js and yarn. I even went through release notes to see if the changes are mentioned there...

My instance is not complaining before I start the update. And when I am updating, I am trying to minimize downtime. The installer is out of question for me as it does not support Fedora. I really do not need the inventree cli binary, but it keep poping up in most of the issues that are related to similar issues here. Like #6117 .

I've read the docs that I need to do invoke update after new docker image is pulled. Trust me, I've read them through, but as I said, my installation was a very old one, way from before any installers or dockers were available. And trust me, that I would prefer not to touch the database migrations, but for some reason it seems that they are somehow badly ordered. The thing that I find strange is that they were bugging me on sqlite and then I migrated to postgresql (via json export/import) and they are bugging me still. If it would be something wrong in the database, it should have been solved by the export/import.

And about the docs. Let's say I want to restore data from backup. All I found in the docs is:
https://docs.inventree.org/en/latest/start/backup/#restore-from-backup :

To restore from a previous backup, run the following command from the shell (within virtual environment if configured):
invoke restore

There is not even mention which file it will try to restore.... And definitely not what to do when the restore fails with something like unable to DROP INDEX when replaying TOC. Just for the fun, I will try to repeat the process and see....

[SchrodingersGat]

@Kedarius if you do run into error messages please raise an issue and include the complete error log. The update process should definitely not be a chore. We have a lot of tests which run over the migration / update process, so if you have found an edge case or condition where the update fails we would love to know about it.

[Kedarius]

So just for the sake of completeness, I just tried to use docs to install new instance and restore data from backup into it...

I am starting at https://docs.inventree.org/en/latest/start/docker_install/#required-files

1. I've downloaded the 3 mentioned files
2. I've edited the .env file (url and local path)
3. docker-compose run --rm inventree-server invoke update worked without problems
4. Now I continue according the restore docs
5. I removed the backups that were created before the update and moved my backups into the backups directory
6. docker-compose run --rm inventree-server invoke restore

Finding latest backup
Restoring backup for database 'default' and server 'None'
Restoring: default-amp.twobears.cz-2024-04-20-162722.psql.bin.gz
Restore tempfile created: 696.6 KiB
Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 24, in <module>
    main()
  File "/home/inventree/InvenTree/manage.py", line 20, in main
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/management/commands/dbrestore.py", line 71, in handle
    self._restore_backup()
  File "/root/.local/lib/python3.11/site-packages/dbbackup/management/commands/dbrestore.py", line 120, in _restore_backup
    self.connector.restore_dump(input_file)
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/base.py", line 105, in restore_dump
    return self._restore_dump(dump)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/postgresql.py", line 124, in _restore_dump
    stdout, stderr = self.run_command(cmd, stdin=dump, env=self.restore_env)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running:  pg_restore --dbname=postgresql://pguser:pgpassword@inventree-db:5432/inventree --single-transaction --clean
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 3675; 2606 3346924 FK CONSTRAINT socialaccount_socialapp_sites socialaccount_social_socialapp_id_97fb6e7d_fk_socialacc inventree
pg_restore: error: could not execute query: ERROR:  relation "public.socialaccount_socialapp_sites" does not exist
Command was: ALTER TABLE ONLY public.socialaccount_socialapp_sites DROP CONSTRAINT socialaccount_social_socialapp_id_97fb6e7d_fk_socialacc;

ERROR: 1

Just for the sake of the experiment, I tried to restore the actual backup from the running instance.

1. I moved the backups I made yesterday after I've got inventree working again to the backups directory
2. docker-compose run --rm inventree-server invoke restore

Finding latest backup
Restoring backup for database 'default' and server 'None'
Restoring: default-5ebdf59cdf11-2024-04-20-204914.psql.bin.gz
Restore tempfile created: 564.8 KiB
Traceback (most recent call last):
  File "/home/inventree/InvenTree/manage.py", line 24, in <module>
    main()
  File "/home/inventree/InvenTree/manage.py", line 20, in main
    execute_from_command_line(sys.argv)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/root/.local/lib/python3.11/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 412, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/root/.local/lib/python3.11/site-packages/django/core/management/base.py", line 458, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/management/commands/dbrestore.py", line 71, in handle
    self._restore_backup()
  File "/root/.local/lib/python3.11/site-packages/dbbackup/management/commands/dbrestore.py", line 120, in _restore_backup
    self.connector.restore_dump(input_file)
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/base.py", line 105, in restore_dump
    return self._restore_dump(dump)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/postgresql.py", line 124, in _restore_dump
    stdout, stderr = self.run_command(cmd, stdin=dump, env=self.restore_env)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/dbbackup/db/base.py", line 171, in run_command
    raise exceptions.CommandConnectorError(
dbbackup.db.exceptions.CommandConnectorError: Error running:  pg_restore --dbname=postgresql://pguser:pgpassword@inventree-db:5432/inventree --single-transaction --clean
pg_restore: while PROCESSING TOC:
pg_restore: from TOC entry 217; 1259 16504 TABLE account_emailaddress inventree
pg_restore: error: could not execute query: ERROR:  role "inventree" does not exist
Command was: ALTER TABLE public.account_emailaddress OWNER TO inventree;

I even deleted all the local data, and tried again. Same result.

[Kedarius]

@Kedarius if you do run into error messages please raise an issue and include the complete error log. The update process should definitely not be a chore. We have a lot of tests which run over the migration / update process, so if you have found an edge case or condition where the update fails we would love to know about it.

As I said, my instance was very old and very specific, so I did consider that being the cause of the problem. Mainly because I raised the issue in the past, and it seemed to fix the problem....

I also did think that I am pulling some development version (even though it was master) as it installed node.js, etc....

But I also really would like:
a) to have a list of breaking changes (i.e. not supporting postgres 11 anymore, changes in paths, etc...)
b) the invoke backup command to do also the json export - it seems the most stable one with the highest chance to be restored....

Should I raise a new issue with the backup/restore stuff I posted above?

[SchrodingersGat]

Should I raise a new issue with the backup/restore stuff I posted above?

Yes, please do!

[matmair]

Master is the development version, only releases on stable and the version branches (0.13.x for example) are considered prod-ready.

The breaking changes your are mentioning are not released yet, only in master

[SchrodingersGat]

As an additional note here, it is not reasonable to expect "backup" and "restore" commands to succeed if there have been changes to the database schema (i.e. migrations) in the interim. You should always backup and restore from the same version.

[Kedarius]

I understand that, even though I think it should be able to restore from older version. I.e. do restore and then DB migration. Would I loose my instance and having only backups, it would be frustrating to try to install exactly the version it was taken from.
But that being said, the problem I am mentioning above is from the same version.

1. I installed new docker production version on my production server
2. I managed to get my data into it via json export/import
3. I did invoke backup
4. I installed new docker production version on my test server
5. I tried to restore the backup from step 3 - that means it is from the same version to the same version...

I will create a more detailed issue later.