[ADD] Dockerized Odoo OpenUpgrade methodology

[lasley]

Docker Odoo Upgrade

This image upgrades Odoo major versions using OpenUpgrade. This is an
experimental upgrade methodology, and care should be taken if using for
production environments.

This repository is tagged by the Odoo version that is being targeted, and
you can only upgrade one version at a time. Each tag is represented by a
Dockerfile, prefixed with the tag name. For example, to upgrade version
8 to version 10, you need to use the:

• 9.0 image first
• 10.0 image next

Basic Instructions

High level usage of this image is as follows:

• Run the image with the following options:
  • Mount the root of your current Odoo file store into the container as /var/lib/odoo_old
    • Note that this should not the be filestore directory, but the one that contains the
      filestore and sessions directories (among others).
  • Mount your new Odoo file store into the container as /var/lib/odoo
    • You can easily use the old file store path if you wish
    • Note that this should not the be filestore directory, but the one that contains the
      filestore and sessions directories (among others).
  • Link your current PostgreSQL container as db_old
  • Link your new PostgreSQL container as db
    • You can easily use your old container if you wish. This is primarily to facilitate PSQL
      major version upgrades.
  • Add environment variables for the PostgreSQL credentials (PGUSER, PGPASSWORD)
    • Add environment variables for the old PostgreSQL db. These will default to the new
      db credentials if not defined. (PGUSER_OLD, PGPASSWORD_OLD)
  • Add environment variables for the source and target database names (DB_SOURCE, DB_TARGET)

Given the above, and assuming the below:

• You have a database container named postgresql
  • There is a user named odoo_user with the password odoo_password that has superuser
    access to the database.
• You have an Odoo v8 database named odoo_v8
• You have an Odoo v8 filesystem on your host at /var/lib/odoo_v8
• You want to upgrade to Odoo v10

You would run the following commands:

export PG_USER=odoo_user
export PG_PASSWORD=odoo_password

# Migrate v8 to v9
docker run \
    --link postgresql:db \
    --link postgresql:db_old \
    -v /var/lib/odoo_v8:/var/lib/odoo_old \
    -v /var/lib/odoo_v9:/var/lib/odoo \
    -e "PGDATABASE=odoo_v9" \
    -e "PGUSER=${PG_USER}" \
    -e "PGPASSWORD=${PG_PASSWORD}" \
    -e "DB_SOURCE=odoo_v8" \
    -e "DB_TARGET=odoo_v9" \
    laslabs/odoo-upgrade:9.0
# Migrate v9 to v10
docker run \
    --link postgresql:db \
    --link postgresql:db_old \
    -v /var/lib/odoo_v9:/var/lib/odoo_old \
    -v /var/lib/odoo_v10:/var/lib/odoo \
    -e "PGDATABASE=odoo_v10" \
    -e "PGUSER=${PG_USER}" \
    -e "PGPASSWORD=${PG_PASSWORD}" \
    -e "DB_SOURCE=odoo_v9" \
    -e "DB_TARGET=odoo_v10" \
    laslabs/odoo-upgrade:10.0

# Whatever command to start your v10 environment goes below.
odoo

Running the above, you will be left with an Odoo v9 and an Odoo v10 database &
filesystem. They will be named odoo_v9 and odoo_v10 respectively.

You typically want to run one migration at a time, and validate that it works before
proceeding to the next one. Living on the edge can be fun though.

Docker-Compose Instructions

The above section is mostly instructional usage, in that it uses plain Docker and
does not consider custom addons that you may have in your installation. Instead, you
would want to use a docker-compose.yml file similar to the below:

version: "2"

services:

  postgresql:
    image: postgres:9.6-alpine
    environment:
      PGDATA: /var/lib/postgresql/data/pgdata
      POSTGRES_PASSWORD: odoo_password
      POSTGRES_USER: odoo_user
    volumes:
    - ~/postgresql_data/:/var/lib/postgresql/data/

  # This is your original Odoo v8 deploy.
  # Use this as a reference for changes that need to be made in other sections
  # in order to match your actual deploy. An environment file makes this easier,
  # but was excluded in the interest of a one file example.
  #
  # odoo_v8:
  #  image: your_organization/your_odoo_scaffold_image:8.0
  #  environment:
  #    - PGDATABASE: odoo_v8
  #    - PGPASSWORD: odoo_password
  #    - PGUSER: odoo_user
  #  volumes:
  #    - ~/odoo_v8_data:/var/lib/odoo
  #  links:
  #    - postgresql:db

  openupgrade_v9:
    image: laslabs/odoo-upgrade:9.0
    volumes:
      - ~/odoo_v8_data:/var/lib/odoo_old
      - ~/odoo_v9_data:/var/lib/odoo
    environment:
      - DB_SOURCE: odoo_v8
      - DB_TARGET: odoo_v9
      - PGDATABASE: odoo_v9
      - PGUSER: odoo_user
      - PGPASSWORD: odoo_password
    links:
      - postgresql:db
      - postgresql:db_old

  custom_upgrader:
    image: your_organization/your_odoo_scaffold_image:9.0
    command: |
      odoo --workers 0 \
           --limit-time-cpu 0 \
           --limit-time-real 0 \
           --stop-after-init \
           --update all
    volumes:
      - ~/odoo_v9_data:/var/lib/odoo
    environment:
      - PGDATABASE: odoo_v9
      - PGUSER: odoo_user
      - PGPASSWORD: odoo_password
    links:
      - postgresql:db

  custom_upgrader:
    image: your_organization/your_odoo_scaffold_image:9.0
    volumes:
      - ~/odoo_v9_data:/var/lib/odoo
    environment:
      - PGDATABASE: odoo_v9
      - PGUSER: odoo_user
      - PGPASSWORD: odoo_password
    links:
      - postgresql:db

After running the above, you will be left with a running Odoo v9 instance
that was upgraded from your Odoo v8 instance.

Using a Live Instance

This image provides the ability to obtain an Odoo backup on the fly, instead
of having to mount the old volume and database. To enable this functionality,
you should set the following environment variables on the upgrader instance:

• ODOO_URI_OLD: The base URI of the old Odoo instance, including http(s)://
• ADMIN_PASSWORD_OLD: The database administration password for the old Odoo
  instance.

The Odoo instance that you provide must have the capability of delivering a
database backup through its web interface. This is usually not possible in
production due to time limits, but that is easy enough to work around and out
of scope here.

Assuming the following:

• You have a PostgreSQL container named postgresql running
• You want your new file store to be located at ~/odoo_v10_test on your host
• The old Odoo is located at https://odoo.example.com
• The old Odoo database is named odoo_v9
• You want to name the new Odoo database odoo_v10
• Your PostgreSQL user and password are both odoo

Then a docker run command to upgrade a v9 to v10 would look something like this:

docker run \
       -e "DB_SOURCE=odoo_v9" \
       -e "DB_TARGET=odoo_v10" \
       -e "PGDATABASE=odoo_v10" \
       -e "PGUSER=odoo" \
       -e "PGPASSWORD=odoo" \
       -e "ODOO_URI_OLD=https://odoo.example.com" \
       -e "ADMIN_PASSWORD_OLD=test" \
       --link postgresql:db \
       -v ~/odoo_v10_test:/var/lib/odoo \
       laslabs/odoo-upgrade:10.0

You would subsequently launch your v10 environment as normal, making sure
to configure the data_dir and db_host parameters appropriately.

Upon first launch, your custom modules will be upgraded. This is because the
upgrade image is only scoped to Odoo core modules, but makes sure to flag the
remainder of the modules for upgrade as well.

[pedrobaeza]

We are already making migrations with Doodba, but make a more general approach, as you frequently don't perform the migration on the same system. What we make basically is:

• Prepare the scaffolding, with the needed addons + repositories - private or public -, and using openupgrade as Odoo source.
• Export DB and filestore on old version.
• Import DB and filestore on new version instance.
• Perform update all (migration).
• Switch Odoo repository to OCB/odoo.

[lasley]

Thanks for the elaboration @pedrobaeza. I need something dedicated though unfortunately, otherwise I will not have an upgrade strategy for my SaaS.

My current testing here is to see if an upgrade can take place on core using just a standard OpenUpgrade install as you see here, with the final step of the upgrade taking place on the real Odoo instance on a proper scaffold. Something like this:

OldOdoo => OpenUpgrade --update all --stop-after-init => NewOdoo --update all --stop-after-init => NewOdoo

[lasley]

@pedrobaeza - I updated the instructions in my main PR comment.

Basically the goal of this is to provide one centralized upgrade-centric image, then subsequently nail the rest of the contextual upgrades via the scaffolds themselves.

This is operating under the assumption, however, that OpenUpgrade is scoped only to core Odoo & that custom modules should contain their own migration scripts that would be executed via the standard update mechanisms. Is this assumption incorrect?

cc @yajo

[yajo]

I like the idea, although I'm thinking that upgrading separately core addons from extra/private addons does not seem a very good idea... 😕

Apart from the extra time, there's the problem of Odoo being more strict on errors that OpenUpgrade. A normal odoo instance with some addons upgraded and some others not, could easily just not even boot, don't you think?

[yajo]

It's common that when migrating an addon from, say v8 to v9, the migration scripts from versions 7 and older are removed.

So, I guess you'd have to add 2 extra commands here that build the new project and upgrade it.

[lasley]

I'm not following? You're meant to step up one version at a time, so I don't think this would be an issue? Assuming upgrading to v9, I would expect the v8=>v9 migration scripts in v9 branch (of OpenUpgrade for core stuff, and the addon dirs themselves for custom). Am I wrong there?

[yajo]

No, I mean the migration scripts in the addon itself, if any.

[yajo]

This should only work when having installed module_auto_update, and when that addon is already migrated to the new odoo version, right?

[lasley]

Ahhh crap good call. I'll switch this to a standard odoo --update=all --workers=0 then.

I also submitted Tecnativa/doodba#113 to add a note of such

[yajo]

Also, since you usually will want to CI/CD your updates, and since usage of this needs access to a docker socket, maybe we could consider the upgrade mechanism a part of doodba-qa?

[pedrobaeza]

Does this method allows to prepare a scaffolding similar to doodba? That's the main reason why we use the regular doodba method, because a migration consists not only on OpenUpgrade, but the rest of the OCA modules (or even private ones) that contains migration scripts.

I still don't see the reasoning behind this... even for SaaS.

[lasley]

Apart from the extra time,

We're talking maybe 15 extra minutes on an Odoo instance that is not live. Comparing this to the cost of having to maintain upgrade images for all of my scaffolds, it's a clear win I think.

It's also worth noting that there are zero out of the box solutions (other than EE) to upgrade an Odoo major version. I think this is a serious problem in the community, and it is causing me issues in discussions with potential customers. While the process may be simple to us, it's an absolute black box for anyone not deeply familiar with Odoo, OCA, and OpenUpgrade specifically.

there's the problem of Odoo being more strict on errors that OpenUpgrade

I'm not sure I'm following here. What errors is OpenUpgrade getting rid of?

A normal odoo instance with some addons upgraded and some others not, could easily just not even boot, don't you think?

My testing so far is pointing towards this not being the case. The Odoo instance missing the custom addons will simply spit out some warnings stating that it is ignoring those addons when booting. The subsequent --update=all on the scaffold image also works fine and updates all the custom addons.

I think this works because of the dependency graph. While updating with the core modules, we can be naive of the custom ones entirely. A typical upgrade process works up this dependency chain, so by the time the custom modules are touched during a standard process, the core ones they depend on have been upgraded already. The only thing we're doing differently here is double-upgrading core (once with OpenUpgrade core, then another time on the new scaffold in order to update everything else).

Taking this a step further, this is how the Odoo Enterprise upgrade works, is it not? I don't think you supply Odoo SA with all your custom addons in order to facilitate that upgrade. I could be wrong there though.

Also, since you usually will want to CI/CD your updates,

Agreed, but I think that serves a different purpose - both CI and CD would be for the scaffold testing and production deploy respectively. In a typical CI/CD cycle, we would want to end up replacing the previous instance(s) with ones running on a new image by using a fully automated process (other than our standard software QA cycle).

The paradigm with Odoo major version upgrades is different - at a minimum you will end up with 2:1 ratio of instances upgraded vs production instances (a testing and an eventual prod). The v11 migrations (which aren't done) really skew this ratio because of failed upgrades, as do custom addons.

When it boils down to it, an Odoo major version migration will basically always require manual verification. It would be extremely reckless in my opinion to blindly replace a production Odoo vX with a production vX+1. The goal of this project is to help facilitate the needs of this workflow in as universal of a way as possible.

This is a stepping stone to a larger picture though. In one of the systems I need to be able to offer a backup upload mechanism + a form to configure options. My tool will spit out a working, testable instance on the fly. In this light, my upgrade mechanism must be completely different from the actual image in order for me to keep from having to build an image for each upgrade.

and since usage of this needs access to a docker socket

You only need link instances with the standard links (or external_links with Rancher and external stacks). No access to the socket is required by any of the containers.

Does this method allows to prepare a scaffolding similar to doodba? That's the main reason why we use the regular doodba method, because a migration consists not only on OpenUpgrade, but the rest of the OCA modules (or even private ones) that contains migration scripts.

I still don't see the reasoning behind this... even for SaaS.

This mechanism is meant to play intermediary between two Doodba (or any other Odoo) images. The custom addons exist on both sides of the equation, with the missing piece being OpenUpgrade itself + the filestore/database copies.

The proposed process from a high level is:

1. Have a working Odoo vX instance using YourScaffold vX
2. Clone the filesystem and database to a new one
3. Launch OpenUpgrade core vX+1 and upgrade the core addons
4. Launch YourScaffold vX+1 and upgrade the core addons again + the custom ones
5. Launch YourScaffold vX+1 for manual verification

Numbers 2 and 3 from the above list are within scope of this image. 2-5 is all being handled in the sample docker-compose though, as well as our Rancher template.

Edit: better words

[yajo]

Thanks for the elaboration. Indeed it makes sense to have this brick into our wall. Let me explain a couple of my points:

you usually will want to CI/CD your updates

What I meant is that version updates should be tested before deployed to production. Thus, I imagine a good minimal upgrade CI pipeline would be:

1. Restore latest production db backup.
2. Perform upgrade.
3. Boot runbot.

this needs access to a docker socket

The instructions you supplied are in the form of docker (or compose) commands, and those need access to the docker socket. Since doodba-qa is designed to have such access, that's why I meant it would be a good place to put this in. If such a thing is there, one could define this pipeline in a .gitlab-ci.yml pipeline (garbage stripped, ok?):

variables:
  COMPOSE_FILE: test.yaml

stages:
- restore
- migrate
- runbot

Restore latest production database:
  stage: restore
  script:
    - docker-compose -f prod.yaml run --rm -e PGDATABASE=test backup sh -c 'restore --force && psql -d postgres -c "CREATE DATABASE $PGDATABASE" && pg_restore /mnt/src/prod.pgdump | psql' # This is a good candidate to a script called i.e. "restore"

Migrate DB to next version:
  stage: migrate
  variables:
    UPGRADE_FROM_ODOO: 8.0
  script:
    - migrate # This script would contain the migration magic, possibly expanding a little bit the instructions linked above

Boot runbot:
  stage: runbot
  script:
    - docker-compose up -d

If it all worked, you can go to runbot, download zip of database, and upload it to the new, empty, production vX+1 server 😊 (or do a new script that deploys it, and is ran only manually).

Besides, if we do this, we can leverage all the doodba power and make it flip the switch on odoo repo remotes before doing all of this, so you don't actually have to maintain your upgrade scaffolding, nor this one, and still skip the problem of upgrading twice. It would be quite more powerful for us Doodbers, although hard to use in for others I guess.

[lasley]

Thanks for the explanation, @yajo. I think we share a similar vision here, just with a different strategy.

What I meant is that version updates should be tested before deployed to production. Thus, I imagine a good minimal upgrade CI pipeline would be:

I think this is a good idea. This would allow for a demo-data upgrade test process too. Basically unit tests for upgrades.

If it all worked, you can go to runbot, download zip of database, and upload it to the new, empty, production vX+1 server 😊 (or do a new script that deploys it, and is ran only manually).

Wouldn't Runbot also have a live image to functionally review too? Similar to our current Runbot QA process?

and still skip the problem of upgrading twice.

If you really feel that this is a problem, I can add a script to simply look through the module manifest, exclude the core addons, and add a comma separated list to the --update command. I feel that this is a minor detail in the grand scheme of things though.

It would be quite more powerful for us Doodbers, although hard to use in for others I guess.

This is my problem - my scope is larger than people using our specific tools. Our community is fractured, and continuing to expand this rift is not in our best interests.

The situation from my end is that I am not currently recommending Odoo for my new (non-managed) customers due to this rift (SpreeCommerce is the recommended at the moment). There are too many specifics, and there are too many gotchas [with the major version upgrade process & packaging/installation of modules]. Version lock is an absolute real issue, and my customers are not bound to using my frameworks for their deploys. Adding another layer of specific infrastructure is what I am trying to ward against here.

The strategy I am proposing here requires only Docker. I am even in the process of making a strategy for bare metal conversions using this same image. Basically it downloads the database dump from an external Odoo, then spits the files you need out in another dump on a volume.

A web based tool could expand upon this for an upload/download migration cycle, with the final steps being a pg_restore for the db, rsync for the filesystem, and --update all on the final instance for first boot. Any junior sysadmin can understand those + they can be converted to Windows instructions easily, which is what I need.

[lasley]

I have some interesting news as I refine this further. When starting the second instance, if you omit --update=all it will actually start upgrading at the exact place OpenUpgrade leaves off.

This is because the --update flag merely sets the proper modules in the database to state="to upgrade", which includes the custom ones. This is what the appropriate part of the DB looks like once the upgrade instance has stopped:

test_v10_smd=# SELECT COUNT(*) FROM ir_module_module WHERE state = 'to upgrade';
 count 
-------
    20
(1 row)

test_v10_smd=# SELECT name FROM ir_module_module WHERE state = 'to upgrade';
                  name                   
-----------------------------------------
 website_field_autocomplete_related
 medical_physician
 medical_prescription
 medical_pathology
 website_medical
 medical_medicament
 web_theme_smdrugstore
 medical_patient_disease
 medical_medication
 medical
 medical_prescription_sale
 website_medical_patient
 medical_prescription_thread
 medical_pharmacy
 sale_medical_medicament
 website_sale_medical_medicament
 website_field_autocomplete
 medical_prescription_sale_stock
 website_sale_medical_prescription
 website_medical_prescription_order_line
(20 rows)

I subsequently started the target image as normal, which updated the rest of the modules and left me with:

test_v10_smd=# SELECT COUNT(*) FROM ir_module_module WHERE state = 'to upgrade';
 count 
-------
     0
(1 row)

test_v10_smd=# SELECT name FROM ir_module_module WHERE state = 'to upgrade';
 name 
------
(0 rows)

I now fully believe this is a valid strategy for migrating instances, and am pretty excited about the results 🚀

Edit:

Ah crap so I was looking further into the target image, and I forgot that I have a odoo --update base in the entrypoint of my images due to some major issues when dependencies shift around.

So yeah that's the solution, but as I'm looking even further I'm thinking it's still updating the addons twice.

[yajo]

Wouldn't Runbot also have a live image to functionally review too? Similar to our current Runbot QA process?

Of course, that's a runbot! I meant that you can do that if your functional tests pass.

Our community is fractured, and continuing to expand this rift is not in our best interests.

I know, I tried to fix that, but it seems just impossible right now, so we decided just to be pragmatic and start this Doodba micro-community. It's actually growing faster than expected...

Adding another layer of specific infrastructure is what I am trying to ward against here.

Well, our way has become much easier and our evolution much faster since we took the path of making decisions and setting infrastructure minimal requirements, and since it's been such a good experience, I'm just recommending you to do the same, although maybe it's not your case, of course.

If that doesn't fit you, then I'd recommend you the way of ansible-container, which gets ansible roles and uses them to build docker images. You can still use the roles in bare metal or wherever, or just use Docker as usual.

When starting the second instance, if you omit --update=all it will actually start upgrading at the exact place OpenUpgrade leaves off.

😮 Nice!

[pedrobaeza]

When starting the second instance, if you omit --update=all it will actually start upgrading at the exact place OpenUpgrade leaves off.

The only doubt I have with this is if end-migration scripts for already migrated modules are executed in that case...

[yajo]

Is that an OpenUpgrade's or Odoo's feature?

[pedrobaeza]

end-migration? Both.

[yajo]

Then I guess OpenUpgrade ones will trigger after 1st migration, and single scripts will trigger after 2nd migration

[lasley]

That's my thought as well. Do you by any chance know of some module combination I could test to see?

[pedrobaeza]

Well, base module has an end-migration in v9, v10 and v11, so you can test any of them, interrupt the process, and see if at the end disable_invalid_filters call is made.

[lasley]

Hmmmm ok so maybe I actually don't understand what an end-migration is. I also tried searching the (v11) code for disable_invalid_filters in hopes that would point me in the right direction, but came up with no results. Mind elaborating?

[pedrobaeza]

end migrations are run at the very end, when all modules have been already update. Example: https://github.com/OCA/OpenUpgrade/blob/11.0/odoo/addons/base/migrations/11.0.1.3/end-migration.py

[lasley]

Ohhhh cool so this is in OpenUpgrade - I completely get it now. I was in the odoo/odoo base module and there weren't even any migrations. I was rather confused. Thanks for the elaboration Pedro, I'll test and get back to you all!

[pedrobaeza]

The engine is in Odoo itself, but it's not used there. You can find doc on https://github.com/odoo/odoo/blob/22e458775a1757e9952bfb498b0014906825358a/odoo/modules/migration.py#L51-L74

[lasley]

It looks like end migrations are run every time the server is booted, although base is actually treated differently. Here's the relevant code vs. here is the actual migration process.

The migration process itself seems to be capsuled within its own transaction, with this module loading happening in a transaction of its own as well.

Looks safe enough to me, and if not I assert that it is a bug in Odoo. I've had plenty of servers die in the middle of a migration process, and they all came back fine once I resolved the underlying issue (usually missing dependencies before I added --update=base to all my image entrypoints)

[pedrobaeza]

OK then. It seems correct.