Upgrade Postgres to 10 for 5.33.0

[EugenMayer]

• upgrade python-dev to python3-dev since python2 is no longer available
• py-pi to pi3-pip

fixes #528 #489

For anybody in need and avoiding the complexity, you can use the DB images from
https://hub.docker.com/repository/docker/eugenmayer/mattermost-db
build by https://github.com/EugenMayer/mattermost-boilerplate - you will find versions prior 5.33.0 based on pg 9.6 and versions above base on pg 10.

Those images are unmodified variants of this repo - just publish to docker-hub for obvious convenience reasons i guess :)

[mattermod]

Hello @EugenMayer,

Thanks for your pull request! A Core Committer will review your pull request soon. For code contributions, you can learn more about the review process here.

[davidbartonau]

@EugenMayer you're a ninja :-) Worked perfectly for me on Ubuntu 18.04, although wow it's a slow build!

Is it worth adding a warning to the documentation? Upgrading the postgres version is not a "safe" operation and requires the user to do some work to backup and restore their files.

This is the procedure I followed:

# We need to delete the database files, start a fresh postgres and then restore our backup.
# Backup your postgres database.  Go to the docker-compose root.
export MM_DOCKER_DB=`/usr/bin/docker ps --filter ancestor=mattermost-docker_db --format "{{.Names}}"`
echo Docker container is ${MM_DOCKER_DB}
docker container exec -i ${MM_DOCKER_DB} /usr/local/bin/pg_dump -c -U mmuser -h localhost mattermost > /tmp/mattermost.sql

# Take down docker
docker-compose down

# Move the files ... just to be sure
mv volumes/db volumes/db.old
mkdir volumes/db

# Start just the database
docker-compose up -d db 

# Check it started
export MM_DOCKER_DB=`/usr/bin/docker ps --filter ancestor=mattermost-docker_db --format "{{.Names}}"`
docker logs ${MM_DOCKER_DB}

# Restore our database
docker container exec -i ${MM_DOCKER_DB} /usr/local/bin/psql -U mmuser -h localhost mattermost < /tmp/mattermost.sql

# Check the restore has some data
docker container exec -i ${MM_DOCKER_DB} /usr/local/bin/psql -U mmuser -h localhost mattermost -c "SELECT count(*) FROM audits;" 

# Bring up the other services
docker-compose up -d

[EugenMayer]

You can test the image from https://github.com/EugenMayer/mattermost-boilerplate which i use to build the mattermost images and publish them, since i cannot understand why mattermost is not doing exactly that.

Using a pg 10 image is using eugenmayer/mattermost-db:team-5.33.2 if you do not want to build it.

I used https://github.com/tianon/docker-postgres-upgrade to migrate from 9.6 to 10.

It works for me and i already run it production now

[craph]

Any news on the availability of Postgres 10 ?

[davidbartonau]

@craph manually edit the dockerfile per the pull request and then do a build. That should have you sorted until the powers that be merge it.

I did that and it works, EXCEPT you must upgrade your container manually. My instructions above work, or you can follow @EugenMayer instructions

[mattermod]

This PR has been automatically labelled "stale" because it hasn't had recent activity.
A core team member will check in on the status of the PR to help with questions.
Thank you for your contribution!

/cc @jasonblais @jfrerich @emilyacook

[Stasmo]

In its current state, the docker deployment in this project is not usable until this PR is merged. This project is currently broken.

[Stasmo]

Probably a different topic, but it looks like the tests that travis is running aren't capturing this issue.

[craph]

When this PR will be merged ? it seems there is no automatic way to upgrade MM to latest version .. @stylianosrigas ? @emilyacook do you have any idea ?

[cobenash]

I faced the same situation too and look forward on this pull request. :)

[craph]

@hanzei @cpanato @metanerd is it possible to have a status about this PR ?
The community needs an official and automatic way to upgrade Mattermost docker.

Thank you very much for all your work.

Best regards,

[cpanato]

@craph sorry for the delay, I'm preparing some announcements for this repository and my personal opinion is we should not have a custom DB build to run MM the normal image should do the work

but those changes looks good to me
will merge it

[EugenMayer]

@craph sorry for the delay, I'm preparing some announcements for this repository and my personal opinion is we should not have a custom DB build to run MM the normal image should do the work

but those changes looks good to me
will merge it

Thanks for merging.

I hope you do not mind the honesty, the current repository is really far of the standard an best practices. Not only the packaged Postgres DB without any reason, but even without publishing the actual image anywhere but letting people build them themself over and over again.

Happy to join a discussion though, maybe this can be ironed out! :)

[cpanato]

it is, and I know, but this repo is was a community effort and we don't officially support those things here.
I will make an announcement and make this official again because in the past was like this but got lost :/

I recommend not use the mattermost-prod-app and instead of that use mattermost/mattermost-team-edition or mattermost/mattermost-enterprise-edition those ones we support and you can find it in the mattermost-server repo.

for the other images like DB and proxy, we don't use those custom ones and we use the default images or managed DB from AWS or any other cloud provider.

If you want to help to maintain this repository I'm glad to help you to onboard and discuss next steps

[cpanato]

the images for custom DB and others as I said we don't officially give support and that is the reason we don't push those images, if you want to use you can build them and push them to your own registry.

[craph]

@craph sorry for the delay, I'm preparing some announcements for this repository and my personal opinion is we should not have a custom DB build to run MM the normal image should do the work

but those changes looks good to me
will merge it

Hello @cpanato the repository mattermost/mattermost-docker is not officially maintained ?

Is it possible to use mattermost/mattermost-team-edition with docker or k8s and do you have any documentation for that ?

It is possible to help to maintain this repo ?

Best regards,

[cpanato]

no, and was maintained by me and others as you can see https://github.com/mattermost/mattermost-docker/blob/master/MAINTENANCE.md in the past when I was not an employee.

our mattermost-operator and mattermost-helm repos uses the image from mattermost/mattermost-team-edition.

I'm open to chat with in any time we can make a zoom meeting if you want :)

[EugenMayer]

the images for custom DB and others as I said we don't officially give support and that is the reason we don't push those images, if you want to use you can build them and push them to your own registry.

@cpanato as you can see here https://github.com/EugenMayer/mattermost-boilerplate I'am already doing this https://hub.docker.com/r/eugenmayer/mattermost-app for quiet a while now.

I also maintain a rancher catalog https://github.com/EugenMayer/docker-rancher-extra-catalogs/tree/master/templates/mattermost for mattermost.

What is mattermost/mattermost-team-edition? It seems neither to be a docker hub image nor a repository, or did I fail to find it?

[cpanato]

@EugenMayer thanks for maintaining those projects!

you can find the repositories here:

• https://hub.docker.com/repository/docker/mattermost/mattermost-team-edition
• https://hub.docker.com/repository/docker/mattermost/mattermost-enterprise-edition

for example

docker pull mattermost/mattermost-team-edition:5.35.1

[craph]

@cpanato sorry for the delay but I have a question.

You said that this repository is not officially maintained but in the official documentation, I see this repository mentionned when we want to install Mattermost in Production with Docker. Here is the link :

1. https://github.com/mattermost/mattermost-server#install-on-your-own-machine
2. https://docs.mattermost.com/install/prod-docker.html#docker-setup-on-ubuntu

Is that normal ?

Moreover, you mentionned https://hub.docker.com/repository/docker/mattermost/mattermost-team-edition and I can see that this image is based on mattermost-server.

So my question is : what is the best way to install Mattermost with Docker ?

And why the dockerfile mentionned in mattermost-team-edition in dockerhub is the same for mattermost-enterprise-edition ?
https://github.com/mattermost/mattermost-server/blob/master/build/Dockerfile

[cpanato]

@craph thanks the links above is interesting because I thought that was already removed (cc @svelle @mrckndt @cwarnermm @jasonblais ) will work with my colleagues to remove this references

mattermost-team-edition and mattermost-enterprise-edition use the same dockerfile, but the only difference is the package we use to download, if is the team edition we download the team edition package.

My colleagues are working to publish a new repository that will show how to run MM with docker and other services as well

[craph]

@craph thanks the links above is interesting because I thought that was already removed (cc @svelle @mrckndt @cwarnermm @jasonblais ) will work with my colleagues to remove this references

mattermost-team-edition and mattermost-enterprise-edition use the same dockerfile, but the only difference is the package we use to download, if is the team edition we download the team edition package.

My colleagues are working to publish a new repository that will show how to run MM with docker and other services as well

Thank you very much @cpanato for your reply.

But why your colleagues are creating a new repository ? This repository is called mattermost-docker, the name is not correct ? will you be able to give us the new repository ?

Moreover about migrating from pg 9 to pg 10 I was thinking it could be a great idea to do the same as https://github.com/mattermost/mattermost-docker#upgrading-to-team-edition-30x-from-2x but I'm unable to see what is done by the option -upgrade_db_30 where I can see what is done by this argument ? I say that because of this new issue : #540

About mattermost-team-edition and mattermost-enterprise-edition, I can't see the difference about the package you use to download. The link in the Dockerfile are no difference that's why I mention that.

[craph]

@cpanato here also you have the old link regarding your comment : https://docs.mattermost.com/install/ee-install.html#installing-enterprise-edition

[craph]

My colleagues are working to publish a new repository that will show how to run MM with docker and other services as well

@cpanato, @svelle , @mrckndt , @cwarnermm , @jasonblais do you have the link of the new repository ?
And does this repository will explain how to have pg10 or higher with the latest version of MM ?

Thank you very much

[jasonblais]

@craph Yes, we are in the process of publishing the new repository soon. We're conducting internal reviews (including security reviews) after which we'll publish it and open it for the community to try and test it out. It will support postgres 10. We also plan to have migration guides for those who are using the Docker image in this repository.

As for the Docker instructions in our documentation (https://docs.mattermost.com/install/prod-docker.html#docker-setup-on-ubuntu), it is an unofficial guide maintained by the community.

We definitely acknowledge the confusion here, and hope to resolve this confusion soon.

---

@cpanato Regarding the different packages downloaded for TE and EE docker images, can we add a link or description to the DockerHub pages that explains this? Since I agree with @craph that these currently look identical.

• https://hub.docker.com/r/mattermost/mattermost-team-edition
• https://hub.docker.com/r/mattermost/mattermost-enterprise-edition

[craph]

@jasonblais Thank you very much for your reply.

Please, will you be able to give the link of the new repo here when it will be available ?

We also plan to have migration guides for those who are using the Docker image in this repository.

It's a good news. I'm using this repository for my Mattermost in production :)

If you want, I'll be available to test the migration guides ;)

Best regards,

[jasonblais]

@craph

Please, will you be able to give the link of the new repo here when it will be available ?

Absolutely. We're hoping to open the new repository soon (in a few weeks) for initial testing and feedback from the community, stay tuned!

[craph]

@jasonblais any news about the new repositories ?

Thank you very much.

[jasonblais]

@craph we're actively working on it, we're doing some internal testing and security reviews at the moment after which we'll open the new repo for community testing

[EugenMayer]

Sorry for bumping this, but since we talked a bunch about the official images, some quick notes from my side:

• i just migrated my https://github.com/EugenMayer/mattermost-boilerplate to the official docker team edition docker images from https://hub.docker.com/r/mattermost/mattermost-team-edition
• Also to an non customize postgres build postgres:10

It seems fairly easy beside:

• the exposed port is no longer 8000, it is 8065 if you are setting up a new stack. For older stacks ( just upgrades ) the old service port is kept. I personally adopted my production config to 8065 to ensure i do not run into missunderstandings later with fresh stacks.
• the plugin folder is ./plugins for new stacks and still /mattermost/plugins for older stacks. For client plugins it is relative in both cases ./client/plugins - AFAICS this does not lead to any issues (yet). I can see an probably not critical but maybe related error in the logs

7/18/2021 2:59:13 PM2021-07-18T12:59:13.527Z	error	log/log.go:181	[ERR] plugin: plugin acceptAndServe error: timeout waiting for accept	{"source": "stdlog"}
7/18/2021 2:59:13 PM2021-07-18T12:59:13.552Z	error	log/log.go:181	[ERR] plugin: plugin acceptAndServe error: timeout waiting for accept	{"source": "stdlog"}

All in one migrating to pg10 and the official image works just fine, i just wished it would all be a lot better documented :)

Thanks for giving us the team-edition official builds, very appritiated!

[jasonblais]

@EugenMayer glad to hear it helped!

@craph the new repository is now public! https://github.com/mattermost/docker. It is still a work-in-progress, and we recommend testing a fresh set up first. If you decide to migrate to the new image, please make sure to have full data backups as we haven't fully tested the migration process yet, so there may be unforeseen issues until those tests have been completed. If any questions or feedback, let us know!

[wiersgallak]

For anyone else following along, please see also see this note for more information and context of the solution.

[craph]

@jasonblais thank you very much for the update. I will test it soon and keep you posted if I have some issues.

[craph]

Hello @jasonblais ,

Sorry for the time without no news about my tests.
After reviewing the code in the docker-compose.yml I think there is something missing.
The 13 may 2020 I fix an issue about self-signed certificate : #474 for this issue : mattermost/mattermost#13059

I think, this is missing in the new repository.
Can I reopen a PR for this ?

Thank you very much.