diff --git a/.gitignore b/.gitignore index e76b2ff0..ef28191a 100644 --- a/.gitignore +++ b/.gitignore @@ -2,6 +2,9 @@ # See https://docs.docker.com/compose/environment-variables/#the-env-file /.env +# Bash history of musicbrainz container in development mode +/.musicbrainz-dev.bash_history.d/ + # File created and managed by the script admin/create-amqp-extension /admin/.create-amqp-extension.sql diff --git a/README.md b/README.md index 88ca73f6..5020a7f9 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ search and replication in docker. * CPU: 16 threads (or 2 without indexed search), x86-64 architecture * RAM: 16 GB (or 4 without indexed search) -* Disk Space: 200 GB (or 100 without indexed search) +* Disk Space: 250 GB (or 100 without indexed search) ### Required software @@ -87,9 +87,9 @@ If you use [UFW](https://help.ubuntu.com/community/UFW) to manage your firewall: ## Components version -* Current MB Branch: [v-2023-10-24-hotfix](build/musicbrainz/Dockerfile#L53) -* Current DB_SCHEMA_SEQUENCE: [28](build/musicbrainz/Dockerfile#L129) -* Postgres Version: [12](docker-compose.yml) +* Current MB Branch: [v-2025-02-24.0](build/musicbrainz/Dockerfile#L88) +* Current DB_SCHEMA_SEQUENCE: [29](build/musicbrainz/Dockerfile#L125) +* Postgres Version: [16](docker-compose.yml) (can be changed by setting the environment variable `POSTGRES_VERSION`) * MB Solr search server: [3.4.2](docker-compose.yml#L88) (can be changed by setting the environment variable `MB_SOLR_VERSION`) @@ -152,7 +152,7 @@ tables, the server will generally fall back to slower queries in their place. If you wish to configure the materialized tables, you can run: ```bash -sudo docker-compose exec musicbrainz bash -c './admin/BuildMaterializedTables --database=MAINTENANCE all' +sudo docker-compose exec musicbrainz bash -c 'carton exec -- ./admin/BuildMaterializedTables --database=MAINTENANCE all' ``` ### Start website @@ -364,7 +364,7 @@ This number can be changed using the Docker environment variable #### Customize download server By default, data dumps and pre-built search indexes are downloaded from -`http://ftp.eu.metabrainz.org/pub/musicbrainz`. +`https://data.metabrainz.org/pub/musicbrainz`. The download server can be changed using the Docker environment variable `MUSICBRAINZ_BASE_DOWNLOAD_URL`. @@ -401,6 +401,40 @@ This can be changed by creating a custom configuration file under [and finally](https://docs.docker.com/storage/bind-mounts/#choose-the--v-or---mount-flag) setting the Docker environment variable `SIR_CONFIG_PATH` to its path. +#### Customize backend Postgres server + +By default, the services `indexer` and `musicbrainz` are trying to connect to the host `db` (for both read-only and write host) but the hosts can +be customized using the `MUSICBRAINZ_POSTGRES_SERVER` and `MUSICBRAINZ_POSTGRES_READONLY_SERVER` environment variables. + +Notes: +* After switching to another Postgres server: + * If not transferring data, it is needed to create the database again. + * For live indexing, the RabbitMQ server has to still be reachable from the Postgres server. +* The helper scripts `check-search-indexes` and `create-amqp-extension` won’t work anymore. +* The service `db` will still be up even if unused. + +#### Customize backend RabbitMQ server + +By default, the services `db`, `indexer` and `musicbrainz` are trying to connect to the host `mq` +but the host can be customized using the `MUSICBRAINZ_RABBITMQ_SERVER` environment variable. + +Notes: +* After switching to another RabbitMQ server: + - Live indexing requires to go through AMQP Setup again. + - If not transferring data, it might be needed to build search indexes again. +* The helper script `purge-message-queues` won’t work anymore. +* The service `mq` will still be up even if unused. + +#### Customize backend Redis server + +By default, the service `musicbrainz` is trying to connect to the host `redis` +but the host can be customized using the `MUSICBRAINZ_REDIS_SERVER` environment variable. + +Notes: +* After switching to another Redis server: + - If not transferring data, MusicBrainz user sessions will be reset. +* The service `redis` will still be running even if unused. + ### Docker Compose overrides In Docker Compose, it is possible to override the base configuration using @@ -421,6 +455,16 @@ Try `admin/configure help` for more information. #### Publish ports of all services +:warning: The service `search` is currently running Solr 7 in +standalone mode which is vulnerable to privilege escalation. +See [CVE-2025-24814](https://lists.apache.org/thread/gl291pn8x9f9n52ys5l0pc0b6qtf0qw1) for details. +We are working on upgrading to Solr 9 in SolrCloud mode. +See [SEARCH-685](https://tickets.metabrainz.org/browse/SEARCH-685) for follow-up. +In general, Solr is strongly recommended to be accessible to your own clients only. +See [Solr Security](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity) for details. +Similarly, other services have not been configured to be safely publicly accessible either. +Take this warning in consideration when publishing their ports. + To publish ports of services `db`, `mq`, `redis` and `search` (additionally to `musicbrainz`) on the host, simply run: @@ -546,9 +590,21 @@ Simply restart the container when checking out a new branch. This is very similar to the above but for Search Index Rebuilder (SIR): -1. Set the variable `SIR_LOCAL_ROOT` in the `.env` file +1. Optionally set the following variables in the `.env` file: + - `SIR_DEV_CONFIG_PATH` + (Default: `./default/config.ini` replacing `SIR_CONFIG_PATH`) + - `SIR_DEV_LOCAL_ROOT` + (Default: `../sir` assuming that `musicbrainz-docker` and `sir` + have been cloned under the same parent directory) + - `SIR_DEV_PYTHON_VERSION` + (Default: `2.7` matching `metabrainz/python` image tag) + - `SIR_DEV_BASE_IMAGE_DATE` + (Default: `20220421` matching `metabrainz/python` image tag) + - `SIR_DEV_VERSION` + (Default: `py27-stage1` which is informative only) 2. Run `admin/configure add sir-dev` -3. Run `sudo docker-compose up -d` +3. Run `sudo docker-compose build indexer` +4. Run `sudo docker-compose up -d` Notes: diff --git a/TROUBLESHOOTING.md b/TROUBLESHOOTING.md index 4edd2bbb..04208283 100644 --- a/TROUBLESHOOTING.md +++ b/TROUBLESHOOTING.md @@ -8,6 +8,7 @@ - [Resolving name failed](#resolving-name-failed) - [Loadable library and perl binaries are mismatched](#loadable-library-and-perl-binaries-are-mismatched) - [ImportError: No module named](#importerror-no-module-named) +- [Unknown error executing apt-key](#unknown-error-executing-apt-key) @@ -109,3 +110,29 @@ sudo docker-compose restart indexer Python packages are downloaded again and installed again when the service `indexer` restarts. + +## Unknown error executing apt-key + +When building Docker image for the service `musicbrainz`: + +``` log +Err:1 https://deb.nodesource.com/node_20.x nodistro InRelease + Unknown error executing apt-key +[...] +W: GPG error: https://deb.nodesource.com/node_20.x nodistro InRelease: Unknown error executing apt-key +E: The repository 'https://deb.nodesource.com/node_20.x nodistro InRelease' is not signed. +``` + +This may happen if your system is hindering file permissions. +You can find out by adding `RUN ls -l file` commands in the +Dockerfile. + +Solution: + +Configure your system to keep the file permissions defined in the Git repository +and to preserve the permissions of the files copied through Docker. + +If it isn’t possible, for example with the Unraid operating system, +run additional `chmod` commands in the Dockerfile; See comments to the +issue [#263](https://github.com/metabrainz/musicbrainz-docker/pull/263). + diff --git a/admin/lib/upgrade-to-postgres12/do-pg_upgrade.sh b/admin/lib/upgrade-to-postgres16/do-pg_upgrade.sh similarity index 60% rename from admin/lib/upgrade-to-postgres12/do-pg_upgrade.sh rename to admin/lib/upgrade-to-postgres16/do-pg_upgrade.sh index 179fc1dd..58f26b22 100644 --- a/admin/lib/upgrade-to-postgres12/do-pg_upgrade.sh +++ b/admin/lib/upgrade-to-postgres16/do-pg_upgrade.sh @@ -4,8 +4,8 @@ set -e shopt -s extglob PGDATA=/var/lib/postgresql/data -PGDATA_OLD="$PGDATA"/9.5 -PGDATA_NEW="$PGDATA"/12 +PGDATA_OLD="$PGDATA"/12 +PGDATA_NEW="$PGDATA"/16 PGAMQP_DIR=/tmp/pg_amqp function cleanup() { @@ -15,8 +15,8 @@ function cleanup() { fi if [[ -d "$PGDATA_OLD" ]]; then - echo "Clean $PG_DATA_OLD off but 9.5 data" - rm -rf "$PGDATA"/!(9.5) + echo "Clean $PG_DATA_OLD off but 12 data" + rm -rf "$PGDATA"/!(12) mv -v "$PGDATA_OLD"/* "$PGDATA" rmdir "$PGDATA_OLD" fi @@ -28,35 +28,38 @@ function cleanup() { } trap cleanup EXIT -sudo -u postgres /usr/lib/postgresql/9.5/bin/pg_ctl stop -w -D "$PGDATA" \ +sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl stop -w -D "$PGDATA" 2>/dev/null \ || echo 'Assuming server is stopped...' # We use the --link flag on pg_upgrade below to make hard links instead of # copying files, drastically improving the speed of the upgrade. Hard links, # of course, require the linked files to be on the same file system, but # $PGDATA is the volume *root*. To work around that, we have to move the -# existing v9.5 cluster to a '9.5' subdir, and create the new v12 cluster in -# a '12' subdir. Once we're finished, we'll move the new cluster's files +# existing v12 cluster to a '12' subdir, and create the new v16 cluster in +# a '16' subdir. Once we're finished, we'll move the new cluster's files # back into $PGDATA. cd "$PGDATA" -sudo -u postgres mkdir -p 9.5 12 -chmod 700 9.5 12 -sudo -u postgres mv !(9.5|12) 9.5 +sudo -u postgres mkdir -p 12 16 +chmod 700 12 16 +sudo -u postgres mv !(12|16) 12 -sudo -u postgres /usr/lib/postgresql/12/bin/initdb \ +sudo -u postgres /usr/lib/postgresql/16/bin/initdb \ --encoding utf8 \ --username musicbrainz \ "$PGDATA_NEW" -git clone -b "v0.4.1" --depth=1 https://github.com/omniti-labs/pg_amqp.git "$PGAMQP_DIR" +# There is no tag v0.4.2 (or 0.5.0) yet +PG_AMQP_GIT_REF="240d477d40c5e7a579b931c98eb29cef4edda164" +git clone https://github.com/omniti-labs/pg_amqp.git "$PGAMQP_DIR" cd "$PGAMQP_DIR" -make PG_CONFIG=/usr/lib/postgresql/12/bin/pg_config +git checkout "$PG_AMQP_GIT_REF" +make PG_CONFIG=/usr/lib/postgresql/16/bin/pg_config PG_CPPFLAGS=-Wno-error=implicit-int make install cd "$PGDATA" -sudo -u postgres /usr/lib/postgresql/12/bin/pg_upgrade \ - --old-bindir=/usr/lib/postgresql/9.5/bin/ \ - --new-bindir=/usr/lib/postgresql/12/bin/ \ +sudo -u postgres /usr/lib/postgresql/16/bin/pg_upgrade \ + --old-bindir=/usr/lib/postgresql/12/bin/ \ + --new-bindir=/usr/lib/postgresql/16/bin/ \ --old-datadir="$PGDATA_OLD" \ --new-datadir="$PGDATA_NEW" \ --jobs=3 \ @@ -72,9 +75,5 @@ mv "$PGDATA_NEW"/* "$PGDATA"/ rmdir "$PGDATA_NEW" cp -a "$PGDATA_OLD"/{postgresql.conf,pg_hba.conf} . -# Start the new cluster in the background, so we can apply -# 20200518-pg12-after-upgrade.sql via the website container. -sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl start -w -D "$PGDATA" - ./delete_old_cluster.sh rm delete_old_cluster.sh diff --git a/admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml b/admin/lib/upgrade-to-postgres16/pg-12-stopped.yml similarity index 66% rename from admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml rename to admin/lib/upgrade-to-postgres16/pg-12-stopped.yml index 375b8feb..501ebe73 100644 --- a/admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml +++ b/admin/lib/upgrade-to-postgres16/pg-12-stopped.yml @@ -4,8 +4,8 @@ services: db: build: args: - - POSTGRES_VERSION=9.5 - image: musicbrainz-docker_db:9.5 + - POSTGRES_VERSION=12 + image: musicbrainz-docker_db:12 command: /bin/sleep infinity musicbrainz: diff --git a/admin/repository/prebuild-db b/admin/repository/prebuild-db new file mode 100755 index 00000000..4b1933ba --- /dev/null +++ b/admin/repository/prebuild-db @@ -0,0 +1,104 @@ +#!/usr/bin/env bash + +set -e -u + +# shellcheck source=admin/lib/common.inc.bash +source "$(dirname "${BASH_SOURCE[0]}")/../lib/common.inc.bash" + +HELP=$(cat < +where tag is based on Postgres major version and a build sequence number. + +Prerequisites: + + The version of Postgres must be up-to-date in: + README.md + build/postgres/Dockerfile + build/postgres-prebuilt/Dockerfile + + The build sequence must be incremented, if image tag exists already + (for example when only files under build/postgres/ have changed), in: + build/postgres-prebuilt/Dockerfile +EOH +) + +if [[ $# -ne 0 && $1 =~ -*h(elp)? ]] +then + echo "$HELP" + exit 0 # EX_OK +elif [[ $# -ne 0 ]] +then + echo >&2 "$SCRIPT_NAME: unrecognized argument: $1" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 64 # EX_USAGE +fi + +# Retrieve destination image tag from the default Dockerfile + +POSTGRES_VERSION=$(sed -n 's/^ARG POSTGRES_VERSION=\([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile) +# shellcheck disable=SC2034 # as it used in eval below +DB_BUILD_SEQUENCE=$(sed -n 's/^ARG DB_BUILD_SEQUENCE=\([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile) +DEST_IMAGE_TAG=$(eval echo "$(sed -n 's/^FROM \([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile)") + +# Check that the Postgres major version is the same in all other files + +README_POSTGRES_VERSION=$(sed -n 's/^.*Postgres Version: \[\([^]]*\)\].*$/\1/p' README.md) + +if [[ $POSTGRES_VERSION != "$README_POSTGRES_VERSION" ]] +then + echo >&2 "$SCRIPT_NAME: File differs in Postgres major version:" + echo >&2 "$SCRIPT_NAME: * build/postgres-prebuilt/Dockerfile: '$POSTGRES_VERSION'" + echo >&2 "$SCRIPT_NAME: * README.md: '$README_POSTGRES_VERSION'" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +SOURCE_POSTGRES_VERSION=$(sed -n 's/^ARG POSTGRES_VERSION=\([^ ]*\).*$/\1/p' build/postgres/Dockerfile) + +if [[ $POSTGRES_VERSION != "$SOURCE_POSTGRES_VERSION" ]] +then + echo >&2 "$SCRIPT_NAME: File differs in Postgres major version:" + echo >&2 "$SCRIPT_NAME: * build/postgres-prebuilt/Dockerfile: '$POSTGRES_VERSION'" + echo >&2 "$SCRIPT_NAME: * build/postgres/Dockerfile: '$SOURCE_POSTGRES_VERSION'" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +# Check that the image tag doesn’t exist already + +if $DOCKER_CMD image pull "$DEST_IMAGE_TAG" 2>/dev/null \ + || $DOCKER_CMD image ls "$DEST_IMAGE_TAG" 2>/dev/null | grep -Eq "$DEST_IMAGE_TAG" +then + echo >&2 "$SCRIPT_NAME: image tag exists already: $DEST_IMAGE_TAG" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +# Do the thing + +echo Building... + +DOCKER_COMPOSE_OPTS='-f docker-compose.yml -f compose/db-own-build.yml' + +# shellcheck disable=SC2086 # intentional word splitting of options +$DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS build --progress=plain --pull db + +echo Tagging... + +# shellcheck disable=SC2086 # intentional word splitting of options +LOCAL_IMAGE_TAG=$($DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS config --images | grep -o "musicbrainz-docker_db:$POSTGRES_VERSION$") + +$DOCKER_CMD tag "$LOCAL_IMAGE_TAG" "$DEST_IMAGE_TAG" + +echo Pushing... + +$DOCKER_CMD push "$DEST_IMAGE_TAG" + +echo Done. + +# vi: set et sts=2 sw=2 ts=2 : diff --git a/admin/repository/prebuild-musicbrainz b/admin/repository/prebuild-musicbrainz new file mode 100755 index 00000000..cf5057a6 --- /dev/null +++ b/admin/repository/prebuild-musicbrainz @@ -0,0 +1,104 @@ +#!/usr/bin/env bash + +set -e -u + +# shellcheck source=admin/lib/common.inc.bash +source "$(dirname "${BASH_SOURCE[0]}")/../lib/common.inc.bash" + +HELP=$(cat < +where tag is based on MusicBrainz Server version and a build sequence number. + +Prerequisites: + + The version of MusicBrainz Server must be up-to-date in: + README.md + build/musicbrainz/Dockerfile + build/musicbrainz-prebuilt/Dockerfile + + The build sequence must be incremented, if image tag exists already + (for example when only files under build/musicbrainz/ have changed), in: + build/musicbrainz-prebuilt/Dockerfile +EOH +) + +if [[ $# -ne 0 && $1 =~ -*h(elp)? ]] +then + echo "$HELP" + exit 0 # EX_OK +elif [[ $# -ne 0 ]] +then + echo >&2 "$SCRIPT_NAME: unrecognized argument: $1" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 64 # EX_USAGE +fi + +# Retrieve destination image tag from the default Dockerfile + +MUSICBRAINZ_SERVER_VERSION=$(sed -n 's/^ARG MUSICBRAINZ_SERVER_VERSION=\([^ ]*\).*$/\1/p' build/musicbrainz-prebuilt/Dockerfile) +# shellcheck disable=SC2034 # as it used in eval below +MUSICBRAINZ_BUILD_SEQUENCE=$(sed -n 's/^ARG MUSICBRAINZ_BUILD_SEQUENCE=\([^ ]*\).*$/\1/p' build/musicbrainz-prebuilt/Dockerfile) +DEST_IMAGE_TAG=$(eval echo "$(sed -n 's/^FROM \([^ ]*\).*$/\1/p' build/musicbrainz-prebuilt/Dockerfile)") + +# Check that the MusicBrainz Server version is the same in all other files + +README_MUSICBRAINZ_SERVER_VERSION=$(sed -n 's/^.*Current MB Branch: \[\([^]]*\)\].*$/\1/p' README.md) + +if [[ $MUSICBRAINZ_SERVER_VERSION != "$README_MUSICBRAINZ_SERVER_VERSION" ]] +then + echo >&2 "$SCRIPT_NAME: File differs in MusicBrainz Server version:" + echo >&2 "$SCRIPT_NAME: * build/musicbrainz-prebuilt/Dockerfile: '$MUSICBRAINZ_SERVER_VERSION'" + echo >&2 "$SCRIPT_NAME: * README.md: '$README_MUSICBRAINZ_SERVER_VERSION'" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +SOURCE_MUSICBRAINZ_SERVER_VERSION=$(sed -n 's/^ARG MUSICBRAINZ_SERVER_VERSION=\([^ ]*\).*$/\1/p' build/musicbrainz/Dockerfile) + +if [[ $MUSICBRAINZ_SERVER_VERSION != "$SOURCE_MUSICBRAINZ_SERVER_VERSION" ]] +then + echo >&2 "$SCRIPT_NAME: File differs in MusicBrainz Server version:" + echo >&2 "$SCRIPT_NAME: * build/musicbrainz-prebuilt/Dockerfile: '$MUSICBRAINZ_SERVER_VERSION'" + echo >&2 "$SCRIPT_NAME: * build/musicbrainz/Dockerfile: '$SOURCE_MUSICBRAINZ_SERVER_VERSION'" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +# Check that the image tag doesn’t exist already + +if $DOCKER_CMD image pull "$DEST_IMAGE_TAG" 2>/dev/null \ + || $DOCKER_CMD image ls "$DEST_IMAGE_TAG" 2>/dev/null | grep -Eq "$DEST_IMAGE_TAG" +then + echo >&2 "$SCRIPT_NAME: image tag exists already: $DEST_IMAGE_TAG" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 65 # EX_DATAERR +fi + +# Do the thing + +echo Building... + +DOCKER_COMPOSE_OPTS='-f docker-compose.yml -f compose/musicbrainz-own-build.yml' + +# shellcheck disable=SC2086 # intentional word splitting of options +$DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS build --progress=plain --pull musicbrainz + +echo Tagging... + +# shellcheck disable=SC2086 # intentional word splitting of options +LOCAL_IMAGE_TAG=$($DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS config --images | grep -o '[^ ]*musicbrainz$') + +$DOCKER_CMD tag "$LOCAL_IMAGE_TAG" "$DEST_IMAGE_TAG" + +echo Pushing... + +$DOCKER_CMD push "$DEST_IMAGE_TAG" + +echo Done. + +# vi: set et sts=2 sw=2 ts=2 : diff --git a/admin/upgrade-to-postgres12 b/admin/upgrade-to-postgres12 deleted file mode 100755 index 2ea4b673..00000000 --- a/admin/upgrade-to-postgres12 +++ /dev/null @@ -1,79 +0,0 @@ -#!/bin/bash - -set -e -o pipefail -u - -# shellcheck source=admin/lib/common.inc.bash -source "$(dirname "${BASH_SOURCE[0]}")/lib/common.inc.bash" - -HELP=$(cat <&2 "$SCRIPT_NAME: unrecognized argument: $1" - echo >&2 "Try '$SCRIPT_NAME help' for usage." - exit 64 # EX_USAGE -fi - -$DOCKER_COMPOSE_CMD down -$DOCKER_COMPOSE_CMD \ - -f docker-compose.yml \ - -f admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml \ - up -d -$DOCKER_COMPOSE_CMD stop indexer search mq redis - -$DOCKER_COMPOSE_CMD exec db apt-get update -$DOCKER_COMPOSE_CMD exec db apt-get install \ - --no-install-suggests \ - --no-install-recommends \ - -y \ - postgresql-12 \ - postgresql-client-12 \ - postgresql-server-dev-12 \ - sudo - -PGDATA=/var/lib/postgresql/data -$DOCKER_COMPOSE_CMD exec db \ - sudo -u postgres /usr/lib/postgresql/9.5/bin/pg_ctl start -w -D "$PGDATA" -sleep 5 - -CURRENT_PG_VERSION=$($DOCKER_COMPOSE_CMD exec db psql -U musicbrainz -d musicbrainz_db -tAq -c 'SHOW server_version_num') -if echo "$CURRENT_PG_VERSION" | grep -v '^905'; then - echo "Error: Current postgres version should be < 90600, not $CURRENT_PG_VERSON" - exit 1 -fi - -$DOCKER_COMPOSE_CMD exec musicbrainz git fetch --depth=1 --update-shallow origin '+refs/heads/pg12:refs/remotes/origin/pg12' -$DOCKER_COMPOSE_CMD exec musicbrainz git reset --hard origin/pg12 - -$DOCKER_COMPOSE_CMD exec musicbrainz \ - sh -c 'eval "$(perl -Mlocal::lib)"; cd admin; ./psql --system READWRITE < sql/updates/20200518-pg12-before-upgrade.sql 2>&1' - -$DOCKER_COMPOSE_CMD stop db - -$DOCKER_COMPOSE_CMD \ - -f docker-compose.yml \ - -f admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml \ - up -d db - -$DOCKER_CMD cp admin/lib/upgrade-to-postgres12/do-pg_upgrade.sh musicbrainz-docker_db_1:/tmp/ -$DOCKER_COMPOSE_CMD exec db /bin/bash /tmp/do-pg_upgrade.sh - -$DOCKER_COMPOSE_CMD exec musicbrainz \ - sh -c 'eval "$(perl -Mlocal::lib)"; cd admin; ./psql --system READWRITE < sql/updates/20200518-pg12-after-upgrade.sql 2>&1' - -$DOCKER_COMPOSE_CMD exec db sudo -u postgres "$PGDATA"/analyze_new_cluster.sh -$DOCKER_COMPOSE_CMD exec db rm "$PGDATA"/analyze_new_cluster.sh - -$DOCKER_COMPOSE_CMD rm --stop --force db -$DOCKER_COMPOSE_CMD up --build -d - -echo 'Upgrade complete!' diff --git a/admin/upgrade-to-postgres16 b/admin/upgrade-to-postgres16 new file mode 100755 index 00000000..e10282c4 --- /dev/null +++ b/admin/upgrade-to-postgres16 @@ -0,0 +1,81 @@ +#!/bin/bash + +set -e -o pipefail -u + +# shellcheck source=admin/lib/common.inc.bash +source "$(dirname "${BASH_SOURCE[0]}")/lib/common.inc.bash" + +HELP=$(cat <&2 "$SCRIPT_NAME: unrecognized argument: $1" + echo >&2 "Try '$SCRIPT_NAME help' for usage." + exit 64 # EX_USAGE +fi + +$DOCKER_COMPOSE_CMD down +$DOCKER_COMPOSE_CMD \ + -f docker-compose.yml \ + -f admin/lib/upgrade-to-postgres16/pg-12-stopped.yml \ + up -d +$DOCKER_COMPOSE_CMD stop indexer search mq musicbrainz redis + +$DOCKER_COMPOSE_CMD exec db apt-get update +$DOCKER_COMPOSE_CMD exec db apt-get install \ + --no-install-suggests \ + --no-install-recommends \ + -y \ + postgresql-16 \ + postgresql-client-16 \ + postgresql-server-dev-16 \ + sudo + +PGDATA=/var/lib/postgresql/data +$DOCKER_COMPOSE_CMD exec db \ + sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl start -w -D "$PGDATA" +$DOCKER_COMPOSE_CMD exec db \ + sudo -u postgres /usr/lib/postgresql/12/bin/pg_isready -t 60 + +CURRENT_PG_VERSION=$($DOCKER_COMPOSE_CMD exec db psql -U musicbrainz -d musicbrainz_db -tAq -P pager=off -c 'SHOW server_version_num') +if echo "$CURRENT_PG_VERSION" | grep -v '^12'; then + echo "Error: Current postgres version should be < 13, not $CURRENT_PG_VERSON" + exit 1 +fi + +$DOCKER_COMPOSE_CMD stop db + +$DOCKER_COMPOSE_CMD \ + -f docker-compose.yml \ + -f admin/lib/upgrade-to-postgres16/pg-12-stopped.yml \ + up -d db + +DB_CONTAINER_ID=$($DOCKER_COMPOSE_CMD ps -q db) +$DOCKER_CMD cp admin/lib/upgrade-to-postgres16/do-pg_upgrade.sh "$DB_CONTAINER_ID":/tmp/ +$DOCKER_COMPOSE_CMD exec db /bin/bash /tmp/do-pg_upgrade.sh + +$DOCKER_COMPOSE_CMD exec db \ + sudo -u postgres /usr/lib/postgresql/16/bin/pg_ctl start -w -D "$PGDATA" +$DOCKER_COMPOSE_CMD exec db \ + sudo -u postgres /usr/lib/postgresql/16/bin/pg_isready -t 60 + +$DOCKER_COMPOSE_CMD exec db psql -U musicbrainz -d musicbrainz_db -tA -P pager=off -f "$PGDATA"/update_extensions.sql +$DOCKER_COMPOSE_CMD exec db rm "$PGDATA"/update_extensions.sql + +$DOCKER_COMPOSE_CMD exec db \ + sudo -u postgres /usr/lib/postgresql/16/bin/vacuumdb -U musicbrainz --all --analyze-in-stages + +$DOCKER_COMPOSE_CMD stop db +$DOCKER_COMPOSE_CMD rm --stop --force db +$DOCKER_COMPOSE_CMD up --build -d + +echo 'Upgrade complete!' diff --git a/build/musicbrainz-dev/DBDefs.pm b/build/musicbrainz-dev/DBDefs.pm index 5d050b60..65ad668f 100644 --- a/build/musicbrainz-dev/DBDefs.pm +++ b/build/musicbrainz-dev/DBDefs.pm @@ -54,7 +54,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "musicbrainz_db", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect to a test database @@ -62,7 +62,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "musicbrainz_test", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect to a Selenium test database. This database is created @@ -73,7 +73,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( schema => 'musicbrainz', username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect for read-only access. See "REPLICATION_TYPE" (below) @@ -81,7 +81,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "musicbrainz_db", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_READONLY_SERVER}", port => "5432", }, # How to connect for read-only access to the production database in standby mode. @@ -101,7 +101,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "template1", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect when running maintenance scripts located under admin/. @@ -111,7 +111,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( # database => "musicbrainz_db", # username => "$ENV{POSTGRES_USER}", # password => "$ENV{POSTGRES_PASSWORD}", -# host => "db", +# host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", # port => "5432", # }, # Fill out only if RAWDATA lives on a different host from the READWRITE server. @@ -122,7 +122,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( # replication_control.current_schema_sequence. # This is required, there is no default in order to prevent it changing without # manual intervention. -sub DB_SCHEMA_SEQUENCE { 28 } +sub DB_SCHEMA_SEQUENCE { 29 } # What type of server is this? # * RT_MASTER - This is a master replication server. Changes are allowed, and @@ -274,7 +274,7 @@ sub PLUGIN_CACHE_OPTIONS { my $self = shift; return { class => 'MusicBrainz::Server::CacheWrapper::Redis', - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", namespace => $self->CACHE_NAMESPACE . 'Catalyst:', }; } @@ -289,7 +289,7 @@ sub CACHE_MANAGER_OPTIONS { external => { class => 'MusicBrainz::Server::CacheWrapper::Redis', options => { - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", namespace => $self->CACHE_NAMESPACE, }, }, @@ -328,7 +328,7 @@ sub DATASTORE_REDIS_ARGS { return { database => 0, namespace => $self->CACHE_NAMESPACE, - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", test_database => 1, }; } @@ -430,8 +430,10 @@ sub DEVELOPMENT_SERVER { $ENV{MUSICBRAINZ_DEVELOPMENT_SERVER} == 1 ? 1 : 0 } # Please activate the officially approved languages here. Not every .po # file is active because we might have fully translated languages which # are not yet properly supported, like right-to-left languages +# +# The corresponding language packs must be installed; See NOTE-LANGUAGES-1 sub MB_LANGUAGES { shift->DEVELOPMENT_SERVER() - ? qw( de el es-es et he fi fr it ja nl sq en ) + ? qw( de el es et he fi fr it ja nl sq en ) : qw( de fr it nl en ) } diff --git a/build/musicbrainz-dev/Dockerfile b/build/musicbrainz-dev/Dockerfile index 45485456..4982d0b2 100644 --- a/build/musicbrainz-dev/Dockerfile +++ b/build/musicbrainz-dev/Dockerfile @@ -1,4 +1,4 @@ -ARG METABRAINZ_BASE_IMAGE_VERSION=focal-1.0.0-alpha1 +ARG METABRAINZ_BASE_IMAGE_VERSION=jammy-1.0.1-v0.4 ARG METABRAINZ_BASE_IMAGE_VARIANT= ARG METABRAINZ_BASE_IMAGE_TAG=${METABRAINZ_BASE_IMAGE_VERSION}${METABRAINZ_BASE_IMAGE_VARIANT} FROM metabrainz/base-image:${METABRAINZ_BASE_IMAGE_TAG} @@ -15,51 +15,20 @@ RUN curl -sSLO --retry 5 https://github.com/jwilder/dockerize/releases/download/ tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && \ rm -f dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz -ARG NODE_VERSION=18.17.1 -ARG PGP_SERVERS="keys.openpgp.org keyserver.ubuntu.com pgp.mit.edu" -ARG POSTGRES_VERSION=12 +ARG PERL_VERSION=5.38.2 +ARG PERL_SRC_SUM=a0a31534451eb7b83c7d6594a497543a54d488bc90ca00f5e34762577f40655e + +ARG CPANMINUS_VERSION=1.7047 +ARG CPANMINUS_SRC_SUM=963e63c6e1a8725ff2f624e9086396ae150db51dd0a337c3781d09a994af05a5 + +COPY keyrings/* /etc/apt/keyrings/ +ARG NODE_MAJOR_VERSION=20 +ARG POSTGRES_VERSION=16 SHELL ["/bin/bash", "-o", "pipefail", "-c"] -RUN mkdir -p /usr/local/share/keyrings && \ - curl -sSL --retry 5 https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/postgres-key.asc && \ - gpg --no-default-keyring --keyring /tmp/postgres-keyring.gpg --import /tmp/postgres-key.asc && \ - gpg --no-default-keyring --keyring /tmp/postgres-keyring.gpg --export --output /usr/local/share/keyrings/apt.postgresql.org.gpg && \ - rm -f /tmp/postgres-key.asc /tmp/postgres-keyring.gpg && \ - echo "deb [signed-by=/usr/local/share/keyrings/apt.postgresql.org.gpg] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \ - curl -sSL --retry 5 https://dl.yarnpkg.com/debian/pubkey.gpg > /tmp/yarn-key.asc && \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --import /tmp/yarn-key.asc && \ - \ - # Refresh Yarn signing keys - refreshed_keys=''; \ - for pgp_server in $(tr ' ' '\n' <<<"$PGP_SERVERS"); do \ - for attempt in {1..3}; do \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --keyserver "$pgp_server" --refresh-keys Yarn 2>&1 && \ - { refreshed_keys='yes'; break; } || \ - echo "Temporary failure: gpg returned error code '$?' on attempt #$attempt to reach '$pgp_server'."; \ - done; \ - if [[ $refreshed_keys == yes ]]; then break; fi; \ - done; \ - if [[ $refreshed_keys != yes ]]; then \ - echo >&2 'Fatal error: Failed all attempts to refresh PGP keys.'; \ - echo >&2 'Try passing a list of PGP servers that work for you as build argument:'; \ - echo >&2 ''; \ - echo >&2 ' docker-compose build --build-arg PGP_SERVERS="LIST" musicbrainz'; \ - echo >&2 ''; \ - echo >&2 "Current LIST is \"$PGP_SERVERS\""; \ - EX_TEMPFAIL=75; \ - exit $EX_TEMPFAIL; \ - fi; \ - unset attempt pgp_server refreshed_keys; \ - \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --export --output /usr/local/share/keyrings/dl.yarnpkg.com.gpg && \ - rm -f /tmp/yarn-key.asc /tmp/yarn-keyring.gpg && \ - echo "deb [signed-by=/usr/local/share/keyrings/dl.yarnpkg.com.gpg] https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarnpkg.list && \ +RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.asc] https://deb.nodesource.com/node_${NODE_MAJOR_VERSION}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list && \ + echo "deb [signed-by=/etc/apt/keyrings/pgdg.asc] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \ apt-get update && \ apt-get install --no-install-recommends -qy \ - python2-minimal && \ - curl -sSLO --retry 5 https://deb.nodesource.com/node_18.x/pool/main/n/nodejs/nodejs_${NODE_VERSION}-deb-1nodesource1_amd64.deb && \ - dpkg -i nodejs_${NODE_VERSION}-deb-1nodesource1_amd64.deb && \ - apt-get install --no-install-recommends -qy \ - cpanminus \ bash-completion \ build-essential \ bzip2 \ @@ -68,6 +37,7 @@ RUN mkdir -p /usr/local/share/keyrings && \ git \ # Needed to generate SVG from database schema DOT diagrams graphviz \ + # NOTE-LANGUAGES-1: These language packs must match the definition of MB_LANGUAGES in DBDefs.pm language-pack-de \ language-pack-el \ language-pack-es \ @@ -83,14 +53,12 @@ RUN mkdir -p /usr/local/share/keyrings && \ libdb-dev \ libexpat1-dev \ libicu-dev \ - liblocal-lib-perl \ libpq-dev \ libssl-dev \ # Needed for XML::LibXML libxml2-dev \ make \ - # Needed for ts - moreutils \ + nodejs \ # Needed for Unicode::ICU::Collator pkg-config \ postgresql-${POSTGRES_VERSION} \ @@ -98,28 +66,68 @@ RUN mkdir -p /usr/local/share/keyrings && \ scour \ # Needed to decompress sample data xz-utils \ - yarn \ # Needed for XML:LibXML zlib1g-dev \ - && rm -rf /var/lib/apt/lists/* + && \ + rm -rf /var/lib/apt/lists/* && \ + # Install perl (in a more recent version than available with apt) + cd /usr/src && \ + curl -sSLO https://cpan.metacpan.org/authors/id/P/PE/PEVANS/perl-$PERL_VERSION.tar.gz && \ + echo "$PERL_SRC_SUM *perl-$PERL_VERSION.tar.gz" | sha256sum --strict --check - && \ + tar -xzf perl-$PERL_VERSION.tar.gz && \ + cd perl-$PERL_VERSION && \ + gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" && \ + archBits="$(dpkg-architecture --query DEB_BUILD_ARCH_BITS)" && \ + archFlag="$([ "$archBits" = '64' ] && echo '-Duse64bitall' || echo '-Duse64bitint')" && \ + ./Configure \ + -Darchname="$gnuArch" "$archFlag" \ + -Duselargefiles -Duseshrplib -Dusethreads \ + -Dvendorprefix=/usr/local -Dman1dir=none -Dman3dir=none \ + -des && \ + make -j$(nproc) && \ + make install && \ + rm -fr /usr/src/perl-$PERL_VERSION* && \ + # Install cpanm (needed to help with updating Perl modules) + cd /usr/src && \ + curl -sSLO https://www.cpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-$CPANMINUS_VERSION.tar.gz && \ + echo "$CPANMINUS_SRC_SUM *App-cpanminus-$CPANMINUS_VERSION.tar.gz" | sha256sum --strict --check - && \ + tar -xzf App-cpanminus-$CPANMINUS_VERSION.tar.gz && \ + cd App-cpanminus-$CPANMINUS_VERSION && \ + perl bin/cpanm . && \ + rm -fr /usr/src/App-cpanminus-$CPANMINUS_VERSION* && \ + cd /root && \ + # Install local::lib (needed to persistently update Perl modules) + cpanm local::lib && \ + rm -fr /root/.cpanm && \ + # Install ts (needed to run admin background task scripts locally) + curl -sSL https://git.joeyh.name/index.cgi/moreutils.git/plain/ts?h=0.69 -o /usr/local/bin/ts && \ + echo '01b67f3d81e6205f01cc0ada87039293ebc56596955225300dd69ec1257124f5 */usr/local/bin/ts' | sha256sum --strict --check - && \ + chmod +x /usr/local/bin/ts && \ + # Install yarn from nodejs + corepack enable WORKDIR /musicbrainz-server -RUN git config --global --add safe.directory /musicbrainz-server +RUN git config --global --add safe.directory /musicbrainz-server && \ + ln -s /root/.bash_history.d/bash_history ~/.bash_history COPY DBDefs.pm / COPY scripts/* /usr/local/bin/ -RUN cat /usr/local/bin/snippet.perllocallib.bashrc >> ~/.bashrc \ - && rm /usr/local/bin/snippet.perllocallib.bashrc \ - && ln -s /usr/local/bin/docker-entrypoint.sh / +RUN cat /usr/local/bin/snippet.perllocallib.bashrc >> ~/.bashrc && \ + rm /usr/local/bin/snippet.perllocallib.bashrc && \ + ln -s /usr/local/bin/docker-entrypoint.sh / # Postgres user/password would be solely needed to compile tests ARG POSTGRES_USER=doesntmatteraslongasyoudontcompiletests ARG POSTGRES_PASSWORD=doesntmatteraslongasyoudontcompiletests ENV BASH_ENV=/noninteractive.bash_env \ - MUSICBRAINZ_BASE_DOWNLOAD_URL=http://ftp.eu.metabrainz.org/pub/musicbrainz \ + MUSICBRAINZ_BASE_DOWNLOAD_URL=https://data.metabrainz.org/pub/musicbrainz \ MUSICBRAINZ_CATALYST_DEBUG=0 \ MUSICBRAINZ_DEVELOPMENT_SERVER=1 \ + MUSICBRAINZ_POSTGRES_SERVER=db \ + MUSICBRAINZ_POSTGRES_READONLY_SERVER=db \ + MUSICBRAINZ_RABBITMQ_SERVER=mq \ + MUSICBRAINZ_REDIS_SERVER=redis \ MUSICBRAINZ_SEARCH_SERVER=search:8983/solr \ MUSICBRAINZ_SERVER_PROCESSES=1 \ MUSICBRAINZ_STANDALONE_SERVER=1 \ diff --git a/build/musicbrainz-dev/keyrings/nodesource.asc b/build/musicbrainz-dev/keyrings/nodesource.asc new file mode 100644 index 00000000..b7637b89 --- /dev/null +++ b/build/musicbrainz-dev/keyrings/nodesource.asc @@ -0,0 +1,29 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQENBFdDN1ABCADaNd/I3j3tn40deQNgz7hB2NvT+syXe6k4ZmdiEcOfBvFrkS8B +hNS67t93etHsxEy7E0qwsZH32bKazMqe9zDwoa3aVImryjh6SHC9lMtW27JPHFeM +Srkt9YmH1WMwWcRO6eSY9B3PpazquhnvbammLuUojXRIxkDroy6Fw4UKmUNSRr32 +9Ej87jRoR1B2/57Kfp2Y4+vFGGzSvh3AFQpBHq51qsNHALU6+8PjLfIt+5TPvaWR +TB+kAZnQZkaIQM2nr1n3oj6ak2RATY/+kjLizgFWzgEfbCrbsyq68UoY5FPBnu4Z +E3iDZpaIqwKr0seUC7iA1xM5eHi5kty1oB7HABEBAAG0Ik5Tb2xpZCA8bnNvbGlk +LWdwZ0Bub2Rlc291cmNlLmNvbT6JATgEEwECACIFAldDN1ACGwMGCwkIBwMCBhUI +AgkKCwQWAgMBAh4BAheAAAoJEC9ZtfmbG+C0y7wH/i4xnab36dtrYW7RZwL8i6Sc +NjMx4j9+U1kr/F6YtqWd+JwCbBdar5zRghxPcYEq/qf7MbgAYcs1eSOuTOb7n7+o +xUwdH2iCtHhKh3Jr2mRw1ks7BbFZPB5KmkxHaEBfLT4d+I91ZuUdPXJ+0SXs9gzk +Dbz65Uhoz3W03aiF8HeL5JNARZFMbHHNVL05U1sTGTCOtu+1c/33f3TulQ/XZ3Y4 +hwGCpLe0Tv7g7Lp3iLMZMWYPEa0a7S4u8he5IEJQLd8bE8jltcQvrdr3Fm8kI2Jg +BJmUmX4PSfhuTCFaR/yeCt3UoW883bs9LfbTzIx9DJGpRIu8Y0IL3b4sj/GoZVq5 +AQ0EV0M3UAEIAKrTaC62ayzqOIPa7nS90BHHck4Z33a2tZF/uof38xNOiyWGhT8u +JeFoTTHn5SQq5Ftyu4K3K2fbbpuu/APQF05AaljzVkDGNMW4pSkgOasdysj831cu +ssrHX2RYS22wg80k6C/Hwmh5F45faEuNxsV+bPx7oPUrt5n6GMx84vEP3i1+FDBi +0pt/B/QnDFBXki1BGvJ35f5NwDefK8VaInxXP3ZN/WIbtn5dqxppkV/YkO7GiJlp +Jlju9rf3kKUIQzKQWxFsbCAPIHoWv7rH9RSxgDithXtG6Yg5R1aeBbJaPNXL9wpJ +YBJbiMjkAFaz4B95FOqZm3r7oHugiCGsHX0AEQEAAYkBHwQYAQIACQUCV0M3UAIb +DAAKCRAvWbX5mxvgtE/OB/0VN88DR3Y3fuqy7lq/dthkn7Dqm9YXdorZl3L152eE +IF882aG8FE3qZdaLGjQO4oShAyNWmRfSGuoH0XERXAI9n0r8m4mDMxE6rtP7tHet +y/5M8x3CTyuMgx5GLDaEUvBusnTD+/v/fBMwRK/cZ9du5PSG4R50rtst+oYyC2ao +x4I2SgjtF/cY7bECsZDplzatN3gv34PkcdIg8SLHAVlL4N5tzumDeizRspcSyoy2 +K2+hwKU4C4+dekLLTg8rjnRROvplV2KtaEk6rxKtIRFDCoQng8wfJuIMrDNKvqZw +FRGt7cbvW5MCnuH8MhItOl9Uxp1wHp6gtav/h8Gp6MBa +=MARt +-----END PGP PUBLIC KEY BLOCK----- diff --git a/build/musicbrainz-dev/keyrings/pgdg.asc b/build/musicbrainz-dev/keyrings/pgdg.asc new file mode 100644 index 00000000..8480576e --- /dev/null +++ b/build/musicbrainz-dev/keyrings/pgdg.asc @@ -0,0 +1,77 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBE6XR8IBEACVdDKT2HEH1IyHzXkb4nIWAY7echjRxo7MTcj4vbXAyBKOfjja +UrBEJWHN6fjKJXOYWXHLIYg0hOGeW9qcSiaa1/rYIbOzjfGfhE4x0Y+NJHS1db0V +G6GUj3qXaeyqIJGS2z7m0Thy4Lgr/LpZlZ78Nf1fliSzBlMo1sV7PpP/7zUO+aA4 +bKa8Rio3weMXQOZgclzgeSdqtwKnyKTQdXY5MkH1QXyFIk1nTfWwyqpJjHlgtwMi +c2cxjqG5nnV9rIYlTTjYG6RBglq0SmzF/raBnF4Lwjxq4qRqvRllBXdFu5+2pMfC +IZ10HPRdqDCTN60DUix+BTzBUT30NzaLhZbOMT5RvQtvTVgWpeIn20i2NrPWNCUh +hj490dKDLpK/v+A5/i8zPvN4c6MkDHi1FZfaoz3863dylUBR3Ip26oM0hHXf4/2U +A/oA4pCl2W0hc4aNtozjKHkVjRx5Q8/hVYu+39csFWxo6YSB/KgIEw+0W8DiTII3 +RQj/OlD68ZDmGLyQPiJvaEtY9fDrcSpI0Esm0i4sjkNbuuh0Cvwwwqo5EF1zfkVj +Tqz2REYQGMJGc5LUbIpk5sMHo1HWV038TWxlDRwtOdzw08zQA6BeWe9FOokRPeR2 +AqhyaJJwOZJodKZ76S+LDwFkTLzEKnYPCzkoRwLrEdNt1M7wQBThnC5z6wARAQAB +tBxQb3N0Z3JlU1FMIERlYmlhbiBSZXBvc2l0b3J5iQJOBBMBCAA4AhsDBQsJCAcD +BRUKCQgLBRYCAwEAAh4BAheAFiEEuXsK/KoaR/BE8kSgf8x9RqzMTPgFAlhtCD8A +CgkQf8x9RqzMTPgECxAAk8uL+dwveTv6eH21tIHcltt8U3Ofajdo+D/ayO53LiYO +xi27kdHD0zvFMUWXLGxQtWyeqqDRvDagfWglHucIcaLxoxNwL8+e+9hVFIEskQAY +kVToBCKMXTQDLarz8/J030Pmcv3ihbwB+jhnykMuyyNmht4kq0CNgnlcMCdVz0d3 +z/09puryIHJrD+A8y3TD4RM74snQuwc9u5bsckvRtRJKbP3GX5JaFZAqUyZNRJRJ +Tn2OQRBhCpxhlZ2afkAPFIq2aVnEt/Ie6tmeRCzsW3lOxEH2K7MQSfSu/kRz7ELf +Cz3NJHj7rMzC+76Rhsas60t9CjmvMuGONEpctijDWONLCuch3Pdj6XpC+MVxpgBy +2VUdkunb48YhXNW0jgFGM/BFRj+dMQOUbY8PjJjsmVV0joDruWATQG/M4C7O8iU0 +B7o6yVv4m8LDEN9CiR6r7H17m4xZseT3f+0QpMe7iQjz6XxTUFRQxXqzmNnloA1T +7VjwPqIIzkj/u0V8nICG/ktLzp1OsCFatWXh7LbU+hwYl6gsFH/mFDqVxJ3+DKQi +vyf1NatzEwl62foVjGUSpvh3ymtmtUQ4JUkNDsXiRBWczaiGSuzD9Qi0ONdkAX3b +ewqmN4TfE+XIpCPxxHXwGq9Rv1IFjOdCX0iG436GHyTLC1tTUIKF5xV4Y0+cXIOI +RgQQEQgABgUCTpdI7gAKCRDFr3dKWFELWqaPAKD1TtT5c3sZz92Fj97KYmqbNQZP ++ACfSC6+hfvlj4GxmUjp1aepoVTo3weJAhwEEAEIAAYFAk6XSQsACgkQTFprqxLS +p64F8Q//cCcutwrH50UoRFejg0EIZav6LUKejC6kpLeubbEtuaIH3r2zMblPGc4i ++eMQKo/PqyQrceRXeNNlqO6/exHozYi2meudxa6IudhwJIOn1MQykJbNMSC2sGUp +1W5M1N5EYgt4hy+qhlfnD66LR4G+9t5FscTJSy84SdiOuqgCOpQmPkVRm1HX5X1+ +dmnzMOCk5LHHQuiacV0qeGO7JcBCVEIDr+uhU1H2u5GPFNHm5u15n25tOxVivb94 +xg6NDjouECBH7cCVuW79YcExH/0X3/9G45rjdHlKPH1OIUJiiX47OTxdG3dAbB4Q +fnViRJhjehFscFvYWSqXo3pgWqUsEvv9qJac2ZEMSz9x2mj0ekWxuM6/hGWxJdB+ ++985rIelPmc7VRAXOjIxWknrXnPCZAMlPlDLu6+vZ5BhFX0Be3y38f7GNCxFkJzl +hWZ4Cj3WojMj+0DaC1eKTj3rJ7OJlt9S9xnO7OOPEUTGyzgNIDAyCiu8F4huLPaT +ape6RupxOMHZeoCVlqx3ouWctelB2oNXcxxiQ/8y+21aHfD4n/CiIFwDvIQjl7dg +mT3u5Lr6yxuosR3QJx1P6rP5ZrDTP9khT30t+HZCbvs5Pq+v/9m6XDmi+NlU7Zuh +Ehy97tL3uBDgoL4b/5BpFL5U9nruPlQzGq1P9jj40dxAaDAX/WKJAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlB5KywFCQPDFt8ACgkQf8x9RqzM +TPhuCQ//QAjRSAOCQ02qmUAikT+mTB6baOAakkYq6uHbEO7qPZkv4E/M+HPIJ4wd +nBNeSQjfvdNcZBA/x0hr5EMcBneKKPDj4hJ0panOIRQmNSTThQw9OU351gm3YQct +AMPRUu1fTJAL/AuZUQf9ESmhyVtWNlH/56HBfYjE4iVeaRkkNLJyX3vkWdJSMwC/ +LO3Lw/0M3R8itDsm74F8w4xOdSQ52nSRFRh7PunFtREl+QzQ3EA/WB4AIj3VohIG +kWDfPFCzV3cyZQiEnjAe9gG5pHsXHUWQsDFZ12t784JgkGyO5wT26pzTiuApWM3k +/9V+o3HJSgH5hn7wuTi3TelEFwP1fNzI5iUUtZdtxbFOfWMnZAypEhaLmXNkg4zD +kH44r0ss9fR0DAgUav1a25UnbOn4PgIEQy2fgHKHwRpCy20d6oCSlmgyWsR40EPP +YvtGq49A2aK6ibXmdvvFT+Ts8Z+q2SkFpoYFX20mR2nsF0fbt1lfH65P64dukxeR +GteWIeNakDD40bAAOH8+OaoTGVBJ2ACJfLVNM53PEoftavAwUYMrR910qvwYfd/4 +6rh46g1Frr9SFMKYE9uvIJIgDsQB3QBp71houU4H55M5GD8XURYs+bfiQpJG1p7e +B8e5jZx1SagNWc4XwL2FzQ9svrkbg1Y+359buUiP7T6QXX2zY++JAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlEqbZUFCQg2wEEACgkQf8x9RqzM +TPhFMQ//WxAfKMdpSIA9oIC/yPD/dJpY/+DyouOljpE6MucMy/ArBECjFTBwi/j9 +NYM4ynAk34IkhuNexc1i9/05f5RM6+riLCLgAOsADDbHD4miZzoSxiVr6GQ3YXMb +OGld9kV9Sy6mGNjcUov7iFcf5Hy5w3AjPfKuR9zXswyfzIU1YXObiiZT38l55pp/ +BSgvGVQsvbNjsff5CbEKXS7q3xW+WzN0QWF6YsfNVhFjRGj8hKtHvwKcA02wwjLe +LXVTm6915ZUKhZXUFc0vM4Pj4EgNswH8Ojw9AJaKWJIZmLyW+aP+wpu6YwVCicxB +Y59CzBO2pPJDfKFQzUtrErk9irXeuCCLesDyirxJhv8o0JAvmnMAKOLhNFUrSQ2m ++3EnF7zhfz70gHW+EG8X8mL/EN3/dUM09j6TVrjtw43RLxBzwMDeariFF9yC+5bL +tnGgxjsB9Ik6GV5v34/NEEGf1qBiAzFmDVFRZlrNDkq6gmpvGnA5hUWNr+y0i01L +jGyaLSWHYjgw2UEQOqcUtTFK9MNzbZze4mVaHMEz9/aMfX25R6qbiNqCChveIm8m +Yr5Ds2zdZx+G5bAKdzX7nx2IUAxFQJEE94VLSp3npAaTWv3sHr7dR8tSyUJ9poDw +gw4W9BIcnAM7zvFYbLF5FNggg/26njHCCN70sHt8zGxKQINMc6SJAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlLpFRkFCQ6EJy0ACgkQf8x9RqzM +TPjOZA//Zp0e25pcvle7cLc0YuFr9pBv2JIkLzPm83nkcwKmxaWayUIG4Sv6pH6h +m8+S/CHQij/yFCX+o3ngMw2J9HBUvafZ4bnbI0RGJ70GsAwraQ0VlkIfg7GUw3Tz +voGYO42rZTru9S0K/6nFP6D1HUu+U+AsJONLeb6oypQgInfXQExPZyliUnHdipei +4WR1YFW6sjSkZT/5C3J1wkAvPl5lvOVthI9Zs6bZlJLZwusKxU0UM4Btgu1Sf3nn +JcHmzisixwS9PMHE+AgPWIGSec/N27a0KmTTvImV6K6nEjXJey0K2+EYJuIBsYUN +orOGBwDFIhfRk9qGlpgt0KRyguV+AP5qvgry95IrYtrOuE7307SidEbSnvO5ezNe +mE7gT9Z1tM7IMPfmoKph4BfpNoH7aXiQh1Wo+ChdP92hZUtQrY2Nm13cmkxYjQ4Z +gMWfYMC+DA/GooSgZM5i6hYqyyfAuUD9kwRN6BqTbuAUAp+hCWYeN4D88sLYpFh3 +paDYNKJ+Gf7Yyi6gThcV956RUFDH3ys5Dk0vDL9NiWwdebWfRFbzoRM3dyGP889a +OyLzS3mh6nHzZrNGhW73kslSQek8tjKrB+56hXOnb4HaElTZGDvD5wmrrhN94kby +Gtz3cydIohvNO9d90+29h0eGEDYti7j7maHkBKUAwlcPvMg5m3Y= +=DA1T +-----END PGP PUBLIC KEY BLOCK----- diff --git a/build/musicbrainz-dev/scripts/createdb.sh b/build/musicbrainz-dev/scripts/createdb.sh index 237df91c..4fdaaf4a 100755 --- a/build/musicbrainz-dev/scripts/createdb.sh +++ b/build/musicbrainz-dev/scripts/createdb.sh @@ -94,7 +94,7 @@ for F in "${DUMP_FILES[@]}"; do done echo "found existing dumps" -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 update-perl.sh diff --git a/build/musicbrainz-dev/scripts/indexer-triggers.sh b/build/musicbrainz-dev/scripts/indexer-triggers.sh index e4a17e30..eb821f27 100755 --- a/build/musicbrainz-dev/scripts/indexer-triggers.sh +++ b/build/musicbrainz-dev/scripts/indexer-triggers.sh @@ -12,7 +12,7 @@ fi INDEXER_SQL_DIR="$1" -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 cd /musicbrainz-server diff --git a/build/musicbrainz-dev/scripts/recreatedb.sh b/build/musicbrainz-dev/scripts/recreatedb.sh index b325d420..86ac8548 100755 --- a/build/musicbrainz-dev/scripts/recreatedb.sh +++ b/build/musicbrainz-dev/scripts/recreatedb.sh @@ -2,5 +2,5 @@ set -e -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 -psql postgres -U musicbrainz -h db -c "DROP DATABASE IF EXISTS musicbrainz_db;"; createdb.sh "$@" +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 +psql postgres -U musicbrainz -h "${MUSICBRAINZ_POSTGRES_SERVER}" -c "DROP DATABASE IF EXISTS musicbrainz_db;"; createdb.sh "$@" diff --git a/build/musicbrainz-dev/scripts/replication.sh b/build/musicbrainz-dev/scripts/replication.sh index 8d98a28d..238db901 100755 --- a/build/musicbrainz-dev/scripts/replication.sh +++ b/build/musicbrainz-dev/scripts/replication.sh @@ -2,5 +2,5 @@ set -e -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 exec /musicbrainz-server/admin/cron/mirror.sh diff --git a/build/musicbrainz-dev/scripts/update-javascript.sh b/build/musicbrainz-dev/scripts/update-javascript.sh index b4788fd8..aa7286cc 100755 --- a/build/musicbrainz-dev/scripts/update-javascript.sh +++ b/build/musicbrainz-dev/scripts/update-javascript.sh @@ -6,6 +6,6 @@ cd /musicbrainz-server yarn -dockerize -wait tcp://db:5432 -timeout 60s -wait tcp://mq:5672 -timeout 60s -wait tcp://redis:6379 -timeout 60s +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s -wait "tcp://${MUSICBRAINZ_RABBITMQ_SERVER}:5672" -timeout 60s -wait "tcp://${MUSICBRAINZ_REDIS_SERVER}:6379" -timeout 60s ./script/compile_resources.sh --watch client server tests &> /compile_resources.log & diff --git a/build/musicbrainz-dev/scripts/update-perl.sh b/build/musicbrainz-dev/scripts/update-perl.sh index 3684c7ce..e5d32773 100755 --- a/build/musicbrainz-dev/scripts/update-perl.sh +++ b/build/musicbrainz-dev/scripts/update-perl.sh @@ -8,12 +8,8 @@ diff /DBDefs.pm lib/DBDefs.pm || cat /DBDefs.pm > lib/DBDefs.pm cpanm --installdeps --notest --with-develop . cpanm --notest \ - Catalyst::Plugin::Cache::HTTP \ Catalyst::Plugin::StackTrace \ - Digest::MD5::File \ File::Slurp \ - JSON::Any \ - LWP::Protocol::https \ Plack::Handler::Starlet \ Plack::Middleware::Debug::Base \ Server::Starter \ diff --git a/build/musicbrainz-prebuilt/Dockerfile b/build/musicbrainz-prebuilt/Dockerfile new file mode 100644 index 00000000..349a9be8 --- /dev/null +++ b/build/musicbrainz-prebuilt/Dockerfile @@ -0,0 +1,3 @@ +ARG MUSICBRAINZ_SERVER_VERSION=v-2025-02-24.0 +ARG MUSICBRAINZ_BUILD_SEQUENCE=0 +FROM metabrainz/musicbrainz-docker-musicbrainz:${MUSICBRAINZ_SERVER_VERSION}-build${MUSICBRAINZ_BUILD_SEQUENCE} diff --git a/build/musicbrainz/DBDefs.pm b/build/musicbrainz/DBDefs.pm index 4b9d23e7..c1c07a8e 100644 --- a/build/musicbrainz/DBDefs.pm +++ b/build/musicbrainz/DBDefs.pm @@ -54,7 +54,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "musicbrainz_db", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect to a test database @@ -80,7 +80,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "musicbrainz_db", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_READONLY_SERVER}", port => "5432", }, # How to connect for administrative access @@ -88,7 +88,7 @@ MusicBrainz::Server::DatabaseConnectionFactory->register_databases( database => "template1", username => "$ENV{POSTGRES_USER}", password => "$ENV{POSTGRES_PASSWORD}", - host => "db", + host => "$ENV{MUSICBRAINZ_POSTGRES_SERVER}", port => "5432", }, # How to connect when running maintenance scripts located under admin/. @@ -261,7 +261,7 @@ sub PLUGIN_CACHE_OPTIONS { my $self = shift; return { class => 'MusicBrainz::Server::CacheWrapper::Redis', - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", namespace => $self->CACHE_NAMESPACE . 'Catalyst:', }; } @@ -276,7 +276,7 @@ sub CACHE_MANAGER_OPTIONS { external => { class => 'MusicBrainz::Server::CacheWrapper::Redis', options => { - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", namespace => $self->CACHE_NAMESPACE, }, }, @@ -315,7 +315,7 @@ sub DATASTORE_REDIS_ARGS { return { database => 0, namespace => $self->CACHE_NAMESPACE, - server => 'redis:6379', + server => "$ENV{MUSICBRAINZ_REDIS_SERVER}:6379", test_database => 1, }; } @@ -417,8 +417,10 @@ sub DEVELOPMENT_SERVER { $ENV{MUSICBRAINZ_DEVELOPMENT_SERVER} == 1 ? 1 : 0 } # Please activate the officially approved languages here. Not every .po # file is active because we might have fully translated languages which # are not yet properly supported, like right-to-left languages +# +# The corresponding language packs must be installed; See NOTE-LANGUAGES-1 sub MB_LANGUAGES { shift->DEVELOPMENT_SERVER() - ? qw( de el es-es et fi fr he it ja nl sq en ) + ? qw( de el es et fi fr he it ja nl sq en ) : qw( de fr it nl en ) } diff --git a/build/musicbrainz/Dockerfile b/build/musicbrainz/Dockerfile index 3b81a7c7..004c4233 100644 --- a/build/musicbrainz/Dockerfile +++ b/build/musicbrainz/Dockerfile @@ -1,4 +1,4 @@ -ARG METABRAINZ_BASE_IMAGE_VERSION=focal-1.0.0-alpha1 +ARG METABRAINZ_BASE_IMAGE_VERSION=jammy-1.0.1-v0.4 ARG METABRAINZ_BASE_IMAGE_VARIANT= ARG METABRAINZ_BASE_IMAGE_TAG=${METABRAINZ_BASE_IMAGE_VERSION}${METABRAINZ_BASE_IMAGE_VARIANT} FROM metabrainz/base-image:${METABRAINZ_BASE_IMAGE_TAG} @@ -15,31 +15,41 @@ RUN curl -sSLO --retry 5 https://github.com/jwilder/dockerize/releases/download/ tar -C /usr/local/bin -xzvf dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz && \ rm -f dockerize-linux-amd64-$DOCKERIZE_VERSION.tar.gz -ARG POSTGRES_VERSION=12 +ARG PERL_VERSION=5.38.2 +ARG PERL_SRC_SUM=a0a31534451eb7b83c7d6594a497543a54d488bc90ca00f5e34762577f40655e + +ARG CPANMINUS_VERSION=1.7047 +ARG CPANMINUS_SRC_SUM=963e63c6e1a8725ff2f624e9086396ae150db51dd0a337c3781d09a994af05a5 + +COPY keyrings/* /etc/apt/keyrings/ +ARG NODE_MAJOR_VERSION=20 +ARG POSTGRES_VERSION=16 SHELL ["/bin/bash", "-o", "pipefail", "-c"] -RUN mkdir -p /usr/local/share/keyrings && \ - curl -sSL --retry 5 https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/postgres-key.asc && \ - gpg --no-default-keyring --keyring /tmp/postgres-keyring.gpg --import /tmp/postgres-key.asc && \ - gpg --no-default-keyring --keyring /tmp/postgres-keyring.gpg --export --output /usr/local/share/keyrings/apt.postgresql.org.gpg && \ - rm -f /tmp/postgres-key.asc /tmp/postgres-keyring.gpg && \ - echo "deb [signed-by=/usr/local/share/keyrings/apt.postgresql.org.gpg] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \ +RUN echo "deb [signed-by=/etc/apt/keyrings/nodesource.asc] https://deb.nodesource.com/node_${NODE_MAJOR_VERSION}.x nodistro main" > /etc/apt/sources.list.d/nodesource.list && \ + echo "deb [signed-by=/etc/apt/keyrings/pgdg.asc] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list && \ apt-get update && \ apt-get install --no-install-recommends -qy \ - cpanminus \ + # Needed for building perl + build-essential \ bzip2 \ gettext \ g++ \ git \ + # NOTE-LANGUAGES-1: These language packs must match the definition of MB_LANGUAGES in DBDefs.pm + language-pack-de \ + language-pack-fr \ + language-pack-it \ + language-pack-nl \ # Needed for Cache in DB_File libdb-dev \ libexpat1-dev \ libicu-dev \ - liblocal-lib-perl \ libpq-dev \ libssl-dev \ # Needed for XML::LibXML libxml2-dev \ make \ + nodejs \ # Needed for Unicode::ICU::Collator pkg-config \ postgresql-client-${POSTGRES_VERSION} \ @@ -48,86 +58,85 @@ RUN mkdir -p /usr/local/share/keyrings && \ xz-utils \ # Needed for XML:LibXML zlib1g-dev \ - && rm -rf /var/lib/apt/lists/* + && \ + rm -rf /var/lib/apt/lists/* && \ + # Install perl (in a more recent version than available with apt) + cd /usr/src && \ + curl -sSLO https://cpan.metacpan.org/authors/id/P/PE/PEVANS/perl-$PERL_VERSION.tar.gz && \ + echo "$PERL_SRC_SUM *perl-$PERL_VERSION.tar.gz" | sha256sum --strict --check - && \ + tar -xzf perl-$PERL_VERSION.tar.gz && \ + cd perl-$PERL_VERSION && \ + gnuArch="$(dpkg-architecture --query DEB_BUILD_GNU_TYPE)" && \ + archBits="$(dpkg-architecture --query DEB_BUILD_ARCH_BITS)" && \ + archFlag="$([ "$archBits" = '64' ] && echo '-Duse64bitall' || echo '-Duse64bitint')" && \ + ./Configure \ + -Darchname="$gnuArch" "$archFlag" \ + -Duselargefiles -Duseshrplib -Dusethreads \ + -Dvendorprefix=/usr/local -Dman1dir=none -Dman3dir=none \ + -des && \ + make -j$(nproc) && \ + make install && \ + rm -fr /usr/src/perl-$PERL_VERSION* && \ + # Install cpanm (needed to help with updating Perl modules) + cd /usr/src && \ + curl -sSLO https://www.cpan.org/authors/id/M/MI/MIYAGAWA/App-cpanminus-$CPANMINUS_VERSION.tar.gz && \ + echo "$CPANMINUS_SRC_SUM *App-cpanminus-$CPANMINUS_VERSION.tar.gz" | sha256sum --strict --check - && \ + tar -xzf App-cpanminus-$CPANMINUS_VERSION.tar.gz && \ + cd App-cpanminus-$CPANMINUS_VERSION && \ + perl bin/cpanm . && \ + rm -fr /usr/src/App-cpanminus-$CPANMINUS_VERSION* && \ + cd /root && \ + # Install local::lib (needed to persistently update Perl modules) + cpanm local::lib && \ + rm -fr /root/.cpanm && \ + # Install yarn from nodejs + corepack enable && \ + # Clean up perl build dependencies + apt-get purge --auto-remove -y \ + build-essential -ARG MUSICBRAINZ_SERVER_VERSION=v-2023-10-24-hotfix +ARG MUSICBRAINZ_SERVER_VERSION=v-2025-02-24.0 LABEL org.metabrainz.musicbrainz-server.version="${MUSICBRAINZ_SERVER_VERSION}" RUN git clone --depth=1 --branch $MUSICBRAINZ_SERVER_VERSION https://github.com/metabrainz/musicbrainz-server.git musicbrainz-server WORKDIR /musicbrainz-server -ARG NODE_VERSION=18.17.1 -ARG PGP_SERVERS="keys.openpgp.org keyserver.ubuntu.com pgp.mit.edu" -RUN cp docker/yarn_pubkey.txt /tmp/yarn-key.asc && \ - cd /tmp && \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --import /tmp/yarn-key.asc && \ - \ - # Refresh Yarn signing keys - refreshed_keys=''; \ - for pgp_server in $(tr ' ' '\n' <<<"$PGP_SERVERS"); do \ - for attempt in {1..3}; do \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --keyserver "$pgp_server" --refresh-keys Yarn 2>&1 && \ - { refreshed_keys='yes'; break; } || \ - echo "Temporary failure: gpg returned error code '$?' on attempt #$attempt to reach '$pgp_server'."; \ - done; \ - if [[ $refreshed_keys == yes ]]; then break; fi; \ - done; \ - if [[ $refreshed_keys != yes ]]; then \ - echo >&2 'Fatal error: Failed all attempts to refresh PGP keys.'; \ - echo >&2 'Try passing a list of PGP servers that work for you as build argument:'; \ - echo >&2 ''; \ - echo >&2 ' docker-compose build --build-arg PGP_SERVERS="LIST" musicbrainz'; \ - echo >&2 ''; \ - echo >&2 "Current LIST is \"$PGP_SERVERS\""; \ - EX_TEMPFAIL=75; \ - exit $EX_TEMPFAIL; \ - fi; \ - unset attempt pgp_server refreshed_keys; \ - \ - gpg --no-default-keyring --keyring /tmp/yarn-keyring.gpg --export --output /usr/local/share/keyrings/dl.yarnpkg.com.gpg && \ - rm -f /tmp/yarn-key.asc /tmp/yarn-keyring.gpg && \ - echo "deb [signed-by=/usr/local/share/keyrings/dl.yarnpkg.com.gpg] https://dl.yarnpkg.com/debian/ stable main" > /etc/apt/sources.list.d/yarnpkg.list && \ - apt-get update -o Dir::Etc::sourcelist="sources.list.d/yarnpkg.list" -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0" && \ - curl -sSLO --retry 5 https://deb.nodesource.com/node_18.x/pool/main/n/nodejs/nodejs_${NODE_VERSION}-deb-1nodesource1_amd64.deb && \ - dpkg -i nodejs_${NODE_VERSION}-deb-1nodesource1_amd64.deb && \ - apt-get purge -qy cmdtest && \ - apt-get install --no-install-recommends -qy \ - yarn && \ - rm /tmp/* && \ - rm -rf /var/lib/apt/lists/* - -RUN eval "$(perl -Mlocal::lib)" \ - && cpanm --installdeps --notest . \ - && cpanm --notest \ - Catalyst::Plugin::Cache::HTTP \ +RUN eval "$(perl -Mlocal::lib)" && \ + cpanm Carton && \ + carton install --deployment && \ + cpanm --notest \ Catalyst::Plugin::StackTrace \ - Digest::MD5::File \ - JSON::Any \ - LWP::Protocol::https \ Plack::Handler::Starlet \ Plack::Middleware::Debug::Base \ Server::Starter \ Starlet \ Starlet::Server \ Term::Size::Any \ - TURNSTEP/DBD-Pg-3.16.0.tar.gz \ - && rm -rf /root/.cpan* /root/perl5/man/ + && \ + rm -rf /root/.cpan* /root/perl5/man/ +RUN install -m 0755 \ + admin/replication/hooks/post-process.sample \ + admin/replication/hooks/post-process COPY DBDefs.pm /musicbrainz-server/lib/ COPY scripts/* /usr/local/bin/ -RUN cat /usr/local/bin/snippet.perllocallib.bashrc >> ~/.bashrc \ - && rm /usr/local/bin/snippet.perllocallib.bashrc \ - && ln -s /usr/local/bin/docker-entrypoint.sh / +RUN cat /usr/local/bin/snippet.perllocallib.bashrc >> ~/.bashrc && \ + rm /usr/local/bin/snippet.perllocallib.bashrc && \ + ln -s /usr/local/bin/docker-entrypoint.sh / # Postgres user/password would be solely needed to compile tests ARG POSTGRES_USER=doesntmatteraslongasyoudontcompiletests ARG POSTGRES_PASSWORD=doesntmatteraslongasyoudontcompiletests ENV BASH_ENV=/noninteractive.bash_env \ - MUSICBRAINZ_BASE_DOWNLOAD_URL=http://ftp.eu.metabrainz.org/pub/musicbrainz \ + MUSICBRAINZ_BASE_DOWNLOAD_URL=https://data.metabrainz.org/pub/musicbrainz \ MUSICBRAINZ_CATALYST_DEBUG=0 \ - MUSICBRAINZ_DB_SCHEMA_SEQUENCE=28 \ + MUSICBRAINZ_DB_SCHEMA_SEQUENCE=29 \ MUSICBRAINZ_DEVELOPMENT_SERVER=0 \ + MUSICBRAINZ_POSTGRES_SERVER=db \ + MUSICBRAINZ_POSTGRES_READONLY_SERVER=db \ + MUSICBRAINZ_RABBITMQ_SERVER=mq \ + MUSICBRAINZ_REDIS_SERVER=redis \ MUSICBRAINZ_SEARCH_SERVER=search:8983/solr \ MUSICBRAINZ_SERVER_PROCESSES=10 \ MUSICBRAINZ_STANDALONE_SERVER=0 \ @@ -135,13 +144,14 @@ ENV BASH_ENV=/noninteractive.bash_env \ MUSICBRAINZ_WEB_SERVER_PORT=5000 \ # Needed for yarn to install production dependencies only NODE_ENV=production \ + PERL_CARTON_CPANFILE=/musicbrainz-server/cpanfile \ POSTGRES_USER=musicbrainz \ POSTGRES_PASSWORD=musicbrainz -RUN yarn install \ - && yarn cache clean \ - && eval "$(perl -Mlocal::lib)" \ - && /musicbrainz-server/script/compile_resources.sh +RUN yarn install && \ + yarn cache clean && \ + eval "$(perl -Mlocal::lib)" && \ + carton exec -- /musicbrainz-server/script/compile_resources.sh ENTRYPOINT ["docker-entrypoint.sh"] CMD ["start.sh"] diff --git a/build/musicbrainz/keyrings/nodesource.asc b/build/musicbrainz/keyrings/nodesource.asc new file mode 100644 index 00000000..b7637b89 --- /dev/null +++ b/build/musicbrainz/keyrings/nodesource.asc @@ -0,0 +1,29 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQENBFdDN1ABCADaNd/I3j3tn40deQNgz7hB2NvT+syXe6k4ZmdiEcOfBvFrkS8B +hNS67t93etHsxEy7E0qwsZH32bKazMqe9zDwoa3aVImryjh6SHC9lMtW27JPHFeM +Srkt9YmH1WMwWcRO6eSY9B3PpazquhnvbammLuUojXRIxkDroy6Fw4UKmUNSRr32 +9Ej87jRoR1B2/57Kfp2Y4+vFGGzSvh3AFQpBHq51qsNHALU6+8PjLfIt+5TPvaWR +TB+kAZnQZkaIQM2nr1n3oj6ak2RATY/+kjLizgFWzgEfbCrbsyq68UoY5FPBnu4Z +E3iDZpaIqwKr0seUC7iA1xM5eHi5kty1oB7HABEBAAG0Ik5Tb2xpZCA8bnNvbGlk +LWdwZ0Bub2Rlc291cmNlLmNvbT6JATgEEwECACIFAldDN1ACGwMGCwkIBwMCBhUI +AgkKCwQWAgMBAh4BAheAAAoJEC9ZtfmbG+C0y7wH/i4xnab36dtrYW7RZwL8i6Sc +NjMx4j9+U1kr/F6YtqWd+JwCbBdar5zRghxPcYEq/qf7MbgAYcs1eSOuTOb7n7+o +xUwdH2iCtHhKh3Jr2mRw1ks7BbFZPB5KmkxHaEBfLT4d+I91ZuUdPXJ+0SXs9gzk +Dbz65Uhoz3W03aiF8HeL5JNARZFMbHHNVL05U1sTGTCOtu+1c/33f3TulQ/XZ3Y4 +hwGCpLe0Tv7g7Lp3iLMZMWYPEa0a7S4u8he5IEJQLd8bE8jltcQvrdr3Fm8kI2Jg +BJmUmX4PSfhuTCFaR/yeCt3UoW883bs9LfbTzIx9DJGpRIu8Y0IL3b4sj/GoZVq5 +AQ0EV0M3UAEIAKrTaC62ayzqOIPa7nS90BHHck4Z33a2tZF/uof38xNOiyWGhT8u +JeFoTTHn5SQq5Ftyu4K3K2fbbpuu/APQF05AaljzVkDGNMW4pSkgOasdysj831cu +ssrHX2RYS22wg80k6C/Hwmh5F45faEuNxsV+bPx7oPUrt5n6GMx84vEP3i1+FDBi +0pt/B/QnDFBXki1BGvJ35f5NwDefK8VaInxXP3ZN/WIbtn5dqxppkV/YkO7GiJlp +Jlju9rf3kKUIQzKQWxFsbCAPIHoWv7rH9RSxgDithXtG6Yg5R1aeBbJaPNXL9wpJ +YBJbiMjkAFaz4B95FOqZm3r7oHugiCGsHX0AEQEAAYkBHwQYAQIACQUCV0M3UAIb +DAAKCRAvWbX5mxvgtE/OB/0VN88DR3Y3fuqy7lq/dthkn7Dqm9YXdorZl3L152eE +IF882aG8FE3qZdaLGjQO4oShAyNWmRfSGuoH0XERXAI9n0r8m4mDMxE6rtP7tHet +y/5M8x3CTyuMgx5GLDaEUvBusnTD+/v/fBMwRK/cZ9du5PSG4R50rtst+oYyC2ao +x4I2SgjtF/cY7bECsZDplzatN3gv34PkcdIg8SLHAVlL4N5tzumDeizRspcSyoy2 +K2+hwKU4C4+dekLLTg8rjnRROvplV2KtaEk6rxKtIRFDCoQng8wfJuIMrDNKvqZw +FRGt7cbvW5MCnuH8MhItOl9Uxp1wHp6gtav/h8Gp6MBa +=MARt +-----END PGP PUBLIC KEY BLOCK----- diff --git a/build/musicbrainz/keyrings/pgdg.asc b/build/musicbrainz/keyrings/pgdg.asc new file mode 100644 index 00000000..8480576e --- /dev/null +++ b/build/musicbrainz/keyrings/pgdg.asc @@ -0,0 +1,77 @@ +-----BEGIN PGP PUBLIC KEY BLOCK----- + +mQINBE6XR8IBEACVdDKT2HEH1IyHzXkb4nIWAY7echjRxo7MTcj4vbXAyBKOfjja +UrBEJWHN6fjKJXOYWXHLIYg0hOGeW9qcSiaa1/rYIbOzjfGfhE4x0Y+NJHS1db0V +G6GUj3qXaeyqIJGS2z7m0Thy4Lgr/LpZlZ78Nf1fliSzBlMo1sV7PpP/7zUO+aA4 +bKa8Rio3weMXQOZgclzgeSdqtwKnyKTQdXY5MkH1QXyFIk1nTfWwyqpJjHlgtwMi +c2cxjqG5nnV9rIYlTTjYG6RBglq0SmzF/raBnF4Lwjxq4qRqvRllBXdFu5+2pMfC +IZ10HPRdqDCTN60DUix+BTzBUT30NzaLhZbOMT5RvQtvTVgWpeIn20i2NrPWNCUh +hj490dKDLpK/v+A5/i8zPvN4c6MkDHi1FZfaoz3863dylUBR3Ip26oM0hHXf4/2U +A/oA4pCl2W0hc4aNtozjKHkVjRx5Q8/hVYu+39csFWxo6YSB/KgIEw+0W8DiTII3 +RQj/OlD68ZDmGLyQPiJvaEtY9fDrcSpI0Esm0i4sjkNbuuh0Cvwwwqo5EF1zfkVj +Tqz2REYQGMJGc5LUbIpk5sMHo1HWV038TWxlDRwtOdzw08zQA6BeWe9FOokRPeR2 +AqhyaJJwOZJodKZ76S+LDwFkTLzEKnYPCzkoRwLrEdNt1M7wQBThnC5z6wARAQAB +tBxQb3N0Z3JlU1FMIERlYmlhbiBSZXBvc2l0b3J5iQJOBBMBCAA4AhsDBQsJCAcD +BRUKCQgLBRYCAwEAAh4BAheAFiEEuXsK/KoaR/BE8kSgf8x9RqzMTPgFAlhtCD8A +CgkQf8x9RqzMTPgECxAAk8uL+dwveTv6eH21tIHcltt8U3Ofajdo+D/ayO53LiYO +xi27kdHD0zvFMUWXLGxQtWyeqqDRvDagfWglHucIcaLxoxNwL8+e+9hVFIEskQAY +kVToBCKMXTQDLarz8/J030Pmcv3ihbwB+jhnykMuyyNmht4kq0CNgnlcMCdVz0d3 +z/09puryIHJrD+A8y3TD4RM74snQuwc9u5bsckvRtRJKbP3GX5JaFZAqUyZNRJRJ +Tn2OQRBhCpxhlZ2afkAPFIq2aVnEt/Ie6tmeRCzsW3lOxEH2K7MQSfSu/kRz7ELf +Cz3NJHj7rMzC+76Rhsas60t9CjmvMuGONEpctijDWONLCuch3Pdj6XpC+MVxpgBy +2VUdkunb48YhXNW0jgFGM/BFRj+dMQOUbY8PjJjsmVV0joDruWATQG/M4C7O8iU0 +B7o6yVv4m8LDEN9CiR6r7H17m4xZseT3f+0QpMe7iQjz6XxTUFRQxXqzmNnloA1T +7VjwPqIIzkj/u0V8nICG/ktLzp1OsCFatWXh7LbU+hwYl6gsFH/mFDqVxJ3+DKQi +vyf1NatzEwl62foVjGUSpvh3ymtmtUQ4JUkNDsXiRBWczaiGSuzD9Qi0ONdkAX3b +ewqmN4TfE+XIpCPxxHXwGq9Rv1IFjOdCX0iG436GHyTLC1tTUIKF5xV4Y0+cXIOI +RgQQEQgABgUCTpdI7gAKCRDFr3dKWFELWqaPAKD1TtT5c3sZz92Fj97KYmqbNQZP ++ACfSC6+hfvlj4GxmUjp1aepoVTo3weJAhwEEAEIAAYFAk6XSQsACgkQTFprqxLS +p64F8Q//cCcutwrH50UoRFejg0EIZav6LUKejC6kpLeubbEtuaIH3r2zMblPGc4i ++eMQKo/PqyQrceRXeNNlqO6/exHozYi2meudxa6IudhwJIOn1MQykJbNMSC2sGUp +1W5M1N5EYgt4hy+qhlfnD66LR4G+9t5FscTJSy84SdiOuqgCOpQmPkVRm1HX5X1+ +dmnzMOCk5LHHQuiacV0qeGO7JcBCVEIDr+uhU1H2u5GPFNHm5u15n25tOxVivb94 +xg6NDjouECBH7cCVuW79YcExH/0X3/9G45rjdHlKPH1OIUJiiX47OTxdG3dAbB4Q +fnViRJhjehFscFvYWSqXo3pgWqUsEvv9qJac2ZEMSz9x2mj0ekWxuM6/hGWxJdB+ ++985rIelPmc7VRAXOjIxWknrXnPCZAMlPlDLu6+vZ5BhFX0Be3y38f7GNCxFkJzl +hWZ4Cj3WojMj+0DaC1eKTj3rJ7OJlt9S9xnO7OOPEUTGyzgNIDAyCiu8F4huLPaT +ape6RupxOMHZeoCVlqx3ouWctelB2oNXcxxiQ/8y+21aHfD4n/CiIFwDvIQjl7dg +mT3u5Lr6yxuosR3QJx1P6rP5ZrDTP9khT30t+HZCbvs5Pq+v/9m6XDmi+NlU7Zuh +Ehy97tL3uBDgoL4b/5BpFL5U9nruPlQzGq1P9jj40dxAaDAX/WKJAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlB5KywFCQPDFt8ACgkQf8x9RqzM +TPhuCQ//QAjRSAOCQ02qmUAikT+mTB6baOAakkYq6uHbEO7qPZkv4E/M+HPIJ4wd +nBNeSQjfvdNcZBA/x0hr5EMcBneKKPDj4hJ0panOIRQmNSTThQw9OU351gm3YQct +AMPRUu1fTJAL/AuZUQf9ESmhyVtWNlH/56HBfYjE4iVeaRkkNLJyX3vkWdJSMwC/ +LO3Lw/0M3R8itDsm74F8w4xOdSQ52nSRFRh7PunFtREl+QzQ3EA/WB4AIj3VohIG +kWDfPFCzV3cyZQiEnjAe9gG5pHsXHUWQsDFZ12t784JgkGyO5wT26pzTiuApWM3k +/9V+o3HJSgH5hn7wuTi3TelEFwP1fNzI5iUUtZdtxbFOfWMnZAypEhaLmXNkg4zD +kH44r0ss9fR0DAgUav1a25UnbOn4PgIEQy2fgHKHwRpCy20d6oCSlmgyWsR40EPP +YvtGq49A2aK6ibXmdvvFT+Ts8Z+q2SkFpoYFX20mR2nsF0fbt1lfH65P64dukxeR +GteWIeNakDD40bAAOH8+OaoTGVBJ2ACJfLVNM53PEoftavAwUYMrR910qvwYfd/4 +6rh46g1Frr9SFMKYE9uvIJIgDsQB3QBp71houU4H55M5GD8XURYs+bfiQpJG1p7e +B8e5jZx1SagNWc4XwL2FzQ9svrkbg1Y+359buUiP7T6QXX2zY++JAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlEqbZUFCQg2wEEACgkQf8x9RqzM +TPhFMQ//WxAfKMdpSIA9oIC/yPD/dJpY/+DyouOljpE6MucMy/ArBECjFTBwi/j9 +NYM4ynAk34IkhuNexc1i9/05f5RM6+riLCLgAOsADDbHD4miZzoSxiVr6GQ3YXMb +OGld9kV9Sy6mGNjcUov7iFcf5Hy5w3AjPfKuR9zXswyfzIU1YXObiiZT38l55pp/ +BSgvGVQsvbNjsff5CbEKXS7q3xW+WzN0QWF6YsfNVhFjRGj8hKtHvwKcA02wwjLe +LXVTm6915ZUKhZXUFc0vM4Pj4EgNswH8Ojw9AJaKWJIZmLyW+aP+wpu6YwVCicxB +Y59CzBO2pPJDfKFQzUtrErk9irXeuCCLesDyirxJhv8o0JAvmnMAKOLhNFUrSQ2m ++3EnF7zhfz70gHW+EG8X8mL/EN3/dUM09j6TVrjtw43RLxBzwMDeariFF9yC+5bL +tnGgxjsB9Ik6GV5v34/NEEGf1qBiAzFmDVFRZlrNDkq6gmpvGnA5hUWNr+y0i01L +jGyaLSWHYjgw2UEQOqcUtTFK9MNzbZze4mVaHMEz9/aMfX25R6qbiNqCChveIm8m +Yr5Ds2zdZx+G5bAKdzX7nx2IUAxFQJEE94VLSp3npAaTWv3sHr7dR8tSyUJ9poDw +gw4W9BIcnAM7zvFYbLF5FNggg/26njHCCN70sHt8zGxKQINMc6SJAj0EEwEIACcC +GwMFCwkIBwMFFQoJCAsFFgIDAQACHgECF4AFAlLpFRkFCQ6EJy0ACgkQf8x9RqzM +TPjOZA//Zp0e25pcvle7cLc0YuFr9pBv2JIkLzPm83nkcwKmxaWayUIG4Sv6pH6h +m8+S/CHQij/yFCX+o3ngMw2J9HBUvafZ4bnbI0RGJ70GsAwraQ0VlkIfg7GUw3Tz +voGYO42rZTru9S0K/6nFP6D1HUu+U+AsJONLeb6oypQgInfXQExPZyliUnHdipei +4WR1YFW6sjSkZT/5C3J1wkAvPl5lvOVthI9Zs6bZlJLZwusKxU0UM4Btgu1Sf3nn +JcHmzisixwS9PMHE+AgPWIGSec/N27a0KmTTvImV6K6nEjXJey0K2+EYJuIBsYUN +orOGBwDFIhfRk9qGlpgt0KRyguV+AP5qvgry95IrYtrOuE7307SidEbSnvO5ezNe +mE7gT9Z1tM7IMPfmoKph4BfpNoH7aXiQh1Wo+ChdP92hZUtQrY2Nm13cmkxYjQ4Z +gMWfYMC+DA/GooSgZM5i6hYqyyfAuUD9kwRN6BqTbuAUAp+hCWYeN4D88sLYpFh3 +paDYNKJ+Gf7Yyi6gThcV956RUFDH3ys5Dk0vDL9NiWwdebWfRFbzoRM3dyGP889a +OyLzS3mh6nHzZrNGhW73kslSQek8tjKrB+56hXOnb4HaElTZGDvD5wmrrhN94kby +Gtz3cydIohvNO9d90+29h0eGEDYti7j7maHkBKUAwlcPvMg5m3Y= +=DA1T +-----END PGP PUBLIC KEY BLOCK----- diff --git a/build/musicbrainz/scripts/createdb.sh b/build/musicbrainz/scripts/createdb.sh index 89f136ac..6ea029c0 100755 --- a/build/musicbrainz/scripts/createdb.sh +++ b/build/musicbrainz/scripts/createdb.sh @@ -94,14 +94,14 @@ for F in "${DUMP_FILES[@]}"; do done echo "found existing dumps" -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 mkdir -p $TMP_DIR cd /media/dbdump INITDB_OPTIONS='--echo --import' -if ! /musicbrainz-server/script/database_exists MAINTENANCE; then +if ! carton exec -- /musicbrainz-server/script/database_exists MAINTENANCE; then INITDB_OPTIONS="--createdb $INITDB_OPTIONS" fi # shellcheck disable=SC2086 -/musicbrainz-server/admin/InitDb.pl $INITDB_OPTIONS -- --skip-editor --tmp-dir $TMP_DIR "${DUMP_FILES[@]}" +carton exec -- /musicbrainz-server/admin/InitDb.pl $INITDB_OPTIONS -- --skip-editor --tmp-dir $TMP_DIR "${DUMP_FILES[@]}" diff --git a/build/musicbrainz/scripts/indexer-triggers.sh b/build/musicbrainz/scripts/indexer-triggers.sh index 9c7099c2..de013bcf 100755 --- a/build/musicbrainz/scripts/indexer-triggers.sh +++ b/build/musicbrainz/scripts/indexer-triggers.sh @@ -12,18 +12,18 @@ fi INDEXER_SQL_DIR="$1" -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 cd /musicbrainz-server case "$2" in create) - admin/psql < "$INDEXER_SQL_DIR/CreateFunctions.sql" - admin/psql < "$INDEXER_SQL_DIR/CreateTriggers.sql" + carton exec -- admin/psql < "$INDEXER_SQL_DIR/CreateFunctions.sql" + carton exec -- admin/psql < "$INDEXER_SQL_DIR/CreateTriggers.sql" ;; drop ) - admin/psql < "$INDEXER_SQL_DIR/DropTriggers.sql" - admin/psql < "$INDEXER_SQL_DIR/DropFunctions.sql" + carton exec -- admin/psql < "$INDEXER_SQL_DIR/DropTriggers.sql" + carton exec -- admin/psql < "$INDEXER_SQL_DIR/DropFunctions.sql" rm -frv "$INDEXER_SQL_DIR" ;; * ) diff --git a/build/musicbrainz/scripts/load-crontab-only.sh b/build/musicbrainz/scripts/load-crontab-only.sh index 8dcd6fd8..0935f3ee 100755 --- a/build/musicbrainz/scripts/load-crontab-only.sh +++ b/build/musicbrainz/scripts/load-crontab-only.sh @@ -3,8 +3,8 @@ set -e -u dockerize \ - -wait tcp://db:5432 -timeout 60s \ - -wait tcp://redis:6379 -timeout 60s \ + -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s \ + -wait "tcp://${MUSICBRAINZ_REDIS_SERVER}:6379" -timeout 60s \ true if [ -f /crons.conf -a -s /crons.conf ] diff --git a/build/musicbrainz/scripts/recreatedb.sh b/build/musicbrainz/scripts/recreatedb.sh index b325d420..86ac8548 100755 --- a/build/musicbrainz/scripts/recreatedb.sh +++ b/build/musicbrainz/scripts/recreatedb.sh @@ -2,5 +2,5 @@ set -e -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 -psql postgres -U musicbrainz -h db -c "DROP DATABASE IF EXISTS musicbrainz_db;"; createdb.sh "$@" +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 +psql postgres -U musicbrainz -h "${MUSICBRAINZ_POSTGRES_SERVER}" -c "DROP DATABASE IF EXISTS musicbrainz_db;"; createdb.sh "$@" diff --git a/build/musicbrainz/scripts/replication.sh b/build/musicbrainz/scripts/replication.sh index 8d98a28d..532d07e3 100755 --- a/build/musicbrainz/scripts/replication.sh +++ b/build/musicbrainz/scripts/replication.sh @@ -2,5 +2,5 @@ set -e -dockerize -wait tcp://db:5432 -timeout 60s sleep 0 -exec /musicbrainz-server/admin/cron/mirror.sh +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s sleep 0 +exec carton exec -- /musicbrainz-server/admin/cron/mirror.sh diff --git a/build/musicbrainz/scripts/start.sh b/build/musicbrainz/scripts/start.sh index 6e5b86e0..079b6597 100755 --- a/build/musicbrainz/scripts/start.sh +++ b/build/musicbrainz/scripts/start.sh @@ -6,10 +6,10 @@ if ! grep -q -s \ "//${MUSICBRAINZ_WEB_SERVER_HOST}:${MUSICBRAINZ_WEB_SERVER_PORT}" \ /musicbrainz-server/root/static/build/runtime.js.map then - /musicbrainz-server/script/compile_resources.sh + carton exec -- /musicbrainz-server/script/compile_resources.sh fi -dockerize -wait tcp://db:5432 -timeout 60s -wait tcp://mq:5672 -timeout 60s -wait tcp://redis:6379 -timeout 60s start_mb_renderer.pl +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s -wait "tcp://${MUSICBRAINZ_RABBITMQ_SERVER}:5672" -timeout 60s -wait "tcp://${MUSICBRAINZ_REDIS_SERVER}:6379" -timeout 60s carton exec -- start_mb_renderer.pl if [ -f /crons.conf -a -s /crons.conf ] then @@ -17,4 +17,4 @@ then cron -f & fi -start_server --port=5000 -- plackup -I lib -s Starlet -E deployment --max-workers ${MUSICBRAINZ_SERVER_PROCESSES} --pid fcgi.pid +carton exec -- start_server --port=5000 -- plackup -I lib -s Starlet -E deployment --max-workers ${MUSICBRAINZ_SERVER_PROCESSES} --pid fcgi.pid diff --git a/build/musicbrainz/scripts/upgrade-db-schema.sh b/build/musicbrainz/scripts/upgrade-db-schema.sh index c5da90ff..02e275ff 100755 --- a/build/musicbrainz/scripts/upgrade-db-schema.sh +++ b/build/musicbrainz/scripts/upgrade-db-schema.sh @@ -2,6 +2,6 @@ set -e -u -export MUSICBRAINZ_DB_SCHEMA_SEQUENCE=27 +export MUSICBRAINZ_DB_SCHEMA_SEQUENCE=28 -dockerize -wait tcp://db:5432 -timeout 60s ./upgrade.sh +dockerize -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s carton exec -- ./upgrade.sh diff --git a/build/postgres-prebuilt/Dockerfile b/build/postgres-prebuilt/Dockerfile new file mode 100644 index 00000000..89aa810a --- /dev/null +++ b/build/postgres-prebuilt/Dockerfile @@ -0,0 +1,3 @@ +ARG POSTGRES_VERSION=16 +ARG DB_BUILD_SEQUENCE=0 +FROM metabrainz/musicbrainz-docker-db:${POSTGRES_VERSION}-build${DB_BUILD_SEQUENCE} diff --git a/build/postgres/Dockerfile b/build/postgres/Dockerfile index 413147de..5d13e2e9 100644 --- a/build/postgres/Dockerfile +++ b/build/postgres/Dockerfile @@ -1,8 +1,10 @@ -ARG POSTGRES_VERSION=12 -FROM postgres:${POSTGRES_VERSION} +ARG POSTGRES_VERSION=16 +ARG POSTGRES_IMAGE_VARIANT=bookworm +ARG POSTGRES_IMAGE_TAG=${POSTGRES_VERSION}-${POSTGRES_IMAGE_VARIANT} +FROM postgres:${POSTGRES_IMAGE_TAG} ARG POSTGRES_VERSION -LABEL org.metabrainz.based-on-image="postgres:${POSTGRES_VERSION}" +LABEL org.metabrainz.based-on-image="postgres:${POSTGRES_IMAGE_TAG}" ARG DEBIAN_FRONTEND=noninteractive @@ -33,6 +35,6 @@ ARG PG_AMQP_GIT_REF="240d477d40c5e7a579b931c98eb29cef4edda164" RUN git clone https://github.com/omniti-labs/pg_amqp.git /tmp/pg_amqp \ && cd /tmp/pg_amqp \ && git checkout "$PG_AMQP_GIT_REF" \ - && make \ + && make PG_CPPFLAGS=-Wno-error=implicit-int \ && make install \ && rm -R /tmp/* diff --git a/build/sir-dev/Dockerfile b/build/sir-dev/Dockerfile index 1c18ece1..9f90f33c 100644 --- a/build/sir-dev/Dockerfile +++ b/build/sir-dev/Dockerfile @@ -52,6 +52,11 @@ RUN echo Requirements will be installed at run time from entrypoint. \ WORKDIR /code +ENV MUSICBRAINZ_RABBITMQ_SERVER=mq \ + MUSICBRAINZ_POSTGRES_SERVER=db \ + MUSICBRAINZ_POSTGRES_READONLY_SERVER=db \ + MUSICBRAINZ_SEARCH_SERVER=search:8983/solr + ENV POSTGRES_USER=musicbrainz ENV POSTGRES_PASSWORD=musicbrainz ENV PYTHONUSERBASE="/code/venv-musicbrainz-docker" diff --git a/build/sir-dev/scripts/docker-entrypoint.sh b/build/sir-dev/scripts/docker-entrypoint.sh index a451e6d3..361d5308 100755 --- a/build/sir-dev/scripts/docker-entrypoint.sh +++ b/build/sir-dev/scripts/docker-entrypoint.sh @@ -12,8 +12,8 @@ pip install \ -r requirements_dev.txt dockerize \ - -wait tcp://db:5432 -timeout 60s \ - -wait tcp://mq:5672 -timeout 60s \ + -wait "tcp://${MUSICBRAINZ_POSTGRES_SERVER}:5432" -timeout 60s \ + -wait "tcp://${MUSICBRAINZ_RABBITMQ_SERVER}:5672" -timeout 60s \ -wait tcp://search:8983 -timeout 60s \ sleep 0 diff --git a/build/sir/Dockerfile b/build/sir/Dockerfile index 6666f350..a38c5492 100644 --- a/build/sir/Dockerfile +++ b/build/sir/Dockerfile @@ -46,5 +46,11 @@ RUN git clone --depth=1 --branch "v${SIR_VERSION}" https://github.com/metabrainz WORKDIR /code +ENV MUSICBRAINZ_RABBITMQ_SERVER=mq \ + MUSICBRAINZ_POSTGRES_SERVER=db \ + MUSICBRAINZ_POSTGRES_READONLY_SERVER=db \ + MUSICBRAINZ_SEARCH_SERVER=search:8983/solr + ENV POSTGRES_USER=musicbrainz ENV POSTGRES_PASSWORD=musicbrainz + diff --git a/compose/db-own-build.yml b/compose/db-own-build.yml new file mode 100644 index 00000000..785b8c3d --- /dev/null +++ b/compose/db-own-build.yml @@ -0,0 +1,10 @@ +version: '3.1' + +# Description: Build own image of Postgres with AMQP support for release + +services: + db: + build: + context: build/postgres + args: + - POSTGRES_VERSION=${POSTGRES_VERSION:-16} diff --git a/compose/musicbrainz-dev.yml b/compose/musicbrainz-dev.yml index dd872c4b..5430e98a 100644 --- a/compose/musicbrainz-dev.yml +++ b/compose/musicbrainz-dev.yml @@ -6,8 +6,11 @@ services: musicbrainz: build: context: build/musicbrainz-dev + args: + - POSTGRES_VERSION=${POSTGRES_VERSION:-16} volumes: - ${MUSICBRAINZ_SERVER_LOCAL_ROOT:?Missing path of MusicBrainz Server working copy}:/musicbrainz-server + - ./.musicbrainz-dev.bash_history.d/:/root/.bash_history.d/:z environment: - MUSICBRAINZ_CATALYST_DEBUG=${MUSICBRAINZ_CATALYST_DEBUG:-0} - MUSICBRAINZ_DEVELOPMENT_SERVER=${MUSICBRAINZ_DEVELOPMENT_SERVER:-1} diff --git a/compose/musicbrainz-own-build.yml b/compose/musicbrainz-own-build.yml new file mode 100644 index 00000000..324fc554 --- /dev/null +++ b/compose/musicbrainz-own-build.yml @@ -0,0 +1,10 @@ +version: '3.1' + +# Description: Build own image of MusicBrainz Server for release + +services: + musicbrainz: + build: + context: build/musicbrainz + args: + - POSTGRES_VERSION=${POSTGRES_VERSION:-16} diff --git a/compose/sir-dev.yml b/compose/sir-dev.yml index 504ae69b..996402e6 100644 --- a/compose/sir-dev.yml +++ b/compose/sir-dev.yml @@ -4,12 +4,17 @@ version: '3.1' services: indexer: - build: build/sir-dev + build: + context: build/sir-dev + args: + - PYTHON_VERSION=${SIR_DEV_PYTHON_VERSION:-2.7} + - BASE_IMAGE_DATE=${SIR_DEV_BASE_IMAGE_DATE:-20220421} + - SIR_VERSION=${SIR_DEV_VERSION:-py27-stage1} env_file: - ./default/postgres.env volumes: - - ${SIR_LOCAL_ROOT:?Missing path of sir working copy}:/code - - ${SIR_CONFIG_PATH:-./default/indexer.ini}:/code/config.ini + - ${SIR_DEV_LOCAL_ROOT:-../sir}:/code + - ${SIR_DEV_CONFIG_PATH:-./default/indexer.ini}:/code/config.ini depends_on: - db - mq diff --git a/default/indexer.ini b/default/indexer.ini index 6198c3fd..dd827a21 100644 --- a/default/indexer.ini +++ b/default/indexer.ini @@ -1,12 +1,12 @@ [database] dbname = musicbrainz_db -host = db +host = ${MUSICBRAINZ_POSTGRES_SERVER} password = ${POSTGRES_PASSWORD} port = 5432 user = ${POSTGRES_USER} [solr] -uri = http://search:8983/solr +uri = http://${MUSICBRAINZ_SEARCH_SERVER} batch_size = 200 [sir] @@ -18,11 +18,11 @@ query_batch_size = 5000 wscompat = on [rabbitmq] -host = mq +host = ${MUSICBRAINZ_RABBITMQ_SERVER} user = sir password = sir vhost = /search-index-rebuilder prefetch_count = 350 [sentry] -dsn = "" +dsn = diff --git a/docker-compose.alt.db-only-mirror.yml b/docker-compose.alt.db-only-mirror.yml index fe044831..6a0d1a3d 100644 --- a/docker-compose.alt.db-only-mirror.yml +++ b/docker-compose.alt.db-only-mirror.yml @@ -11,10 +11,8 @@ volumes: services: db: build: - context: build/postgres - args: - - POSTGRES_VERSION=${POSTGRES_VERSION:-12} - image: musicbrainz-docker_db:${POSTGRES_VERSION:-12} + context: build/postgres-prebuilt + image: musicbrainz-docker_db:${POSTGRES_VERSION:-16} logging: driver: "json-file" options: @@ -32,9 +30,7 @@ services: musicbrainz: build: - context: build/musicbrainz - args: - - POSTGRES_VERSION=${POSTGRES_VERSION:-12} + context: build/musicbrainz-prebuilt logging: driver: "json-file" options: @@ -47,7 +43,7 @@ services: - ./default/postgres.env environment: - MUSICBRAINZ_BASE_FTP_URL=${MUSICBRAINZ_BASE_FTP_URL:-} - - MUSICBRAINZ_BASE_DOWNLOAD_URL=${MUSICBRAINZ_BASE_DOWNLOAD_URL:-http://ftp.eu.metabrainz.org/pub/musicbrainz} + - MUSICBRAINZ_BASE_DOWNLOAD_URL=${MUSICBRAINZ_BASE_DOWNLOAD_URL:-https://data.metabrainz.org/pub/musicbrainz} - MUSICBRAINZ_WEB_SERVER_HOST=${MUSICBRAINZ_WEB_SERVER_HOST:-localhost} - MUSICBRAINZ_WEB_SERVER_PORT=${MUSICBRAINZ_WEB_SERVER_PORT:-5000} command: load-crontab-only.sh diff --git a/docker-compose.yml b/docker-compose.yml index 38915773..2c9d8af5 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -17,10 +17,8 @@ volumes: services: db: build: - context: build/postgres - args: - - POSTGRES_VERSION=${POSTGRES_VERSION:-12} - image: musicbrainz-docker_db:${POSTGRES_VERSION:-12} + context: build/postgres-prebuilt + image: musicbrainz-docker_db:${POSTGRES_VERSION:-16} logging: driver: "json-file" options: @@ -38,9 +36,7 @@ services: musicbrainz: build: - context: build/musicbrainz - args: - - POSTGRES_VERSION=${POSTGRES_VERSION:-12} + context: build/musicbrainz-prebuilt logging: driver: "json-file" options: @@ -56,7 +52,7 @@ services: - ./default/postgres.env environment: - MUSICBRAINZ_BASE_FTP_URL=${MUSICBRAINZ_BASE_FTP_URL:-} - - MUSICBRAINZ_BASE_DOWNLOAD_URL=${MUSICBRAINZ_BASE_DOWNLOAD_URL:-http://ftp.eu.metabrainz.org/pub/musicbrainz} + - MUSICBRAINZ_BASE_DOWNLOAD_URL=${MUSICBRAINZ_BASE_DOWNLOAD_URL:-https://data.metabrainz.org/pub/musicbrainz} - MUSICBRAINZ_SERVER_PROCESSES=${MUSICBRAINZ_SERVER_PROCESSES:-10} - MUSICBRAINZ_USE_PROXY=1 - MUSICBRAINZ_WEB_SERVER_HOST=${MUSICBRAINZ_WEB_SERVER_HOST:-localhost}