Pull upstream changes

.gitignore

@@ -2,6 +2,9 @@
 # See https://docs.docker.com/compose/environment-variables/#the-env-file
 /.env
 
+# Bash history of musicbrainz container in development mode
+/.musicbrainz-dev.bash_history.d/
+
 # File created and managed by the script admin/create-amqp-extension
 /admin/.create-amqp-extension.sql
 

README.md

@@ -47,7 +47,7 @@ search and replication in docker.
 
 * CPU: 16 threads (or 2 without indexed search), x86-64 architecture
 * RAM: 16 GB (or 4 without indexed search)
-* Disk Space: 200 GB (or 100 without indexed search)
+* Disk Space: 250 GB (or 100 without indexed search)
 
 ### Required software
 
@@ -87,9 +87,9 @@ If you use [UFW](https://help.ubuntu.com/community/UFW) to manage your firewall:
 
 ## Components version
 
-* Current MB Branch: [v-2023-10-24-hotfix](build/musicbrainz/Dockerfile#L53)
-* Current DB_SCHEMA_SEQUENCE: [28](build/musicbrainz/Dockerfile#L129)
-* Postgres Version: [12](docker-compose.yml)
+* Current MB Branch: [v-2025-02-24.0](build/musicbrainz/Dockerfile#L88)
+* Current DB_SCHEMA_SEQUENCE: [29](build/musicbrainz/Dockerfile#L125)
+* Postgres Version: [16](docker-compose.yml)
   (can be changed by setting the environment variable `POSTGRES_VERSION`)
 * MB Solr search server: [3.4.2](docker-compose.yml#L88)
   (can be changed by setting the environment variable `MB_SOLR_VERSION`)
@@ -152,7 +152,7 @@ tables, the server will generally fall back to slower queries in their place.
 If you wish to configure the materialized tables, you can run:
 
 ```bash
-sudo docker-compose exec musicbrainz bash -c './admin/BuildMaterializedTables --database=MAINTENANCE all'
+sudo docker-compose exec musicbrainz bash -c 'carton exec -- ./admin/BuildMaterializedTables --database=MAINTENANCE all'
 ```
 
 ### Start website
@@ -364,7 +364,7 @@ This number can be changed using the Docker environment variable
 #### Customize download server
 
 By default, data dumps and pre-built search indexes are downloaded from
-`http://ftp.eu.metabrainz.org/pub/musicbrainz`.
+`https://data.metabrainz.org/pub/musicbrainz`.
 
 The download server can be changed using the Docker environment variable
 `MUSICBRAINZ_BASE_DOWNLOAD_URL`.
@@ -401,6 +401,40 @@ This can be changed by creating a custom configuration file under
 [and finally](https://docs.docker.com/storage/bind-mounts/#choose-the--v-or---mount-flag)
 setting the Docker environment variable `SIR_CONFIG_PATH` to its path.
 
+#### Customize backend Postgres server
+
+By default, the services `indexer` and `musicbrainz` are trying to connect to the host `db` (for both read-only and write host) but the hosts can
+be customized using the `MUSICBRAINZ_POSTGRES_SERVER` and `MUSICBRAINZ_POSTGRES_READONLY_SERVER` environment variables.
+
+Notes:
+* After switching to another Postgres server:
+  * If not transferring data, it is needed to create the database again.
+  * For live indexing, the RabbitMQ server has to still be reachable from the Postgres server.
+* The helper scripts `check-search-indexes` and `create-amqp-extension` won’t work anymore.
+* The service `db` will still be up even if unused.
+
+#### Customize backend RabbitMQ server
+
+By default, the services `db`, `indexer` and `musicbrainz` are trying to connect to the host `mq`
+but the host can be customized using the `MUSICBRAINZ_RABBITMQ_SERVER` environment variable.
+
+Notes:
+* After switching to another RabbitMQ server:
+  - Live indexing requires to go through AMQP Setup again.
+  - If not transferring data, it might be needed to build search indexes again.
+* The helper script `purge-message-queues` won’t work anymore.
+* The service `mq` will still be up even if unused.
+
+#### Customize backend Redis server
+
+By default, the service `musicbrainz` is trying to connect to the host `redis`
+but the host can be customized using the `MUSICBRAINZ_REDIS_SERVER` environment variable.
+
+Notes:
+* After switching to another Redis server:
+  - If not transferring data, MusicBrainz user sessions will be reset.
+* The service `redis` will still be running even if unused.
+
 ### Docker Compose overrides
 
 In Docker Compose, it is possible to override the base configuration using
@@ -421,6 +455,16 @@ Try `admin/configure help` for more information.
 
 #### Publish ports of all services
 
+:warning: The service `search` is currently running Solr 7 in
+standalone mode which is vulnerable to privilege escalation.
+See [CVE-2025-24814](https://lists.apache.org/thread/gl291pn8x9f9n52ys5l0pc0b6qtf0qw1) for details.
+We are working on upgrading to Solr 9 in SolrCloud mode.
+See [SEARCH-685](https://tickets.metabrainz.org/browse/SEARCH-685) for follow-up.
+In general, Solr is strongly recommended to be accessible to your own clients only.
+See [Solr Security](https://cwiki.apache.org/confluence/display/SOLR/SolrSecurity) for details.
+Similarly, other services have not been configured to be safely publicly accessible either.
+Take this warning in consideration when publishing their ports.
+
 To publish ports of services `db`, `mq`, `redis` and `search`
 (additionally to `musicbrainz`) on the host, simply run:
 
@@ -546,9 +590,21 @@ Simply restart the container when checking out a new branch.
 
 This is very similar to the above but for Search Index Rebuilder (SIR):
 
-1. Set the variable `SIR_LOCAL_ROOT` in the `.env` file
+1. Optionally set the following variables in the `.env` file:
+   - `SIR_DEV_CONFIG_PATH`
+     (Default: `./default/config.ini` replacing `SIR_CONFIG_PATH`)
+   - `SIR_DEV_LOCAL_ROOT`
+     (Default: `../sir` assuming that `musicbrainz-docker` and `sir`
+     have been cloned under the same parent directory)
+   - `SIR_DEV_PYTHON_VERSION`
+     (Default: `2.7` matching `metabrainz/python` image tag)
+   - `SIR_DEV_BASE_IMAGE_DATE`
+     (Default: `20220421` matching `metabrainz/python` image tag)
+   - `SIR_DEV_VERSION`
+     (Default: `py27-stage1` which is informative only)
 2. Run `admin/configure add sir-dev`
-3. Run `sudo docker-compose up -d`
+3. Run `sudo docker-compose build indexer`
+4. Run `sudo docker-compose up -d`
 
 Notes:
 

TROUBLESHOOTING.md

@@ -8,6 +8,7 @@
 - [Resolving name failed](#resolving-name-failed)
 - [Loadable library and perl binaries are mismatched](#loadable-library-and-perl-binaries-are-mismatched)
 - [ImportError: No module named](#importerror-no-module-named)
+- [Unknown error executing apt-key](#unknown-error-executing-apt-key)
 
 <!-- tocstop -->
 
@@ -109,3 +110,29 @@ sudo docker-compose restart indexer
 
 Python packages are downloaded again and installed again when the
 service `indexer` restarts.
+
+## Unknown error executing apt-key
+
+When building Docker image for the service `musicbrainz`:
+
+``` log
+Err:1 https://deb.nodesource.com/node_20.x nodistro InRelease
+  Unknown error executing apt-key
+[...]
+W: GPG error: https://deb.nodesource.com/node_20.x nodistro InRelease: Unknown error executing apt-key
+E: The repository 'https://deb.nodesource.com/node_20.x nodistro InRelease' is not signed.
+```
+
+This may happen if your system is hindering file permissions.
+You can find out by adding `RUN ls -l file` commands in the
+Dockerfile.
+
+Solution:
+
+Configure your system to keep the file permissions defined in the Git repository
+and to preserve the permissions of the files copied through Docker.
+
+If it isn’t possible, for example with the Unraid operating system,
+run additional `chmod` commands in the Dockerfile; See comments to the
+issue [#263](https://github.com/metabrainz/musicbrainz-docker/pull/263).
+

admin/lib/upgrade-to-postgres12/do-pg_upgrade.sh → admin/lib/upgrade-to-postgres16/do-pg_upgrade.sh

@@ -4,8 +4,8 @@ set -e
 shopt -s extglob
 
 PGDATA=/var/lib/postgresql/data
-PGDATA_OLD="$PGDATA"/9.5
-PGDATA_NEW="$PGDATA"/12
+PGDATA_OLD="$PGDATA"/12
+PGDATA_NEW="$PGDATA"/16
 PGAMQP_DIR=/tmp/pg_amqp
 
 function cleanup() {
@@ -15,8 +15,8 @@ function cleanup() {
 	fi
 
 	if [[ -d "$PGDATA_OLD" ]]; then
-		echo "Clean $PG_DATA_OLD off but 9.5 data"
-		rm -rf "$PGDATA"/!(9.5)
+		echo "Clean $PG_DATA_OLD off but 12 data"
+		rm -rf "$PGDATA"/!(12)
 		mv -v "$PGDATA_OLD"/* "$PGDATA"
 		rmdir "$PGDATA_OLD"
 	fi
@@ -28,35 +28,38 @@ function cleanup() {
 }
 trap cleanup EXIT
 
-sudo -u postgres /usr/lib/postgresql/9.5/bin/pg_ctl stop -w -D "$PGDATA" \
+sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl stop -w -D "$PGDATA" 2>/dev/null \
 	|| echo 'Assuming server is stopped...'
 
 # We use the --link flag on pg_upgrade below to make hard links instead of
 # copying files, drastically improving the speed of the upgrade. Hard links,
 # of course, require the linked files to be on the same file system, but
 # $PGDATA is the volume *root*. To work around that, we have to move the
-# existing v9.5 cluster to a '9.5' subdir, and create the new v12 cluster in
-# a '12' subdir. Once we're finished, we'll move the new cluster's files
+# existing v12 cluster to a '12' subdir, and create the new v16 cluster in
+# a '16' subdir. Once we're finished, we'll move the new cluster's files
 # back into $PGDATA.
 cd "$PGDATA"
-sudo -u postgres mkdir -p 9.5 12
-chmod 700 9.5 12
-sudo -u postgres mv !(9.5|12) 9.5
+sudo -u postgres mkdir -p 12 16
+chmod 700 12 16
+sudo -u postgres mv !(12|16) 12
 
-sudo -u postgres /usr/lib/postgresql/12/bin/initdb \
+sudo -u postgres /usr/lib/postgresql/16/bin/initdb \
 	--encoding utf8 \
 	--username musicbrainz \
 	"$PGDATA_NEW"
 
-git clone -b "v0.4.1" --depth=1 https://github.com/omniti-labs/pg_amqp.git "$PGAMQP_DIR"
+# There is no tag v0.4.2 (or 0.5.0) yet
+PG_AMQP_GIT_REF="240d477d40c5e7a579b931c98eb29cef4edda164"
+git clone https://github.com/omniti-labs/pg_amqp.git "$PGAMQP_DIR"
 cd "$PGAMQP_DIR"
-make PG_CONFIG=/usr/lib/postgresql/12/bin/pg_config
+git checkout "$PG_AMQP_GIT_REF"
+make PG_CONFIG=/usr/lib/postgresql/16/bin/pg_config PG_CPPFLAGS=-Wno-error=implicit-int
 make install
 cd "$PGDATA"
 
-sudo -u postgres /usr/lib/postgresql/12/bin/pg_upgrade \
-	--old-bindir=/usr/lib/postgresql/9.5/bin/ \
-	--new-bindir=/usr/lib/postgresql/12/bin/ \
+sudo -u postgres /usr/lib/postgresql/16/bin/pg_upgrade \
+	--old-bindir=/usr/lib/postgresql/12/bin/ \
+	--new-bindir=/usr/lib/postgresql/16/bin/ \
 	--old-datadir="$PGDATA_OLD" \
 	--new-datadir="$PGDATA_NEW" \
 	--jobs=3 \
@@ -72,9 +75,5 @@ mv "$PGDATA_NEW"/* "$PGDATA"/
 rmdir "$PGDATA_NEW"
 cp -a "$PGDATA_OLD"/{postgresql.conf,pg_hba.conf} .
 
-# Start the new cluster in the background, so we can apply
-# 20200518-pg12-after-upgrade.sql via the website container.
-sudo -u postgres /usr/lib/postgresql/12/bin/pg_ctl start -w -D "$PGDATA"
-
 ./delete_old_cluster.sh
 rm delete_old_cluster.sh

admin/lib/upgrade-to-postgres12/pg-9.5-stopped.yml → admin/lib/upgrade-to-postgres16/pg-12-stopped.yml

@@ -4,8 +4,8 @@ services:
   db:
     build:
       args:
-        - POSTGRES_VERSION=9.5
-    image: musicbrainz-docker_db:9.5
+        - POSTGRES_VERSION=12
+    image: musicbrainz-docker_db:12
     command: /bin/sleep infinity
 
   musicbrainz:

admin/repository/prebuild-db

@@ -0,0 +1,104 @@
+#!/usr/bin/env bash
+
+set -e -u
+
+# shellcheck source=admin/lib/common.inc.bash
+source "$(dirname "${BASH_SOURCE[0]}")/../lib/common.inc.bash"
+
+HELP=$(cat <<EOH
+Usage: $SCRIPT_NAME
+   or: $SCRIPT_NAME --help
+
+Build a new image for the db service
+from the source code repository of pg_amqp (AMQP Support for Postgres)
+and push it as metabrainz/musicbrainz-docker-db:<tag>
+where tag is based on Postgres major version and a build sequence number.
+
+Prerequisites:
+
+    The version of Postgres must be up-to-date in:
+        README.md
+        build/postgres/Dockerfile
+        build/postgres-prebuilt/Dockerfile
+
+    The build sequence must be incremented, if image tag exists already
+    (for example when only files under build/postgres/ have changed), in:
+        build/postgres-prebuilt/Dockerfile
+EOH
+)
+
+if [[ $# -ne 0 && $1 =~ -*h(elp)? ]]
+then
+  echo "$HELP"
+  exit 0 # EX_OK
+elif [[ $# -ne 0 ]]
+then
+  echo >&2 "$SCRIPT_NAME: unrecognized argument: $1"
+  echo >&2 "Try '$SCRIPT_NAME help' for usage."
+  exit 64 # EX_USAGE
+fi
+
+# Retrieve destination image tag from the default Dockerfile
+
+POSTGRES_VERSION=$(sed -n 's/^ARG POSTGRES_VERSION=\([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile)
+# shellcheck disable=SC2034 # as it used in eval below
+DB_BUILD_SEQUENCE=$(sed -n 's/^ARG DB_BUILD_SEQUENCE=\([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile)
+DEST_IMAGE_TAG=$(eval echo "$(sed -n 's/^FROM \([^ ]*\).*$/\1/p' build/postgres-prebuilt/Dockerfile)")
+
+# Check that the Postgres major version is the same in all other files
+
+README_POSTGRES_VERSION=$(sed -n 's/^.*Postgres Version: \[\([^]]*\)\].*$/\1/p' README.md)
+
+if [[ $POSTGRES_VERSION != "$README_POSTGRES_VERSION" ]]
+then
+  echo >&2 "$SCRIPT_NAME: File differs in Postgres major version:"
+  echo >&2 "$SCRIPT_NAME: * build/postgres-prebuilt/Dockerfile: '$POSTGRES_VERSION'"
+  echo >&2 "$SCRIPT_NAME: * README.md:                          '$README_POSTGRES_VERSION'"
+  echo >&2 "Try '$SCRIPT_NAME help' for usage."
+  exit 65 # EX_DATAERR
+fi
+
+SOURCE_POSTGRES_VERSION=$(sed -n 's/^ARG POSTGRES_VERSION=\([^ ]*\).*$/\1/p' build/postgres/Dockerfile)
+
+if [[ $POSTGRES_VERSION != "$SOURCE_POSTGRES_VERSION" ]]
+then
+  echo >&2 "$SCRIPT_NAME: File differs in Postgres major version:"
+  echo >&2 "$SCRIPT_NAME: * build/postgres-prebuilt/Dockerfile: '$POSTGRES_VERSION'"
+  echo >&2 "$SCRIPT_NAME: * build/postgres/Dockerfile:          '$SOURCE_POSTGRES_VERSION'"
+  echo >&2 "Try '$SCRIPT_NAME help' for usage."
+  exit 65 # EX_DATAERR
+fi
+
+# Check that the image tag doesn’t exist already
+
+if $DOCKER_CMD image pull "$DEST_IMAGE_TAG" 2>/dev/null \
+  || $DOCKER_CMD image ls "$DEST_IMAGE_TAG" 2>/dev/null | grep -Eq "$DEST_IMAGE_TAG"
+then
+  echo >&2 "$SCRIPT_NAME: image tag exists already: $DEST_IMAGE_TAG"
+  echo >&2 "Try '$SCRIPT_NAME help' for usage."
+  exit 65 # EX_DATAERR
+fi
+
+# Do the thing
+
+echo Building...
+
+DOCKER_COMPOSE_OPTS='-f docker-compose.yml -f compose/db-own-build.yml'
+
+# shellcheck disable=SC2086 # intentional word splitting of options
+$DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS build --progress=plain --pull db
+
+echo Tagging...
+
+# shellcheck disable=SC2086 # intentional word splitting of options
+LOCAL_IMAGE_TAG=$($DOCKER_COMPOSE_CMD $DOCKER_COMPOSE_OPTS config --images | grep -o "musicbrainz-docker_db:$POSTGRES_VERSION$")
+
+$DOCKER_CMD tag "$LOCAL_IMAGE_TAG" "$DEST_IMAGE_TAG"
+
+echo Pushing...
+
+$DOCKER_CMD push "$DEST_IMAGE_TAG"
+
+echo Done.
+
+# vi: set et sts=2 sw=2 ts=2 :