Cannot configure replica sets with entrypoint-initdb

[Gaff]

I'm trying to create simple replicaset enabled images using these docker images. @yosifkit had a suggestion in another thread that this can be done by calling rs.initiate() in a /docker-entrypoint-initdb.d/ script.

However if I do this I get the following:

2019-03-26T12:30:25.889+0000 I COMMAND  [conn2] initiate : no configuration specified. Using a default configuration for the set
2019-03-26T12:30:25.889+0000 I COMMAND  [conn2] created this configuration for initiation : { _id: "rs0", version: 1, members: [ { _id: 0, host: "127.0.0.1:27017" } ] }

The problem here is that since the init script binds only to localhost, the replicaset has the wrong hostname.

If I call rs.initiate() after initialization phase I get the following:

2019-03-26T12:32:16.792+0000 I COMMAND  [conn1] initiate : no configuration specified. Using a default configuration for the set
2019-03-26T12:32:16.793+0000 I COMMAND  [conn1] created this configuration for initiation : { _id: "rs0", version: 1, members: [ { _id: 0, host: "mongo:27017" } ] }

This time with the correct hostname.

Is there some way we can resolve this paradox? Either by running a script after the real startup? Or by binding to the proper interfaces during initialisation? Or by forcing the mongo server to accept a replicaset config even if it cannot resolve itself?

Thanks!

[wglambert]

#246 (comment)

you will need something more complex for the other nodes. But no, my hurried Dockerfile will not actually work since the host field in the replica config will be set to 127.0.0.1:27017.

#246 (comment)

Adding automation for setting up a replica set is not something we want to add to the image since it requires an external service like consul in order to reliably coordinate which is the first and where to join.
. . .

[Gaff]

Hmm - I did miss the "my hurried dockerfile will not actually work" point sorry!

Still I think the idea has merit; but-for the squelching of the --bind-ip arguments this would work just fine, and would have some value for single instance deployment / unit test / many other basic cases. [I realise the whole point of force binding localhost is to ensure nobody can connect until the startup scripts are complete. Not sure how to reconcile that.]

If there's a direction you think would be acceptable and could be made to work I'd be willing to have a go at implementing it.

[tianon]

According to https://docs.mongodb.com/manual/reference/method/rs.initiate/, "rs.initiate" can accept a "configuration" argument -- is it possible to provide the hosts list as part of that block, or does "rs.initiate" choke on that? (Worst case you could "rs.reconfig" after your "rs.initiate" to provide the values you want: https://docs.mongodb.com/manual/reference/method/rs.reconfig/)

[Gaff]

@tianon - both rs.initiate and rs.reconfig will fail if you attempt to pass a list of hosts that does not include the current node. So neither of these will work I'm afraid.

[wglambert]

The docker-entrypoint-initdb.d scripts run during an initialization period (and only if the database is empty), during which the container is only listening on localhost so trying to initiate a cluster during that period isn't possible as it won't resolve it's own container hostname.

So you'll probably need some manual intervention after everything is initialized, as using the docker-entrypoint-initdb.d will error with replSet initiate got NodeNotFound: No host described in new configuration 1 for replica set myrepl maps to this node, but then running the same rs.initiate() will work afterwards

[esetnik]

I have the same issue. Was this ever solved?

[zhangyoufu]

Somehow, I wrote an ugly hack to initialize replica set by abusing docker-entrypoint-initdb.d. Hope that helps someone coming to this issue.

[Gaff]

@esetnik - It's kinda hard to fix since Mongodb will sanity check any attempt to configure a replicaset and reject it if it looks invalid. To fix there there would have to be some 'force' option on replicasets within mongo or similar. I'd argue there's a case for this but it's probably not top of anyone's list.

In the end I wrote a sibling docker container that loops waiting for the mongodb to be available externally, configures the replicaset, then shuts down (though for non-test scopes I guess it could forever listen and set the replicasets). A better person might use consul or etcd or similar to co-ordinate all this.

@zhangyoufu's solution is terrible and beautiful at the same time! :D

[esetnik]

What a hack @zhangyoufu! I think there needs to be some support for this in the official docker image. There are several features of MongoDB only available with a replica set and this is a common production configuration. So it makes local development hard if we cannot setup a dev db with the same configuration.

[zhangyoufu]

I think that docker-entrypoint.sh keeps the initdb phase localhost-only intentionally. So that once mongod listens on whatever ip other than localhost, it means mongod is ready to serve.

To keep this mongod instance "private" (do not serve) while initdb is in progress, and circumvent the check insiders.initiate(), we could abuse /etc/hosts and point a hostname/FQDN to 127.0.0.1. After mongod shutdown in the end of initdb, just revert our changes in /etc/hosts and proceed. And maybe we should wait for the finish of the election, in case if any initdb script requires a primary node.

But modifying /etc/hosts is still too hackish to be allowed. HOSTALIASES environment variable looks pretty useful in this situation, as long as we are using glibc (not available for alpine).

[RalfLackinger]

Thank you @zhangyoufu for the hack. It worked (with minor tweaks to be able to handle authentication as well).

It felt a bit too 'hacky' for me though as I don't like abusing or modifying the original entrypoint script behavior. So I did a small workaround with a custom entrypoint that just calls the original one and after that is concluded executes the rs.initiate() command.

#!/usr/bin/env bash

# call default entrypoint
usr/local/bin/docker-entrypoint.sh "$@" &

# check if mongod is already running and the tmp init setup is done
PS_COMMAND="ps aux | grep '[m]ongod .*/etc/mongo/mongod.conf' | grep -v 'docker-entrypoint.sh'"
IS_MONGO_RUNNING=$( bash -c "${PS_COMMAND}" )
while [ -z "${IS_MONGO_RUNNING}" ]
do
  echo "[INFO] Waiting for the MongoDB setup to finish ..."
  sleep 1
  IS_MONGO_RUNNING=$( bash -c "${PS_COMMAND}" )
done
# wait for mongod to be ready for connections
sleep 3

# check if replica set is already initiated
RS_STATUS=$( mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.status().ok" )
if [[ $RS_STATUS -ne 1 ]]
then
  echo "[INFO] Replication set config invalid. Reconfiguring now."
  RS_CONFIG_STATUS=$( mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.status().codeName" )
  if [[ $RS_CONFIG_STATUS == 'InvalidReplicaSetConfig' ]]
  then
    mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin > /dev/null <<EOF
config = rs.config()
config.members[0].host = hostname()
rs.reconfig(config, {force: true})
EOF
  else
    echo "[INFO] MongoDB setup finished. Initiating replicata set."
    mongo --quiet --username $( cat /run/secrets/root-user ) --password $( cat /run/secrets/root-password ) --authenticationDatabase admin --eval "rs.initiate()" > /dev/null
  fi
else
  echo "[INFO] Replication set already initiated."
fi

wait

It is also very hacky, but I think will cope better with updates of the original entrypoint script. Also the code is just some quick first solution, so be aware it might be buggy. But maybe it might be helpful to someone running into the same issue.

[ldeluigi]

Maybe this can be helpful too https://gist.github.com/zhangyoufu/d1d43ac0fa268cda4dd2dfe55a8c834e#gistcomment-3554586

[akshaypuli]

Hi Everyone, We have come across the same issue when trying to initialize a replica set using JS script within the entry point. Was this issue closed with a solution? If so, I would greatly appreciate it if you could point me to it.
Many Thanks

[ericwooley]

@akshaypuli

I was able to do it from this gist: https://gist.github.com/zhangyoufu/d1d43ac0fa268cda4dd2dfe55a8c834e

I added my full setup as a comment as well

[juvenn]

I'v come up a work-around using healthcheck instead of docker-entrypoint-initdb.d .

Define a js script to detect master status, then do init once if necessary:

init = false;
if (!db.isMaster().ismaster) {
  print("Error: primary not ready, initialize ...")
  rs.initiate();
  quit(1);
} else {
  if (!init) {
    admin = db.getSiblingDB("admin");

    admin.createUser(
      {
        user: "test",
        pwd: "pass",
        roles: ["readWriteAnyDatabase"]
      }
    );
    init = true;
  }
}

In docker-compose, define a healthcheck using the script:

    mongodb:
      image: mongo:4.0
      environment:
        - AUTH=no # without password
      tmpfs: /data/db
      hostname: mongodb
      volumes:
        - "./volume/mongo-init2.js:/mongo-init.js"
      command: [mongod, --replSet, 'rs2', --noauth, --maxConns, "10000"]
      healthcheck:
        test: mongo /mongo-init.js
        interval: 5s

Though this requires further healthy coordination between services, where compose version 3.9 is required to support condition: service_healthy

version: "3.9"
...
services:
...
    depends_on:
      mongodb:
        # requires compose version prior 3.0, or 3.9+, but not between
        # see https://stackoverflow.com/a/41854997/108112
        condition: service_healthy

[qhaas]

The docker-entrypoint-initdb.d scripts run during an initialization period (and only if the database is empty), during which the container is only listening on localhost so trying to initiate a cluster during that period isn't possible as it won't resolve it's own container hostname.

Given the current life cycle in which said docker-entrypoint-initdb.d scripts execute won't accommodate the configuration of a replica set (and possibly other configuration changes dependent on later stages) and this issue, as well as various other online discussions, circulate such 'workarounds' to do so, it seems there might be demand for a feature which would provide for a docker-entrypoint-post-dbstartup.d folder to house scripts execute after everything is ready.

[yosifkit]

Given the current life cycle in which said docker-entrypoint-initdb.d scripts execute won't accommodate the configuration of a replica set (and possibly other configuration changes dependent on later stages) and this issue, as well as various other online discussions, circulate such 'workarounds' to do so, it seems there might be demand for a feature which would provide for a docker-entrypoint-post-dbstartup.d folder to house scripts execute after everything is ready

In order to properly do that (run processes after starting the "real" long-term mongod), we would need a supervisor-like process and that is not complexity that we want to add or maintain (and is beyond a "simple" bash script).

By supervisor process, I mean basically the following:

1. stay resident for the life of the container
2. respond to and forward signals
3. reap zombies
4. run current temp server up/down + initdb.d scripts
5. logs from mongod and scripts both appear in stdout/stderr appropriately
6. start "real" mongod and do something if it exits unexpectedly
7. once "real" mongod is "ready", trigger post-dbstartup scripts
8. do something (exit container?) if post-dbstartup scripts fail in any way or if they "never" finish
9. clean-up behavior for failed post-dbstartup scripts?

You don't have to deal with the first 6 if the container is used as-is since that is "free" because of how the image is designed. And then the correct solution is to use your orchestration platform to run something to coordinate the initialization and joining of multiple mongo containers.

Run rs.initiate() on just one and only one mongod instance for the replica set.

- https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/#initiate-the-replica-set

[cherviakovtaskworld]

The docker-entrypoint-initdb.d scripts run during an initialization period (and only if the database is empty), during which the container is only listening on localhost so trying to initiate a cluster during that period isn't possible as it won't resolve it's own container hostname.

So you'll probably need some manual intervention after everything is initialized, as using the docker-entrypoint-initdb.d will error with replSet initiate got NodeNotFound: No host described in new configuration 1 for replica set myrepl maps to this node, but then running the same rs.initiate() will work afterwards

@wglambert can this be made consisten with mongodb behavious outside of the docker? In my case I do have need to initialize replica set of 1 node

[cherviakovtaskworld]

Alright, I already read https://github.com/docker-library/mongo/blob/master/docker-entrypoint.sh and it appears that only valid case for rs initiation will be mine, no good way to make replica set working with other hardcoded arguments.

Can this be properly documented on docker mongodb page? As this was really confusing and time consuming to dig through.

[matthew-holder-revvity]

For anyone looking for an answer to this in a Compose stack, you can set the hostname on the container add extra_hosts for it to point to 127.0.0.1 and then reference it in the in entrypoint extra scripts you copy to the container. Just use $(which hostname) to get your container hostname.

[m-peko]

For anyone looking to the answer, here is a hack that made it working for me. I've made a healthcheck.sh script:

#!/bin/bash

# Initialize MongoDB server as a replica set and seed the database
# NOTE: This is a workaround since initializing the MongoDB server as
# a replica set cannot be done in the script placed in the
# /docker-entrypoint-initdb.d/ directory.
if ! mongosh --quiet --eval "rs.status().ok" &> /dev/null; then
    echo "Replica set not initialized, attempting to initiate..."
    if ! mongosh --quiet --eval "rs.initiate()"; then
        echo "Failed to initiate replica set"
        exit 1
    else
        echo "Replica set initiated successfully."
        mongosh /mongodb/seed.js || echo "Failed to seed the database"
    fi
else
    echo "Replica set is already initialized."
fi

# Final health check
if ! mongosh --quiet --eval "rs.status().ok" &> /dev/null; then
    echo "Health check failed: Replica set is not ok."
    exit 1
else
    echo "Health check passed: Replica set is ok."
fi

and calling that healthcheck.sh script from docker-compose.yml:

  mongodb:
    image: mongo:7
    container_name: mongodb
    ports:
      - "27017:27017"
    volumes:
      - mongodb_data:/data/db
      - ../db/mongodb/seed/seed.js:/mongodb/seed.js
      - ../db/mongodb/healthcheck.sh:/mongodb/healthcheck.sh
    networks:
      - atlas_net
    environment:
      - MONGO_INITDB_DATABASE=atlas
      - MONGO_REPLICA_SET_NAME=rs0
    command: --replSet rs0 --bind_ip_all --port 27017
    healthcheck:
      test: /mongodb/healthcheck.sh
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 20s
    restart: always