No such file or directory: '/home/postgres/pgdata/pgroot/data/postgresql.conf' -> '/home/postgres/pgdata/pgroot/data/postgresql.base.conf'

[jonathon2nd]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using? registry.opensource.zalan.do/acid/postgres-operator:v1.8.2
• Where do you run it - cloud or metal? Kubernetes or OpenShift? On-prem, Rancher, RKE2, Kubernetes Version: v1.24.4
• Are you running Postgres Operator in production? yes
• Type of issue? [Bug report, question, feature request, etc.] Bug

Attempting to setup standby cluster in preperation to migrate away from bitnami/postgresql-ha on another cluster. Using same psql version, and have user+passwords setup correctly.

Replication of data seems to occur, disk space looks about the same. But then the starting of the node fails

FileNotFoundError: [Errno 2] No such file or directory: '/home/postgres/pgdata/pgroot/data/postgresql.conf' -> '/home/postgres/pgdata/pgroot/data/postgresql.base.conf'

I think pgpool was interfering and causing my initial troubles, hence the edits and hidden comments

[jonathon2nd]

OK so pgpool was being a pain, so connecting directly to a replica on the source cluster.

Got all the way through, then the pod that the operator made failed.

2022-12-30 19:21:12,108 INFO: not healthy enough for leader race
--
Fri, Dec 30 2022 11:21:12 am | 2022-12-30 19:21:12,108 INFO: bootstrap_standby_leader in progress
Fri, Dec 30 2022 11:21:16 am | 2022-12-30 19:21:16,024 INFO: replica has been created using basebackup_fast_xlog
Fri, Dec 30 2022 11:21:16 am | 2022-12-30 19:21:16,027 INFO: bootstrapped clone from remote master postgresql://<IP>:5432
Fri, Dec 30 2022 11:21:16 am | 2022-12-30 19:21:16,036 ERROR: Exception during execution of long running task bootstrap_standby_leader
Fri, Dec 30 2022 11:21:16 am | Traceback (most recent call last):
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/async_executor.py", line 97, in run
Fri, Dec 30 2022 11:21:16 am | wakeup = func(*args) if args else func()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 288, in bootstrap_standby_leader
Fri, Dec 30 2022 11:21:16 am | result = self.clone(clone_source, msg)
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 244, in clone
Fri, Dec 30 2022 11:21:16 am | return self.state_handler.follow(node_to_follow) is not False
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/postgresql/__init__.py", line 903, in follow
Fri, Dec 30 2022 11:21:16 am | ret = self.start(timeout=timeout, block_callbacks=change_role, role=role) or None
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/postgresql/__init__.py", line 549, in start
Fri, Dec 30 2022 11:21:16 am | self.config.write_postgresql_conf(configuration)
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/postgresql/config.py", line 390, in write_postgresql_conf
Fri, Dec 30 2022 11:21:16 am | os.rename(self._postgresql_conf, self._postgresql_base_conf)
Fri, Dec 30 2022 11:21:16 am | FileNotFoundError: [Errno 2] No such file or directory: '/home/postgres/pgdata/pgroot/data/postgresql.conf' -> '/home/postgres/pgdata/pgroot/data/postgresql.base.conf'
Fri, Dec 30 2022 11:21:16 am | /var/run/postgresql:5432 - no response
Fri, Dec 30 2022 11:21:16 am | 2022-12-30 19:21:16,056 INFO: removing initialize key after failed attempt to bootstrap the cluster
Fri, Dec 30 2022 11:21:16 am | 2022-12-30 19:21:16,112 INFO: renaming data directory to /home/postgres/pgdata/pgroot/data_2022-12-30-19-21-16
Fri, Dec 30 2022 11:21:16 am | Traceback (most recent call last):
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/bin/patroni", line 11, in <module>
Fri, Dec 30 2022 11:21:16 am | sys.exit(main())
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/__main__.py", line 143, in main
Fri, Dec 30 2022 11:21:16 am | return patroni_main()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/__main__.py", line 135, in patroni_main
Fri, Dec 30 2022 11:21:16 am | abstract_main(Patroni, schema)
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 100, in abstract_main
Fri, Dec 30 2022 11:21:16 am | controller.run()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/__main__.py", line 105, in run
Fri, Dec 30 2022 11:21:16 am | super(Patroni, self).run()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/daemon.py", line 59, in run
Fri, Dec 30 2022 11:21:16 am | self._run_cycle()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/__main__.py", line 108, in _run_cycle
Fri, Dec 30 2022 11:21:16 am | logger.info(self.ha.run_cycle())
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1514, in run_cycle
Fri, Dec 30 2022 11:21:16 am | info = self._run_cycle()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1388, in _run_cycle
Fri, Dec 30 2022 11:21:16 am | return self.post_bootstrap()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1280, in post_bootstrap
Fri, Dec 30 2022 11:21:16 am | self.cancel_initialization()
Fri, Dec 30 2022 11:21:16 am | File "/usr/local/lib/python3.6/dist-packages/patroni/ha.py", line 1273, in cancel_initialization
Fri, Dec 30 2022 11:21:16 am | raise PatroniFatalException('Failed to bootstrap cluster')
Fri, Dec 30 2022 11:21:16 am | patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
Fri, Dec 30 2022 11:21:16 am | /etc/runit/runsvdir/default/patroni: finished with code=1 signal=0
Fri, Dec 30 2022 11:21:16 am | /etc/runit/runsvdir/default/patroni: sleeping 30 seconds
Fri, Dec 30 2022 11:21:46 am | 2022-12-30 19:21:46,662 WARNING: Kubernetes RBAC doesn't allow GET access to the 'kubernetes' endpoint in the 'default' namespace. Disabling 'bypass_api_service'.
Fri, Dec 30 2022 11:21:46 am | 2022-12-30 19:21:46,690 INFO: No PostgreSQL configuration items changed, nothing to reload.
Fri, Dec 30 2022 11:21:46 am | 2022-12-30 19:21:46,693 INFO: Lock owner: None; I am acid-labs-0
Fri, Dec 30 2022 11:21:46 am | 2022-12-30 19:21:46,884 INFO: trying to bootstrap a new standby leader
Fri, Dec 30 2022 11:21:57 am | 2022-12-30 19:21:57,194 INFO: Lock owner: None; I am acid-labs-0
Fri, Dec 30 2022 11:21:57 am | 2022-12-30 19:21:57,194 INFO: not healthy enough for leader race
Fri, Dec 30 2022 11:21:57 am | 2022-12-30 19:21:57,265 INFO: bootstrap_standby_leader in progress
Fri, Dec 30 2022 11:22:07 am | 2022-12-30 19:22:07,193 INFO: Lock owner: None; I am acid-labs-0

no errors on source cluster, file is just missing?

root@acid-labs-0:/home/postgres# ls /home/postgres/pgdata/pgroot/data/
PG_VERSION            backup_label.old      current_logfiles      pg_commit_ts/         pg_ident.conf         pg_multixact/         pg_replslot/          pg_stat/              pg_subtrans/          pg_twophase/          pg_xact/              standby.signal        
backup_label          base/                 global/               pg_dynshmem/          pg_logical/           pg_notify/            pg_snapshots/         pg_stat_tmp/          pg_tblspc/            pg_wal/               postgresql.auto.conf  

[jonathon2nd]

On and off I have not been able to get this working. I have tried replication from the replica and the master, no different. Same error of no such file

[freedbka]

I have the same problem when I try to make a replica from ordinary postgres on a virtual machine.

[MatthiasLohr]

Any solution for this? Currently struggling with the same problem.

[viceice]

Seeing same issue while migrating from bitnami/postgresql chart. 😕

I found a workaround by simply doing touch postgresql.conf in the bitnami data directory.