Cluster Running with 2 nodes and having difficulty to add 3rd node in the cluster

[DipindasThayyil]

As per the below url ,noticed cluster is running with 2 node and we tried with adding 3rd nodes into the cluster via " curl -X PUT "http://xxx.xxx.xxx.xxx/_node/_local/_nodes/node2@yyy.yyy.yyy.yyy" -d {} " however it not working.

Moreover , we have noticed difference in " adminPassword,cookieAuthSecret,erlangCookie" for the 3rd node.Therefore we have manaually updated these values and after that all the nodes were restarted and 3rd node were joined into cluster .

However after some time,again noticed cluster membership is in degraded state . Any suggestion from here ?

version: 3.3.3

https://score-db.cec.net/_membership

{
"all_nodes": [
"couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local"
],
"cluster_nodes": [
"couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local",
"couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local"
]
}

https://score-db.cec.net/_cluster_setup

{
"state": "cluster_finished"
}

Logs :

[notice] 2024-12-16T06:59:18.773373Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.520.0> -------- mem3_reshard finished reloading jobs
[info] 2024-12-16T06:59:18.796240Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.527.0> -------- Apache CouchDB has started. Time to relax.

[info] 2024-12-16T06:59:18.796317Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.527.0> -------- Apache CouchDB has started on http://any:5984/
[notice] 2024-12-16T06:59:18.863063Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.619.0> -------- All system databases exist.
[warning] 2024-12-16T06:59:18.866103Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.617.0> -------- 6 shards in cluster with only 1 copy on nodes that are currently up
[warning] 2024-12-16T06:59:18.868596Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local <0.617.0> -------- 6 shards in cluster with only 1 copy on nodes not in maintenance mode
[error] 2024-12-16T06:59:19.760792Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.688.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}

[error] 2024-12-16T06:59:19.760855Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.688.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-0.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}

[error] 2024-12-16T06:59:20.761868Z couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local emulator -------- Error in process <0.692.0> on node 'couchdb@couchdb-couchdb-2.couchdb-couchdb.apache-couchdb.svc.cluster.local' with exit value:
{{rexi_DOWN,{'couchdb@couchdb-couchdb-1.couchdb-couchdb.apache-couchdb.svc.cluster.local',noconnect}},[{mem3_rpc,rexi_call,3,[{file,"src/mem3_rpc.erl"},{line,384}]},{mem3_seeds,'-start_replication/1-fun-0-',1,[{file,"src/mem3_seeds.erl"},{line,107}]}]}

[janl]

can you double check that that new node is reachable via network from the others, that the required ports (see the docs) are all open and that the erlang cookie value is the same everywhere?