Eturnal not starting on Rocky Linux 8.5

[sando38]

Dear Holger,

discovered an issue with Rocky Linux 8.5 (CentOS 8). It is basically a fresh install.

I installed eturnal both ways with dnf package manager or binary packages without any errors or unforeseeable things with the following result:

[root@eturnal-001 ~]# /opt/eturnal/bin/eturnalctl sessions
Node is not running!
[root@eturnal-001 ~]# /opt/eturnal/bin/eturnalctl version
1.7.0 (not running)
[root@eturnal-001 ~]# /opt/eturnal/bin/eturnalctl daemon
^C
Sitzung beendet, Shell wird abgewürgt … … abgewürgt.

The command above stucks forever, so I killed it...

I previously had CentOS 7 and it runs smoothly without any distortions ( I only used binary install, however).

Here are the outputs from log-files (erlang.log.1) which shows a similar issue as if I would use "eturnalctl foreground":

=====
===== LOGGING STARTED Tue Dec 28 17:18:15 CET 2021
=====
Exec: /opt/eturnal/erts-12.2/bin/erlexec -boot /opt/eturnal/releases/1.7.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/eturnal/lib -config /opt/eturnal/releases/1.7.0/sys.config -args_file /opt/eturnal/releases/1.7.0/vm.args -- console  --relx-disable-hooks
Root: /opt/eturnal
/opt/eturnal
Protocol 'inet_tcp': the name eturnal@127.0.0.1 seems to be in use by another Erlang node

I am unable to investigate further at least "this year" xD ... What makes me wonder is that eturnal is being used already by another Erlang node.. I could not discover any process belonging to eturnal which could block.

Thanks and wishing a great evening!

[weiss]

The Erlang VM starts up a small helper daemon called epmd, which keeps track of the running Erlang/OTP applications (I'm hoping to get rid of this in the future, BTW). What you describe sounds like some eturnal process got hard-killed and wasn't able to deregister itself from epmd, and for some reason the epmd process itself has not been killed (which would usually happen if eturnal was started and stopped via systemd).

Do you get any output from /opt/eturnal/erts-*/bin/epmd -names and/or ps aux | egrep 'eturnal|epmd|beam'? If so, assuming there's no other Erlang/OTP daemons running on that system, you can just kill those processes, and try (re)starting eturnal.

[sando38]

Thanks for the quick reply and side information. I did kill the process, but relaunching brings the same issue:

[root@eturnal-001 ~]# ps aux | egrep 'eturnal|epmd|beam'
eturnal   194492  0.4  0.9 2369340 73124 ?       Ssl  22:39   0:00 /opt/eturnal/bin/eturnal -Bd -- -root /opt/eturnal -progname opt/eturnal/bin/eturnal -- -home /opt/eturnal -- -noshell -noinput -boot /opt/eturnal/releases/1.7.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/eturnal/lib -config /opt/eturnal/releases/1.7.0/sys.config -name eturnal@127.0.0.1 -- -- foreground --
eturnal   194533  0.0  0.0  12136   100 ?        S    22:39   0:00 /opt/eturnal/erts-12.2/bin/epmd -daemon
eturnal   194537  0.0  0.0   4368   856 ?        Ss   22:39   0:00 erl_child_setup 50000
eturnal   194627  0.0  0.0  12132  1156 ?        Ss   22:39   0:00 inet_gethost 4
eturnal   194628  0.0  0.0  39240  2600 ?        S    22:39   0:00 inet_gethost 4
root      200088  0.0  0.0 221928  1160 pts/0    R+   22:41   0:00 grep -E --color=auto eturnal|epmd|beam
[root@eturnal-001 ~]# pidof eturnal
194492
[root@eturnal-001 ~]# kill 194492
[root@eturnal-001 ~]# pidof eturnal
[root@eturnal-001 ~]# ps aux | egrep 'eturnal|epmd|beam'
root      200529  0.0  0.0 221928  1136 pts/0    S+   22:42   0:00 grep -E --color=auto eturnal|epmd|beam
[root@eturnal-001 ~]# systemctl enable --now eturnal
Synchronizing state of eturnal.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable eturnal
Created symlink /etc/systemd/system/multi-user.target.wants/eturnal.service → /etc/systemd/system/eturnal.service.
[root@eturnal-001 ~]# ps aux | egrep 'eturnal|epmd|beam'
eturnal   201012 14.7  0.9 2368572 74140 ?       Ssl  22:42   0:00 /opt/eturnal/bin/eturnal -Bd -- -root /opt/eturnal -progname opt/eturnal/bin/eturnal -- -home /opt/eturnal -- -noshell -noinput -boot /opt/eturnal/releases/1.7.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/eturnal/lib -config /opt/eturnal/releases/1.7.0/sys.config -name eturnal@127.0.0.1 -- -- foreground --
eturnal   201054  0.0  0.0  12136   104 ?        S    22:42   0:00 /opt/eturnal/erts-12.2/bin/epmd -daemon
eturnal   201058  0.0  0.0   4368   876 ?        Ss   22:42   0:00 erl_child_setup 50000
eturnal   201077  0.0  0.0  12132  1152 ?        Ss   22:42   0:00 inet_gethost 4
eturnal   201078  0.0  0.0  39240  2600 ?        S    22:42   0:00 inet_gethost 4
root      201186  0.0  0.0 221928  1076 pts/0    R+   22:42   0:00 grep -E --color=auto eturnal|epmd|beam
[root@eturnal-001 ~]# eturnalctl status
Node is not running!
[root@eturnal-001 ~]# eturnalctl daemon
^C
Sitzung beendet, Shell wird abgewürgt … … abgewürgt.
[root@eturnal-001 ~]# eturnalctl foreground
Exec: /opt/eturnal/erts-12.2/bin/erlexec -noinput +Bd -boot /opt/eturnal/releases/1.7.0/start -mode embedded -boot_var SYSTEM_LIB_DIR /opt/eturnal/lib -config /opt/eturnal/releases/1.7.0/sys.config -args_file /opt/eturnal/releases/1.7.0/vm.args -- foreground
Root: /opt/eturnal
/opt/eturnal
Protocol 'inet_tcp': the name eturnal@127.0.0.1 seems to be in use by another Erlang node

[weiss]

Thanks. I guess killing the hanging daemon startup with Ctrl+C yields the dangling epmd instance, which results in that last error message. But the real problem is somewhere else.

It seems epmd is started. The next step would be the eturnal process telling epmd "hey, my name is eturnal@127.0.0.1, I'm reachable on port 4444". If you then call e.g. eturnalctl status, eturnalctl will contact epmd (on port 4369), ask for eturnal's port number, and then query eturnal on that port (e.g., 4444) using the ~eturnal/.erlang.cookie file for authentication. One of those steps is failing, question is which. We gotta track this down step by step:

• After starting eturnal with systemd, do you see any output from /opt/eturnal/erts-*/bin/epmd -names?

• Could you then systemctl stop eturnal, start it with eturnalctl console (which throws you into an Erlang shell), and try eturnalctl status from another terminal? Do you see any error message in the console shell? (You can then stop the console instance by typing q(). into the shell, including that trailing ..)

[sando38]

I will see if I find time to test tonight! Thank you.

[weiss]

Nevermind, I've installed Rocky in VirtualBox and was able to reproduce the problem. It's a glibc compatibility issue, dnf install libnsl solves it. I'll see how I can avoid that dependency in the binary packages.

[sando38]

Okay, thanks for the feedback.
Installed libnsl and it works as expected.
Thank you for your support!
Wishing a great evening!

[weiss]

Thanks for reporting the issue! Should be fixed, now.

Just in case you're interested, I've published packages which don't depend on libnsl on https://eturnal.net/tmp/.

[sando38]

Thanks for reporting the issue! Should be fixed, now.

Just in case you're interested, I've published packages which don't depend on libnsl on https://eturnal.net/tmp/.

Thanks. Used it and works also without libnsl being installed on the machine.
Thanks again for your efforts. Wishing you a great start in 2022.