ERROR signal: illegal instruction

[mfizz1]

I am not sure what is happening here. I have successfully used gluetun before, now I seem to be getting into a hurdle:

========================================
========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================
Running version latest built on 2022-06-05T16:05:56.477Z (commit f5f65d5)
🔧 Need help? https://github.com/qdm12/gluetun/discussions/new
🐛 Bug? https://github.com/qdm12/gluetun/issues/new
✨ New feature? https://github.com/qdm12/gluetun/issues/new
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2022-06-05T22:30:28Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2
2022-06-05T22:30:28Z INFO [routing] local ethernet link found: eth0
2022-06-05T22:30:28Z INFO [routing] local ipnet found: 172.17.0.0/16
2022-06-05T22:30:28Z INFO [firewall] enabling...
2022-06-05T22:30:28Z INFO [firewall] enabled successfully
2022-06-05T22:30:29Z INFO [storage] creating /gluetun/servers.json with 11427 hardcoded servers
2022-06-05T22:30:30Z INFO Alpine version: 3.16.0
2022-06-05T22:30:30Z INFO OpenVPN 2.4 version: 2.4.12
2022-06-05T22:30:30Z ERROR signal: illegal instruction
2022-06-05T22:30:30Z INFO Shutdown successful

docker run -d \
  --name=gluetun \
  --cap-add=NET_ADMIN \
  -e OPENVPN_USER=p00 \
  -e OPENVPN_PASSWORD= 911 \
  --restart unless-stopped \
	qmcgaw/gluetun

At first I thought it was because it was the wrong architecture but I still cannot get it to work.

[qdm12]

Interesting, I have never seen this!
What's your CPU architecture and host OS?

[mfizz1]

CPU architecture is AARCH64 using libreelec (10.85). I tried arm/v7 and arm64 versions manually by referencing the digest, and I don't get this error, I get something about cannot route iptables. When I get back from work I will post the logs for these ones.

[qdm12]

Does qmcgaw/gluetun:v3.29.0 work? the latest image got the alpine 3.16 upgrade, maybe that's the cause since it's rather recent and possibly untested 🤔

[mfizz1]

Does qmcgaw/gluetun:v3.29.0 work? the latest image got the alpine 3.16 upgrade, maybe that's the cause since it's rather recent and possibly untested 🤔

I did try yesterday with v3, resulting in the same error. I will try with v3.29 and feedback

Question: at this line in the log, what are the instructions before / after it. Maybe I can narrow down the exact command causing this issue.

INFO OpenVPN 2.4 version: 2.4.12

[mfizz1]

Same issue with 3.29

[qdm12]

I did try yesterday with v3, resulting in the same error. I will try with v3.29 and feedback

v3 is the latest v3 so v3.29
Does it work with v3.28.2?

Question: at this line in the log, what are the instructions before / after it. Maybe I can narrow down the exact command causing this issue.

The problem most likely comes from Openvpn 2.5 since it crashes at printing its version

gluetun/cmd/gluetun/main.go

Line 246 in 5359257

{name: "OpenVPN 2.5", getVersion: ovpnConf.Version25},

I'll dig more tomorrow evening.

[mfizz1]

Unfortunately, I am still getting the same error on v3.28.2.

Thank you for the line, I had been searching yesterday trying to find it. Would have have thought it was this line causing the issue since the openvpn version information gets printed:

gluetun/cmd/gluetun/main.go

Line 247 in 5359257

{name: "Unbound", getVersion: dnsConf.Version},

[qdm12]

@mfizz1 there are two openvpn, 2.4.x and 2.5.x. I'm not 100% sure which one causes the crash, but the 2.4.x version gets printed so my guess is it's the 2.5.x version causing the issue.

Try

docker run -it --rm alpine:3.16
apk add openvpn
openvpn --version
exit

Does this work?

[mfizz1]

Nope! upon apk add openvpn it says Illegal instruction!

may have found the guy!

I been trying other things in there too, anything apk related is coming up illegal instruction.

[qdm12]

Hm strange, are you sure you're running the aarch64 version of alpine? Try docker pull qmcgaw/alpine:3.16 and compare the sha256 digest you get with the matching architecture https://hub.docker.com/layers/alpine/library/alpine/3.16.0/images/sha256-4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253?context=explore

Also what's the output of uname -a?

[mfizz1]

uname output : Linux LibreELEC 5.18.0 #1 SMP PREEMPT Sat May 28 12:27:54 UTC 2022 aarch64 GNU/Linux

docker pull qmcgaw/alpine:3.16 comes up with access denied / no image. But I did try the docker pull alpine:3.16

nonetheless getting some peculiar behavior. I am getting the following:

sha256:686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c which I cannot find anywhere on the alpine tags

[mfizz1]

So after continuing to struggle, I genuinely don't know which alpine docker pull is fetching. So I forced it to download

alpine:3.16@sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0

It downloaded but it has no tag (weird). I ran the same commands as what you have listed, (replacing alpine with the actual image id) and the commands followed through and output openvpn 2.5.6

[mfizz1]

Some more progress made by specifying arm64 digest

✨ New feature? https://github.com/qdm12/gluetun/issues/new
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2022-06-07T12:31:28Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2
2022-06-07T12:31:28Z INFO [routing] local ethernet link found: eth0
2022-06-07T12:31:28Z INFO [routing] local ipnet found: 172.17.0.0/16
2022-06-07T12:31:28Z INFO [firewall] enabling...
2022-06-07T12:31:28Z INFO [firewall] enabled successfully
2022-06-07T12:31:29Z INFO [storage] creating /gluetun/servers.json with 11559 hardcoded servers
2022-06-07T12:31:30Z INFO Alpine version: 3.16.0
2022-06-07T12:31:30Z INFO OpenVPN 2.4 version: 2.4.12
2022-06-07T12:31:30Z INFO OpenVPN 2.5 version: 2.5.6
2022-06-07T12:31:30Z INFO Unbound version: 1.15.0
2022-06-07T12:31:30Z INFO IPtables version: v1.8.8
2022-06-07T12:31:30Z INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: private internet access
|   |   └── Server selection settings:
|   |       ├── VPN type: openvpn
|   |       └── OpenVPN server selection settings:
|   |           ├── Protocol: UDP
|   |           └── Private Internet Access encryption preset: strong
|   └── OpenVPN settings:
|       ├── OpenVPN version: 2.5
|       ├── User: [set]
|       ├── Password: [set]
|       ├── Private Internet Access encryption preset: strong
|       ├── Tunnel IPv6: no
|       ├── Network interface: tun0
|       ├── Run OpenVPN as: root
|       └── Verbosity level: 1
├── DNS settings:
|   ├── DNS server address to use: 127.0.0.1
|   ├── Keep existing nameserver(s): no
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: yes
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:7f00:1/104
|               ├── ::ffff:a00:0/104
|               ├── ::ffff:a9fe:0/112
|               ├── ::ffff:ac10:0/108
|               └── ::ffff:c0a8:0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: INFO
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 1000
|   └── Process GID: 1000
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   └── IP file path: /tmp/gluetun/ip
└── Version settings:
    └── Enabled: yes
2022-06-07T12:31:30Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2
2022-06-07T12:31:30Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2
2022-06-07T12:31:30Z INFO [routing] deleting route for 0.0.0.0/0
2022-06-07T12:31:30Z ERROR [routing] cannot reverse routing changes: cannot remove routes for inbound traffic from default IP: cannot delete rule: cannot delete rule for default route interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2: cannot list rules: operation not supported
2022-06-07T12:31:30Z ERROR cannot setup routing: cannot add routes for inbound traffic from default IP: cannot add rule: cannot add rule for default route interface eth0, gateway 172.17.0.1 and assigned IP 172.17.0.2: cannot list rules: operation not supported
2022-06-07T12:31:30Z INFO Shutdown successful

[qdm12]

First of all, yes it's alpine:3.16 without qmcgaw 😄

I found 686d8c9dfa6f3ccfc8230bc3178d23f84eeaf7e457f36f271ab1acc53015037c somewhere on Google, but yeah it doesn't seem like the hash of alpine pushed 20 days ago, strange...

It downloaded but it has no tag (weird)

Yeah that's expected, as long as the hashes match is the same image anyway.

I ran the same commands as what you have listed, (replacing alpine with the actual image id) and the commands followed through and output

Nice! Although I have no idea why your Docker pulls images for another architecture, strange...

You can also use docker manifest inspect <image-name> to get information on the hash for each architecture, maybe that can show up something (or not).

Some more progress made by specifying arm64 digest

Nice 👍 Although operation not supported oh boy you must have some strange Kernel 😄

Try:

docker run -it --rm --cap-add=NET_ADMIN alpine:3.16 ip rule

For me it gives me

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

on both my Linux amd64 and Windows (Linux VM) amd64 machines. I guess that's the 'operation not supported' in your case? 🤔

[mfizz1]

I have managed to sort this out. It was my os issue. The speed however for me is an issue (openvpn). I only get about 10% of my capability. I know you are implementing a pia wireguard please if you can avoid using wg-quick to bring up the interface.

I believe on pia Foss GitHub, there is a user by the name of triffid that gets the interface up without using wg-quick. It will really help the people on legacy systems.

I eagerly await your version of it.

[qdm12]

It was my os issue

What was it if you don't mind sharing?

It will really help the people on legacy systems.

Yeah it does it with Go code only (kernel or userspace). The biggest problem with PIA (and a few others) is they require to exchange with their server to get wireguard keys, and gluetun is designed to not allow any communication until the vpn is up. I'll change that to be optionally disabled to allow such communication, but I have a bunch of more pressing issues to solve first 😅

[mfizz1]

What was it if you don't mind sharing?
I'll do one better and show you the commit that fixed it. It was missing CONFIG_IP_MULTIPLE_TABLES was missing in the kernel.

Commit

Yeah it does it with Go code only (kernel or userspace)

I have tried the userspace and it doesn't work, and I share the same experience as other users (granted people will complain more if it doesn't work as opposed to it working)

[qdm12]

Thanks I added this to the Wiki at https://github.com/qdm12/gluetun/wiki/Routing-errors

I have tried the userspace and it doesn't work, and I share the same experience as other users (granted people will complain more if it doesn't work as opposed to it working)

How did you try? Please create another issue for it I'd be curious to see what was failing! Thanks.

Closing this issue now since it's kind of resolved I guess.