dnsmasq Pod locks on infra-ops server

[Zarquan]

After time dnsmasq becomes unresponsive on the infra-ops server.

Direct requests to the DNS service timeout:

dig '@infra-ops.aglais.uk' 'zeppelin.gaia-prod.aglais.uk'
; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> @infra-ops.aglais.uk zeppelin.gaia-prod.aglais.uk
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

Booting dnsmasq with a SIGHUP signal doesn't solve the problem.

podman kill --signal SIGHUP dnsmasq

Restarting the Pod on the infra-ops server makes it go away.

podman stop dnsmasq
sleep 5
podman start dnsmasq

Direct requests to the DNS service work fine:

dig '@infra-ops.aglais.uk' 'zeppelin.gaia-prod.aglais.uk'

; <<>> DiG 9.11.28-RedHat-9.11.28-1.fc32 <<>> @infra-ops.aglais.uk zeppelin.gaia-prod.aglais.uk
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 887
;; flags: qr aa rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 1

;; QUESTION SECTION:
;zeppelin.gaia-prod.aglais.uk.	IN	A

;; ANSWER SECTION:
zeppelin.gaia-prod.aglais.uk. 300 IN	A	128.232.227.188

;; AUTHORITY SECTION:
gaia-prod.aglais.uk.	300	IN	NS	infra-ops.aglais.uk.

;; Query time: 14 msec
;; SERVER: 46.101.32.198#53(46.101.32.198)
;; WHEN: Tue Jun 08 12:58:16 BST 2021
;; MSG SIZE  rcvd: 125

[Zarquan]

Happened before
https://github.com/wfau/aglais/blob/master/notes/zrq/20210225-01-infra-ops.txt

Resorted to the same non-fix as before:
https://github.com/wfau/aglais/blob/161ee5d041da7b9ad46ea8f742478e4ba0ea15e0/notes/zrq/20210225-01-infra-ops.txt#L711-L717

[Zarquan]

and again

[Zarquan]

Happened again ~ 7th July

[Zarquan]

Happened again Friday 23rd July

[Zarquan]

Happened again, between 7th and 9th Aug

[Zarquan]

Happened again ~17th Aug

[Zarquan]

Happened again ~11th Sept

[Zarquan]

Happened again ~14th Sept

[Zarquan]

Happened ~ 20th Sept