dogstatsd metrics export not working

[DanTulovsky]

Describe the bug
I can't seem to get dogstatsd metrics working in ambassador edge stack.

To Reproduce
I've setup:

  # https://docs.datadoghq.com/integrations/ambassador/
  STATSD_ENABLED: true
  DOGSTATSD: true

envRaw:
  - name: STATSD_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP

The logs on startup have:

time="2021-03-16T14:11:56Z" level=warning msg="statsd is not in use"

which looks related. I don't see any other statsd related logs.

Expected behavior
statsd metrics show up.

Versions (please complete the following information):

• Ambassador: 1.11.2
• Kubernetes environment: GKE
• Version: 1.19.7-gke.1500

Additional context

I can login to the ambassador pod and confirm:

bash-5.0$ env |grep STATSD
DOGSTATSD=true
STATSD_HOST=10.128.0.29
STATSD_ENABLED=true

I can also manually set a metric:

echo -n "custom10.metric.name:98|g"|nc -u -w1 $STATSD_HOST 8125

And this metric shows up. But I still see no envoy metrics. Is the "statsd is not in use" message in the logs a red herring or actually indicative of a problem? On the ambassador debug page I do see: StatsD support enabled.

Thank you
Dan

[DanTulovsky]

Not sure if containernetworking/plugins#553 is possibly related...

[DanTulovsky]

Although running the cleanup command doesn't seem to help.. or restarting ambassador:

gke-cluster0-pool0-6e5903aa-7fv2 /proc/net # conntrack -D -p udp --dport 8125
udp      17 26 src=10.0.3.32 dst=10.128.0.29 sport=58746 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=58746 mark=0 use=1
udp      17 29 src=10.0.3.32 dst=10.128.0.29 sport=50970 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=50970 mark=0 use=1
udp      17 29 src=10.0.3.32 dst=10.128.0.29 sport=50135 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=50135 mark=0 use=1
conntrack v1.4.4 (conntrack-tools): 3 flow entries have been deleted.
gke-cluster0-pool0-6e5903aa-7fv2 /proc/net # cat nf_conntrack |grep 8125
ipv4     2 udp      17 28 src=10.0.3.32 dst=10.128.0.29 sport=58746 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=58746 mark=0 zone=0 use=2
ipv4     2 udp      17 29 src=10.0.3.32 dst=10.128.0.29 sport=50970 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=50970 mark=0 zone=0 use=2
ipv4     2 udp      17 28 src=10.0.3.32 dst=10.128.0.29 sport=50135 dport=8125 [UNREPLIED] src=10.0.3.30 dst=10.0.3.32 sport=8125 dport=50135 mark=0 zone=0 use=2
gke-cluster0-pool0-6e5903aa-7fv2 /proc/net # 

[DanTulovsky]

Alright, it is working. The metrics names are different. envoy.* instead of envoy_*.

sorry for the noise.