CoreDNS provider ownership is broken with multiple A records of the same host

[ytsarev]

Encountered in context of k8gb-io/k8gb#38

The context on setup:

• external-dns with coredns provider and txt registry backend
• --txt-owner-id=ohmyglb ( practically it does not matter as the problem is encountered with default as well)
• source == "CRD" ( I believe it should be source agnostic issue but specifying just in case)

The issue itself:

• Source DNSEndpoint CRD

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
....
spec:
  endpoints:
  - dnsName: localtargets.app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.5
  - dnsName: app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.5

• Initially we observe the following in external-dns pod logs:

time="2020-02-10T16:59:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/3f31b771 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=ohmyglb,external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-02-10T16:59:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/2c604519 to Host=172.17.0.3, Text=, TTL=30"
time="2020-02-10T16:59:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/29a93fa8 to Host=172.17.0.5, Text=, TTL=30"
time="2020-02-10T17:00:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/3f31b771 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=ohmyglb,external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-02-10T17:00:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/3f31b771 to Host=172.17.0.3, Text=, TTL=30"
time="2020-02-10T17:00:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/3f31b771 to Host=172.17.0.5, Text=, TTL=30"

• Notice that Text= is empty on 2nd and 3rd A records in etcd
• During the consequent attempts external-dns "loses" the ownership to its own entries and we have an overall operation troubles

time="2020-02-10T17:01:53Z" level=debug msg="Skipping endpoint app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.3;172.17.0.5 [] because owner id does not match, found: \"\", required: \"ohmyglb\""
time="2020-02-10T17:01:53Z" level=debug msg="Skipping endpoint app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"ohmyglb\""
time="2020-02-10T17:01:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/7017c5bb to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=ohmyglb,external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-02-10T17:01:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/7017c5bb to Host=172.17.0.3, Text=, TTL=30"
time="2020-02-10T17:01:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/7017c5bb to Host=172.17.0.5, Text=, TTL=30"
time="2020-02-10T17:02:53Z" level=debug msg="Skipping endpoint app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.3;172.17.0.5 [] because owner id does not match, found: \"\", required: \"ohmyglb\""
time="2020-02-10T17:02:53Z" level=debug msg="Skipping endpoint app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"ohmyglb\""
time="2020-02-10T17:02:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/7017c5bb to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=ohmyglb,external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-02-10T17:02:53Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/7017c5bb to Host=172.17.0.3, Text=, TTL=30"

I did not find an obvious way to fix it in the code, but here are things I noticed that might help to track it

• https://github.com/kubernetes-sigs/external-dns/blob/master/registry/txt.go#L86
• Similar reliance on Targets[0] in https://github.com/kubernetes-sigs/external-dns/blob/master/provider/coredns.go#L354
• Looks like etcd backend / skydns does not have separate entity for TXT record and it has problems with the current design

Currently I will workaround it with disabling ownership with noop backend as in my scenario it is 'local' coredns, but this issue might be critical for other kinds of setup

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[ytsarev]

it is something to retest with the fix of #1475 . Will take care

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[ytsarev]

#1475 appears to also fix this issue, no lost of txt ownership after the fix. Closing.