Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failed DNS response with EDNS #192

Closed
nicolasbock opened this issue May 19, 2023 · 7 comments
Closed

Failed DNS response with EDNS #192

nicolasbock opened this issue May 19, 2023 · 7 comments

Comments

@nicolasbock
Copy link

In a Yoga OpenStack deployment using ovn-22.03.0 I get failed DNS queries from a VM:

ubuntu@vm-1:~$ resolvectl flush-caches ; dig @10.5.1.118 vm-1.example.com
;; Warning: Message parser reports malformed message packet.

; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> @10.5.1.118 vm-1.example.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14166
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; WARNING: Message has 32 extra bytes at end

;; QUESTION SECTION:
;vm-1.example.com.              IN      A

;; ANSWER SECTION:
.                       0       CLASS1232 OPT   10 8 HAVchF7gf6g=

;; Query time: 3 msec
;; SERVER: 10.5.1.118#53(10.5.1.118) (UDP)
;; WHEN: Fri May 19 17:13:45 UTC 2023
;; MSG SIZE  rcvd: 89

When I disable EDNS I receive a valid response:

ubuntu@vm-1:~$ resolvectl flush-caches ; dig +noedns @10.5.1.118 vm-1.example.com
                                               
; <<>> DiG 9.18.1-1ubuntu1.3-Ubuntu <<>> +noedns @10.5.1.118 vm-1.example.com
; (1 server found)                             
;; global options: +cmd
;; Got answer:                                 
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 15204
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:                           
;vm-1.example.com.              IN      A

;; ANSWER SECTION:                             
vm-1.example.com.       3600    IN      A       10.1.0.40

;; Query time: 3 msec                          
;; SERVER: 10.5.1.118#53(10.5.1.118) (UDP)
;; WHEN: Fri May 19 17:13:41 UTC 2023
;; MSG SIZE  rcvd: 66

I can provide captured packets or other relevant information that might help.
@ar406
Copy link

ar406 commented May 20, 2023

Can you provide packet captures for both cases and the MTU configured on your Ubuntu machine (ip link show dev )?

@nicolasbock
Copy link
Author 

ubuntu@vm-1:~$ ip link show dev ens2 
2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:43:6b:18 brd ff:ff:ff:ff:ff:ff
    altname enp0s2

@nicolasbock
Copy link
Author

The two packet captures are at

https://gist.github.com/nicolasbock/c66f7d7e581a69b3041fe16f99d3ac0a

@nicolasbock
Copy link
Author

I'll add the packet captures inline here. There are short after all:

begin-base64 664 bad-DNS-packet.pcap
1MOyoQIABAAAAAAAAAAAAAAABAABAAAAKs9nZFTmBwBjAAAAYwAAAPoWPimA
vvoWPkNrGAgARQAAVbUYAABAEa/cCgEAKAoFAXa1EQA1AEEV9ushASAAAQAA
AAAAAQR2bS0xB2V4YW1wbGUDY29tAAABAAEAACkE0AAAAAAADAAKAAjdEE1D
OeFJbyrPZ2TG7AcAgwAAAIMAAAD6Fj5Daxj6Fj4pgL4IAEUAAHW1GAAAQBGv
vAoFAXYKAQAoADW1EQBhAADrIYEgAAEAAQAAAAAEdm0tMQdleGFtcGxlA2Nv
bQAAAQABAAApBNAAAAAAAAwACgAI3RBNQznhSW8Edm0tMQdleGFtcGxlA2Nv
bQAAAQABAAAOEAAECgEAKA==
====

begin-base64 664 good-DNS-packet.pcap
1MOyoQIABAAAAAAAAAAAAAAABAABAAAAR89nZDMeAABMAAAATAAAAPoWPimA
vvoWPkNrGAgARQAAPpHMAABAEdM/CgEAKAoFAXblrgA1ACoV31odASAAAQAA
AAAAAAR2bS0xB2V4YW1wbGUDY29tAAABAAFHz2dkrykAAGwAAABsAAAA+hY+
Q2sY+hY+KYC+CABFAABekcwAAEAR0x8KBQF2CgEAKAA15a4ASgAAWh2BIAAB
AAEAAAAABHZtLTEHZXhhbXBsZQNjb20AAAEAAQR2bS0xB2V4YW1wbGUDY29t
AAABAAEAAA4QAAQKAQAo
====

ovsrobot pushed a commit to ovsrobot/ovn that referenced this issue May 22, 2023
@brianphaley @ovsrobot
controller: Ignore DNS queries with RRs
b3ba335 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply sometimes results in mangled responses
that clients do not understand.

Instead, just return early when one is present, which
should trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

Closes issue ovn-org#192
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: 0-day Robot <robot@bytheb.org>
@brianphaley
Copy link
Contributor

I would assign this to myself if I could, but at least dropping a comment that I pushed that patch to the ML for review.

brianphaley added a commit to brianphaley/ovn that referenced this issue May 25, 2023
@brianphaley
Ignore DNS queries with RRs
d560558 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply sometimes results in mangled responses
that clients do not understand.

Instead, just return early when one is present, which
should trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

Closes issue ovn-org#192
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
ovsrobot pushed a commit to ovsrobot/ovn that referenced this issue May 25, 2023
@brianphaley @ovsrobot
controller: Ignore DNS queries with RRs
346bf53 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Closes issue ovn-org#192
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: 0-day Robot <robot@bytheb.org>
ovsrobot pushed a commit to ovsrobot/ovn that referenced this issue May 26, 2023
@brianphaley @ovsrobot
controller: Ignore DNS queries with RRs
5de353e 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: 0-day Robot <robot@bytheb.org>
@brianphaley
Copy link
Contributor

Just an FYI a fix for the job timeouts was proposed at https://mail.openvswitch.org/pipermail/ovs-dev/2023-May/404985.html - until that is merged don't think anything else will.

dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
4b10571 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
4334ce3 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
529ea69 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
0b0dc4a 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
4c19dcf 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
5926234 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
e61d06b 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
dceara pushed a commit to dceara/ovn that referenced this issue May 30, 2023
@brianphaley @dceara
controller: Ignore DNS queries with RRs
6ebbc70 
DNS queries with optional records (RRs), for example, with
cookies for EDNS, are not supported by the OVN resolver.
Trying to reply will result in mangled responses that
clients do not understand - the ANSWER section will
contain an incorrect option.

Instead, just return early when one is present, which
will trigger a negative response and cause clients to
go to the upstream forwarder, hopefully resulting in a
successful query.

In our testing, the resolver only retries if the
response is correctly formatted, which now happens
with this change.

Reported-at: ovn-org#192
Reported-by: Nicolas Bock <nicolasbock@gmail.com>
Signed-off-by: Brian Haley <haleyb.dev@gmail.com>
Signed-off-by: Dumitru Ceara <dceara@redhat.com>
(cherry picked from commit 4b10571)
@dceara
Copy link
Collaborator

dceara commented May 30, 2023

Fixed by 4b10571, thanks!

@dceara dceara closed this as completed May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@nicolasbock @brianphaley @dceara @ar406