Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update implementation of Basic02 #1197

Merged
merged 4 commits into from
Apr 5, 2023
Merged

Conversation

tgreenx
Copy link
Contributor

@tgreenx tgreenx commented Feb 22, 2023

Purpose

This PR proposes an update implementation of Basic02 to latest specification (zonemaster/zonemaster#1085 and zonemaster/zonemaster#1145).

Context

Fixes #1193, #881

Changes

Update implementation of Basic02:

  • Logic
  • Message tags

Update unitary tests:

  • Some are entirely new
  • Modernized utility functions

How to test this PR

Unitary tests should pass.

Also (without an IPv6 connectivity):

$ zonemaster-cli --test basic/basic02 zonemaster.net --level INFO --show-testcase --raw --no-ipv6
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.2
   6.09 INFO      BASIC02        B02_AUTH_RESPONSE_SOA   domain="zonemaster.net"; ns_list=ns2.nic.fr/192.93.0.4;nsa.dnsnode.net/194.58.192.46;nsp.dnsnode.net/194.58.198.32;nsu.dnsnode.net/185.42.137.98

And (from #881):

$ zonemaster-cli --test basic/basic02 e.consistency05.exempelvis.se --level INFO --show-testcase --raw --no-ipv6
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.2
   1.37 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="e.consistency05.exempelvis.se"
   1.37 ERROR     BASIC02        B02_UNEXPECTED_RCODE   ns=ns2.e.consistency05.exempelvis.se/194.18.226.122; rcode=NXDOMAIN
   1.37 ERROR     BASIC02        B02_UNEXPECTED_RCODE   ns=ns1.e.consistency05.exempelvis.se/46.21.97.97; rcode=REFUSED

@tgreenx tgreenx added A-TestCase Area: Test case specification or implementation of test case V-Minor Versioning: The change gives an update of minor in version. labels Feb 22, 2023
@tgreenx tgreenx added this to the v2023.1 milestone Feb 22, 2023
@tgreenx tgreenx linked an issue Feb 22, 2023 that may be closed by this pull request
@matsduf
Copy link
Contributor

matsduf commented Feb 23, 2023

t/util.t .................. 1/? unable to parse RR string at /usr/local/lib/perl5/site_perl/Net/DNS/ZoneFile.pm line 534.
 at /usr/home/matsd/.cpanm/work/1677164454.51730/Zonemaster-Engine-v4.6.1/blib/lib/Zonemaster/Engine/Util.pm line 179.

@tgreenx
Copy link
Contributor Author

tgreenx commented Feb 23, 2023

t/util.t .................. 1/? unable to parse RR string at /usr/local/lib/perl5/site_perl/Net/DNS/ZoneFile.pm line 534.
 at /usr/home/matsd/.cpanm/work/1677164454.51730/Zonemaster-Engine-v4.6.1/blib/lib/Zonemaster/Engine/Util.pm line 179.

This error is also present on master and develop branches, actually.

@matsduf
Copy link
Contributor

matsduf commented Feb 23, 2023

I tested the name lame.dufberg.se that is a lame delegation. The expected message is B02_NO_WORKING_NS and B02_NS_NO_RESPONSE but that is not what we get.

# zonemaster-cli --test basic/basic02 lame.dufberg.se --level INFO --show-testcase --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
  20.86 CRITICAL  BASIC02        B02_NO_DELEGATION   domain="lame.dufberg.se"

@matsduf
Copy link
Contributor

matsduf commented Feb 23, 2023

This error is also present on master and develop branches, actually.

You are right. That is a false error message. It should be hidden.

Comment on lines 417 to 422
# This is not a realistical conditional check considering the current implementation of Engine::Nameserver.
# Any Engine::Nameserver object created will (must) have an IP address. So it is here as a placeholder.
if ( not $ns->address ) {
$ns_cant_resolve{$ns->name} = 1;
next;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That limitation is really bad. That means that we cannot see when a domain is delegated to a name server name that cannot be resolved into an address.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is my understanding as well. I will see what I can do about it.

my %ns_cant_resolve;
my %ns_no_response;
my %unexpected_rcode;

my @ns = @{ Zonemaster::Engine::TestMethods->method4( $zone ) };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does metod4 return an empty array when the delegation is completely lame?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is currently the case.

$ git diff
diff --git a/lib/Zonemaster/Engine/Test/Basic.pm b/lib/Zonemaster/Engine/Test/Basic.pm
index 714b7176..18e45f25 100644
--- a/lib/Zonemaster/Engine/Test/Basic.pm
+++ b/lib/Zonemaster/Engine/Test/Basic.pm
@@ -395,6 +395,18 @@ sub basic02 {
     my %ns_no_response;
     my %unexpected_rcode;

+    say $zone->name;
+    say $zone->parent->name;
+    say "glue names";
+    say @{$zone->glue_names};
+    say "glue";
+    say @{$zone->glue};
+    say "ns names";
+    say @{$zone->ns_names};
+    say "ns";
+    say @{$zone->ns};
+    die;
+
     my @ns = @{ Zonemaster::Engine::TestMethods->method4( $zone ) };

     if ( not scalar @ns ) {

$ zonemaster-cli --show-testcase --test basic/basic02 lame.dufberg.se --level INFO --no-ipv6 --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
lame.dufberg.se
dufberg.se
glue names
ns1.lame.dufberg.sens2.lame.dufberg.se
glue

ns names

ns

  21.25 CRITICAL  UNSPECIFIED    MODULE_ERROR   module=Basic; msg=Died at /home/tgreen/zonemaster/zonemaster-engine/lib/Zonemaster/Engine/Test/Basic.pm line 408.
  
$ zonemaster-cli --show-testcase --test basic/basic02 zonemaster.fr --level INFO --no-ipv6 --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
zonemaster.fr
fr
glue names
ns1.nic.frns3.nic.fr
glue
ns1.nic.fr/192.134.4.1ns1.nic.fr/2001:67c:2218:2::4:1ns3.nic.fr/192.134.0.49ns3.nic.fr/2001:660:3006:1::1:1
ns names
ns1.nic.frns3.nic.fr
ns
ns1.nic.fr/192.134.4.1ns1.nic.fr/2001:67c:2218:2::4:1ns3.nic.fr/192.134.0.49ns3.nic.fr/2001:660:3006:1::1:1
   0.51 CRITICAL  UNSPECIFIED    MODULE_ERROR   module=Basic; msg=Died at /home/tgreen/zonemaster/zonemaster-engine/lib/Zonemaster/Engine/Test/Basic.pm line 408.

So "glue_names" (aka. Method2) is non-empty and has the right name server names for the zone. However, "glue" (i.e, the creation of name server objects, aka. Method4) is empty. The function that populates the "glue" is a special array object Engine::NSArray, filled using Engine::NSArray::_load_name(), which itself calls Engine::Recursor::get_addresses_for(), and in turn, Engine::Recursor::_recurse() which skips unresponsive name servers.

I'll investigate further though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that is currently the case.

That seems to be the wrong method.

So "glue_names" (aka. Method2) is non-empty and has the right name server names for the zone. However, "glue" (i.e, the creation of name server objects, aka. Method4) is empty.

So why not use Method2 to get the delegation (NS) and Method4 to get the glue address records (does the implemented Method4 include non-glue address records)? The specification says:

  1. Retrieve the NS records for *Child Zone* using [Method 2] and the IP
     addresses ([glue records][glue record]) for any [in-bailiwick] name
     servers using [Method 4].
  2. Retrieve the IP addresses for any [out-of-bailiwick] name servers
     using recursive lookup for address records (both IPv4 and IPv6) for
     that name and add resolved addresses, if any, to the set.

Mats

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the implemented Method4 include non-glue address records

Digging into the implementation, I would answer yes (however, I've not tested it yet):

sub method4 {
my ( $class, $zone ) = @_;
return $zone->glue;
}

sub _build_glue {
my ( $self ) = @_;
my @glue_names = @{ $self->glue_names };
my $zname = $self->name->string;
if ( Zonemaster::Engine::Recursor->has_fake_addresses( $zname ) ) {
my @ns_list;
foreach my $ns ( @glue_names ) {
foreach my $ip ( Zonemaster::Engine::Recursor->get_fake_addresses( $zname, $ns ) ) {
push @ns_list, Zonemaster::Engine::Nameserver->new( { name => $ns, address => $ip } );
}
}
return \@ns_list;
}
else {
my $aref = [];
tie @$aref, 'Zonemaster::Engine::NSArray', @glue_names;
return $aref;
}
}

=head1 NAME
Zonemaster::Engine::NSArray - Class implementing arrays that lazily looks up name server addresses from their names
=head1 SYNOPSIS
tie @ary, 'Zonemaster::Engine::NSArray', @ns_names
=head1 DESCRIPTION
This class is used for the C<glue> and C<ns> attributes of the
L<Zonemaster::Engine::Zone> class. It is initially seeded with a list of
names, which will be expanded into proper L<Zonemaster::Engine::Nameserver>
objects on demand. Be careful with using Perl functions that act on
whole arrays (particularly C<foreach>), since they will usually force
the entire array to expand, negating the use of the lazy-loading.

ref: https://perldoc.perl.org/perltie (I'm discovering this tie function, never used nor tested it yet.)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the implemented Method4 include non-glue address records

Digging into the implementation, I would answer yes (however, I've not tested it yet):

Well, then it is probably enough if the implementation uses Method2 to get the NS in the delegation and Method4 to get nameserver IP addresses using Method4.

@tgreenx tgreenx linked an issue Mar 7, 2023 that may be closed by this pull request
@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 7, 2023

@matsduf @PNAX I think I got to a suitable workaround for both issues. See commit 9e700c7. Let me know what you think (note that unitary tests data are still not updated and thus will fail).

$ zonemaster-cli --show-testcase --test basic/basic02 lame.dufberg.se --level INFO --no-ipv6 --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
  20.94 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame.dufberg.se"
  20.94 WARNING   BASIC02        B02_NS_NO_RESPONSE   ns=ns2.lame.dufberg.se/188.126.83.221
  20.94 WARNING   BASIC02        B02_NS_NO_RESPONSE   ns=ns1.lame.dufberg.se/159.253.30.207

@tgreenx tgreenx requested a review from matsduf March 7, 2023 16:20
@matsduf
Copy link
Contributor

matsduf commented Mar 8, 2023

@matsduf @PNAX I think I got to a suitable workaround for both issues. See commit 9e700c7. Let me know what you think (note that unitary tests data are still not updated and thus will fail).

I am not sure that it is a work-around or just the solution for the methods that are available, but it looks fine. I will create some more test delegations for related scenarios to see if they behave the same.

@matsduf
Copy link
Contributor

matsduf commented Mar 8, 2023

I tested the following test zones and I can see that the code behaves correctly. Unfortunately, the specification calls for checking for the AA flag before checking for the RCODE, but I think we can live with that for now.

$ zonemaster-cli --show-testcase --test basic/basic02 lame-ns-no-name.dufberg.se. --level INFO --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
   0.45 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame-ns-no-name.dufberg.se"
   0.45 ERROR     BASIC02        B02_NS_NO_IP_ADDR   nsname=ns1.test.xa

$ zonemaster-cli --show-testcase --test basic/basic02 lame-ns-no-addr.dufberg.se --level INFO --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
   0.49 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame-ns-no-addr.dufberg.se"
   0.49 ERROR     BASIC02        B02_NS_NO_IP_ADDR   nsname=ns.se

$ zonemaster-cli --show-testcase --test basic/basic02 lame-ns-no-response.dufberg.se --level INFO --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
  10.55 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame-ns-no-response.dufberg.se"
  10.55 WARNING   BASIC02        B02_NS_NO_RESPONSE   ns=ns1.lame-ns-no-response.dufberg.se/159.253.30.207

$ zonemaster-cli --show-testcase --test basic/basic02 lame-ns-refused.dufberg.se. --level INFO --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
   0.51 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame-ns-refused.dufberg.se"
   0.51 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=nsa.dnsnode.net/2a01:3f1:46::53
   0.51 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=nsa.dnsnode.net/194.58.192.46

@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 8, 2023

I tested the following test zones and I can see that the code behaves correctly. Unfortunately, the specification calls for checking for the AA flag before checking for the RCODE, but I think we can live with that for now.

[...]

Good to hear! If you want, I can make a quick update to the specification and swap those two checks around, so that the implementation can be updated too. It seems like a very small thing to do.

Also, do you want me to add those zones to the unitary tests?

@matsduf
Copy link
Contributor

matsduf commented Mar 8, 2023

I created yet another test zone (or rather test delegation):

zonemaster-cli --show-testcase --test basic/basic02 lame-ns-servfail.dufberg.se. --level INFO --raw
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.1
   0.55 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="lame-ns-servfail.dufberg.se"
   0.55 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=dns3.narnia.pp.se/2a05:d014:4ae:e900:e026:2d09:8d68:2d43
   0.55 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=dns3.narnia.pp.se/3.124.111.178

Good to hear! If you want, I can make a quick update to the specification and swap those two checks around, so that the implementation can be updated too. It seems like a very small thing to do.

Please review zonemaster/zonemaster#1145

Also, do you want me to add those zones to the unitary tests?

Yes, that would be good.

After an update based on zonemaster/zonemaster#1145, the following zones should return B02_NS_NO_IP_ADDR:

  • lame-ns-no-name.dufberg.se.
  • lame-ns-no-addr.dufberg.se.
  • lame-ns-no-glue.dufberg.se.

The following B02_NS_NO_RESPONSE:

  • lame-ns-no-response.dufberg.se.

The following B02_UNEXPECTED_RCODE:

  • lame-ns-refused.dufberg.se.
  • lame-ns-servfail.dufberg.se.

tgreenx added 3 commits March 9, 2023 10:50
Update implementation of Basic02 to latest specification (Zonemaster PR#1085)
Update unitary tests:
	- some are entirely new, so unitary tests data needs an update but it has not been done yet
	- modernized utility functions
- Handling of lame delegations by making use of Method2 (i.e. Engine::Zone::_build_glue_names()) and Engine::Zone::_build_glue_addresses().
  It is needed because Engine::Recursor::_recurse(), used by Engine::Recursor::get_addresses_for(), ignores unresponsive name servers when doing recursive lookups, which then leads to empty Engine::Zone attributes ('glue', and thus, 'ns_names' and 'ns') when using Method4.
- Handling of name servers that cannot be resolved in an IP address.
- Update to latest specification (Zonemaster PR#1145)
- Add unitary tests
@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 9, 2023

[ ... ]

Please review zonemaster/zonemaster#1145

Done, and implementation is updated accordingly.

Also, do you want me to add those zones to the unitary tests?

Yes, that would be good.

[ ... ]

Done. Also rebased on latest develop.

The only remaining thing is to update unitary test data. I will do that soon.

@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 9, 2023

Unitary tests data are updated, and checks are now passing. Please re-review.

@matsduf matsduf dismissed their stale review March 9, 2023 15:47

It looks good now.

@matsduf
Copy link
Contributor

matsduf commented Mar 9, 2023

Here is another test (expecting B02_NS_BROKEN):

zonemaster-cli --show-testcase --test basic/basic02  --level INFO --raw www.zonemaster.net --ns nsa.dnsnode.net
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.2
   0.11 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="www.zonemaster.net"
   0.11 ERROR     BASIC02        B02_NS_BROKEN   ns=nsa.dnsnode.net/2a01:3f1:46::53
   0.11 ERROR     BASIC02        B02_NS_BROKEN   ns=nsa.dnsnode.net/194.58.192.46

@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 9, 2023

Here is another test:

[ ... ]

I added it to the unitary tests and updated unitary test data.

@matsduf
Copy link
Contributor

matsduf commented Mar 9, 2023

This delegation expects B02_NS_NOT_AUTH:

$ zonemaster-cli --show-testcase --test basic/basic02  --level INFO --raw zonemaster.net --ns g.root-servers.net --ns a.root-servers.net
   0.00 INFO      UNSPECIFIED    GLOBAL_VERSION   version=v4.6.2
   0.26 CRITICAL  BASIC02        B02_NO_WORKING_NS   domain="zonemaster.net"
   0.26 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=g.root-servers.net/2001:500:12::d0d
   0.26 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=a.root-servers.net/2001:503:ba3e::2:30
   0.26 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=a.root-servers.net/198.41.0.4
   0.26 ERROR     BASIC02        B02_NS_NOT_AUTH   ns=g.root-servers.net/192.112.36.4

matsduf
matsduf previously approved these changes Mar 9, 2023
@tgreenx
Copy link
Contributor Author

tgreenx commented Mar 14, 2023

This delegation expects B02_NS_NOT_AUTH:

[ ... ]

Thanks, I included this test in the unitary tests (commit b17fca5). Now we have full testing coverage for Basic02.
Please re-review.

matsduf
matsduf previously approved these changes Mar 14, 2023
@matsduf
Copy link
Contributor

matsduf commented Mar 14, 2023

Now we have full testing coverage for Basic02. Please re-review.

That is really great.

@tgreenx tgreenx merged commit 6bf8c71 into zonemaster:develop Apr 5, 2023
@tgreenx tgreenx deleted the update-basic02 branch April 5, 2023 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-TestCase Area: Test case specification or implementation of test case V-Minor Versioning: The change gives an update of minor in version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update implementation of updated BASIC02 Basic02 gives incorrect error message when zone is completely lame
2 participants