#1448 url encoding with lower case letters

[mstrewe]

Will fix #1448 the lower case url encoding error. Test added

[rzo1]

Formatting seems to be off. Think you can resolve that by running

mvn git-code-format:format-code -Dgcf.globPattern="**/*" -Dskip.format.code=false

[sebastian-nagel]

Hi @mstrewe,

thanks for the PR.

Please, see the comment regarding the uppercase representation of percent-encodings.

[sebastian-nagel]

Shouldn't the expected result be %3D%3D?

This is the canonical representation of percent-encoded characters defined in RFC 3986.

If case variants of percent-encoded chars remain in URLs, this may cause duplicates. Note that in addition to pure lowercase variant, there could be also %3d%3D and %3D%3d.

[sebastian-nagel]

After a closer look into the code: the reason for the issue is likely in line 398 of BasicURLNormalizer.

• a percent character is unconditionally converted to %25 even if it's the first character of a valid percent-encoding
• the "basic" URL normalizers of Nutch and crawler-commons treat the percent character separately and do not unconditionally escape it. All three "basic" URL normalizers share the same origin years ago, so they are still quite similar in their source code.

[mstrewe]

After a closer look into the code: the reason for the issue is likely in line 398 of BasicURLNormalizer.

• a percent character is unconditionally converted to %25 even if it's the first character of a valid percent-encoding
• the "basic" URL normalizers of Nutch and crawler-commons treat the percent character separately and do not unconditionally escape it. All three "basic" URL normalizers share the same origin years ago, so they are still quite similar in their source code.

I dont think so.
In the given URL of the test, the URL is unescaped first in line 146 and then escaped in 147. Until now the encoding differ only in upper and lower case (the percent is not yet encoded again)

// .../NjAxOA%3d%3d     - file
String file2 = unescapePath(file);
// .../NjAxOA==    - file2
file2 = escapePath(file2);
// .../NjAxOA%3D%3D   - file2

So the escaping unescaping works like expected.

But since the letters now upper case equals (without ignore case) will lead to line 152, which will create a new URL with file 2.

urlToFilter = new URL(protocol, host, port, file2).toString();

This line will then encode the percentage character again. Then we have .../NjAxOA%253D%253D

[sebastian-nagel]

Ok, this might require to run a debugger. But it doesn't seem to be the URL constructor:

jshell> new URL("http", "www.example.com", -1, "/NjAxOA%3D%3D").toString();
$1 ==> "http://www.example.com/NjAxOA%3D%3D"

[mstrewe]

I ran the test again, without the fix.

It returned

BasicURLNormalizerTest.testProperURLEncodingWithLowerCase:313 Failed to normalize url encoded url with lower case letters ==> expected: <http://www.example.com/Exhibitions/Detail/NjAxOA%3d%3d> but was: <http://www.example.com/Exhibitions/Detail/NjAxOA%3D%3D>

It seems to work correctly.

I found the bug in my software using maybe an older version.. I will investigate

[mstrewe]

OK found it.

In my Version the line 154 is the following

urlToFilter = new URI(protocol, null, host, port, file2, null, null).toURL().toString();

Sorry for bothering you. I will close the merge Request since problem was my old code.

[sebastian-nagel]

Sorry for bothering you. I will close the merge Request since problem was my old code.

No problem. Thanks!