Use "inherit left, wildcard right" behavior in base URL application and constructor string parsing. #198

jeremyroman · 2023-10-27T20:40:22Z

The following changes apply to patterns which are constructed using a base URL, the constructor string syntax, or both -- but not any pattern which explicitly specifies components separately without a base URL.

Components are not inherited from a base URL if an "earlier" component is explicitly specified.
In the string format, unspecified "later" components are implicitly wildcarded, rather than required to be empty (with the exception of the port, which is always taken to be specified when the hostname is).
Username and password are never implicitly specified or inherited.

For example:

"https://example.com/*" also matches with any username, password, search, and hash. Previously this would be written "https://*:*@example.com/*\\?*#*".
new URLPattern({ pathname: "/login" }, "https://example.com/?q=hello") accepts any query string and hash, not only "?q=hello" and "".
"https://*:*" or {protocol: "https"} means "any HTTPS URL, on any port", and "https://*" means "any HTTPS URL on the default port (443)". These have the same meaning whether or not a base URL is provided, since specifying the protocol prohibits inheritance of other components.

This makes patterns more expansive than before, in cases where wildcards are likely to be desirable.

The logic of inheriting components from a base URL dictionary is also similarly changed in a way that may make it not match where it did before, but more consistently with the above and with how relative URL strings are resolved. For example, new URLPattern("https://example.com/foo?q=1#hello").test({pathname: "/foo", hash: "hello", baseURL: "https://example.com/bar?q=1"}) previously returned true but will now be false, since the search component is not inherited when the pathname is specified. This is analogous to how new URL("/foo#hello", "https://example.com/bar?q=1") works. The reverse is also possible; in both cases this is quite niche.

Fixes #179.

At least two implementers are interested (and none opposed):
- Google Chrome
- Safari (per Base URL inheritance gives unintuitive results #179 (comment))
Tests are written and can be reviewed and commented upon at:
- https://chromium-review.googlesource.com/c/chromium/src/+/4975595
Implementation bugs are filed:
- Chromium: https://bugs.chromium.org/p/chromium/issues/detail?id=1468446
- Gecko: https://bugzilla.mozilla.org/show_bug.cgi?id=1731418 (vendor does not yet implement the spec)
- WebKit: [no known URLPattern bug] (vendor does not yet implement the spec)
- Deno: URLPattern: Implement whatwg/urlpattern#198. denoland/deno#21073
- kenchris/urlpattern-polyfill: Implement "inherit left, wildcard right" behavior from whatwg/urlpattern#198 kenchris/urlpattern-polyfill#122
MDN issue is filed: URLPattern: Reflect "inherit left, wildcard right" behavior from whatwg/urlpattern#198. mdn/content#30024
The top of this comment includes a clear commit message to use.

(See WHATWG Working Mode: Changes for more details.)

Preview | Diff

jeremyroman · 2023-10-27T20:40:37Z

@sisidovski @domenic FYI

jeremyroman · 2023-11-01T01:54:05Z

Updated commit message in the opening comment (hopefully it's reasonably clear).

I assume Gecko and WebKit implementation bugs are not desired since they don't yet (to my knowledge) implement URLPattern; is that right? I can file Deno and urlpattern-polyfill ones shortly before merging if appropriate. Ditto for MDN.

For implementer interest, it sounded like Anne was on board with this direction; what level of assent is required to add WebKit to the implementer interest list?

domenic · 2023-11-01T04:01:32Z

Updated commit message in the opening comment (hopefully it's reasonably clear).

LGTM! You might want to add something about the compatibility implications, and how we've measured them to be low.

I assume Gecko and WebKit implementation bugs are not desired since they don't yet (to my knowledge) implement URLPattern; is that right?

Right, we can treat them as being under the umbrella "implement URLPattern" bugs.

For implementer interest, it sounded like Anne was on board with this direction; what level of assent is required to add WebKit to the implementer interest list?

What I would do is check the box for WebKit, linking to #179 (comment) , but also tag @annevk to confirm, and give a few days before merging in case he has objections or wants to take a closer look.

jeremyroman · 2023-11-02T20:14:21Z

What I would do is check the box for WebKit, linking to #179 (comment) , but also tag @annevk to confirm, and give a few days before merging in case he has objections or wants to take a closer look.

Alright, will do unless @annevk objects.

domenic

Normatively LGTM, just editorial suggestions

spec.bs

annevk · 2023-11-06T09:07:27Z

I didn't do an in-depth review, but happy to agree on behalf of WebKit with the behavior we're aiming for here.

domenic

LGTM with more nits, not all of which I'm sure about. @sisidovski did you want to take a look?

spec.bs

domenic · 2023-11-07T05:41:32Z

spec.bs

+      * protocol, hostname, port, pathname, search, hash
+      * protocol, hostname, port, username, password
+
+      Username and password are also never inherited from a base URL when constructing a {{URLPattern}}. (They are, however, inherited from the base URL supplied as an argument to {{URLPattern/test()}} or {{URLPattern/exec()}}.)


Suggested change

Username and password are also never inherited from a base URL when constructing a {{URLPattern}}. (They are, however, inherited from the base URL supplied as an argument to {{URLPattern/test()}} or {{URLPattern/exec()}}.)

Username and password are also never inherited from a base URL when constructing a {{URLPattern}}. (They are, however, inherited from the base URL when parsing a URL from the string supplied as an argument to {{URLPattern/test()}} or {{URLPattern/exec()}}.)

Added "when parsing a URL". You can actually supply a dictionary rather than a string if you want, and this behavior also applies there. e.g. urlPattern.test({pathname: "/world"}, "https://user:pass@example.com/") is a thing you can do.

spec.bs

sisidovski · 2023-11-07T06:44:13Z

spec.bs

@@ -510,17 +601,14 @@ A [=constructor string parser=] has an associated <dfn export for="constructor s
  <p>First, the URLPattern constructor string parser operates on [=tokens=] generated using the "`lenient`" [=tokenize policy=].  In constrast, [=basic URL parser=] operates on code points.  Operating on [=tokens=] allows the URLPattern constructor string parser to more easily distinguish between code points that are significant pattern syntax and code points that might be a URL component separator.  For example, it makes it trivial to handle named groups like "`:hmm`" in "`https://a.c:hmm.example.com:8080`" without getting confused with the port number.
  <p>Second, the URLPattern constructor string parser needs to avoid applying URL canonicalization to all code points like [=basic URL parser=] does.  Instead we perform canonicalization on only parts of the pattern string we know are safe later when compiling each component pattern string.
  <p>Finally, the URLPattern constructor string parser does not handle some parts of the [=basic URL parser=] state machine.  For example, it does not treat backslashes specially as they would all be treated as pattern characters and would require excessive escaping.  In addition, this parser might not handle some more esoteric parts of the URL parsing algorithm like file URLs with a hostname.  The goal with this parser was to handle the most common URLs while allowing any niche case to be handled instead via the {{URLPatternInit}} constructor.
+  <p>In the constructor string algorithm, the pathname, search, and hash are wildcarded if earlier components are specified but later ones are not. For example, "`https://example.com/foo`" matches any search and any hash. Similarly, "`https://example.com`" matches any URL on that origin. This is analogous to the notion of a more specific component in the notes about [=process a URLPatternInit=] (e.g., a search is more specific than a pathname), but the constructor syntax only has a few cases where it is possible to specify a more specific component without also specifying the less specific components.
+  <p>The username and password components are always wildcard unless they are explicitly specified.


Do we want a mention the exception of port here as other components are explicitly mentioned? It also makes sense not having it since we have a note in L723 though.

Added a mention here.

sisidovski · 2023-11-07T08:15:30Z

spec.bs

-  1. If |init|["{{URLPatternInit/port}}"] is not null then set |result|["{{URLPatternInit/port}}"] to the result of [=process port for init=] given |init|["{{URLPatternInit/port}}"], |result|["{{URLPatternInit/protocol}}"], and |type|.
-  1. If |init|["{{URLPatternInit/pathname}}"] is not null:
+    1. If |init|["{{URLPatternInit/protocol}}"] does not [=map/exist=], then set |result|["{{URLPatternInit/protocol}}"] to the result of [=processing a base URL string=] given |baseURL|'s [=url/scheme=] and |type|.
+    1. If |type| is not "`pattern`" and |init| [=map/contains=] none of "{{URLPatternInit/protocol}}", "{{URLPatternInit/hostname}}", "{{URLPatternInit/port}}" and "{{URLPatternInit/username}}", then set |result|["{{URLPatternInit/username}}"] to the result of [=processing a base URL string=] given |baseURL|'s [=url/username=] and |type|.


Just to confirm, |type| will be pattern only when constructing, right? match() and exec() also process URLPatternInit, but |type| is "url" in that case so I believe this is correct.

Yes, exactly.

sisidovski · 2023-11-07T08:23:04Z

Apologize for the delay, LGTM with a few questions.

jeremyroman · 2023-11-07T21:52:13Z

Chromium CL has also landed, and should ship in M121. 🤞

This CL changes the behavior of the router rule registration in the ServiceWorker Static Routing API, especially when the |urlPattern| condition accepts the URLPatternInit object. Before this CL, the URLPatternInit input is accepted as it is, that means any unspecified fields are resulted in the wildcards (*). This behavior is inconsistent with the case when |urlPattern| accepts a string. When a string is passed, missing fields are complemented by baseURL, which the SW script URL is internally used when adding the new router rule. So some missing components will use the values inherited from the baseURL in that case. After this CL, the URLPatternInit input also internally uses the SW script URL as a baseURL if not provided. It typically happens when the input is an object that complies with the URLPatterInit interface. On the other hand, it wouldn't affect the case when URLPatternInit is the output of `new URLPattern()`. Thank to the recent change on the URLPattern[1], the constructed components in URLPatternInit will inherit or wildcarded. baseURL won't override components if those are not empty. This behavior change is based on the discussion in whatwg/urlpattern#182. [1] whatwg/urlpattern#198 Bug: 1371756 Change-Id: I5cce80fde05cf18237c8b6412b00e017ff5aad5b

jeremyroman added 3 commits October 27, 2023 15:15

changes to base URL init application

b10fa88

factor out the use of map/contains to scan better

2c9346c

constructor string parser state machine changes

466a792

jeremyroman added 2 commits October 27, 2023 17:06

spelling: It -> If

05c12a4

update some notes and the main example

2d122ee

jeremyroman added impacts documentation Used by documentation communities, such as MDN, to track changes that impact documentation needs implementer interest Moving the issue forward requires implementers to express interest labels Nov 1, 2023

jeremyroman requested review from sisidovski and domenic November 2, 2023 20:19

domenic approved these changes Nov 6, 2023

View reviewed changes

jeremyroman added 4 commits November 6, 2023 17:18

Add more examples.

dc6a52d

domenic review

f9e1c99

Use map/exists instead of 'is not null' in existing normative text.

cad7463

unique IDs for the examples

551c882

domenic approved these changes Nov 7, 2023

View reviewed changes

sisidovski approved these changes Nov 7, 2023

View reviewed changes

domenic & sisidovski review

7cad49e

jeremyroman merged commit ed205a4 into whatwg:main Nov 7, 2023
2 checks passed

jeremyroman deleted the syntax-179 branch November 7, 2023 21:53

jeremyroman mentioned this pull request Nov 7, 2023

Apply syntax changes proposed in whatwg/urlpattern#198. kenchris/urlpattern-polyfill#123

Merged

jeremyroman mentioned this pull request Nov 8, 2023

URLPattern object literal syntax results seem unintuitive WICG/nav-speculation#260

Closed

chromium-wpt-export-bot mentioned this pull request Nov 21, 2023

Add baseURL to the URLPattern condition while routing rule registration web-platform-tests/wpt#43272

Merged

crowlKats mentioned this pull request Nov 29, 2023

feat: use "inherit left, wildcard right" behavior denoland/rust-urlpattern#44

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use "inherit left, wildcard right" behavior in base URL application and constructor string parsing. #198

Use "inherit left, wildcard right" behavior in base URL application and constructor string parsing. #198

jeremyroman commented Oct 27, 2023 •

edited by pr-preview bot

Loading

jeremyroman commented Oct 27, 2023

jeremyroman commented Nov 1, 2023

domenic commented Nov 1, 2023

jeremyroman commented Nov 2, 2023

domenic left a comment

annevk commented Nov 6, 2023

domenic left a comment

domenic Nov 7, 2023

jeremyroman Nov 7, 2023

sisidovski Nov 7, 2023

jeremyroman Nov 7, 2023

sisidovski Nov 7, 2023

jeremyroman Nov 7, 2023

sisidovski commented Nov 7, 2023

jeremyroman commented Nov 7, 2023

	Username and password are also never inherited from a base URL when constructing a {{URLPattern}}. (They are, however, inherited from the base URL supplied as an argument to {{URLPattern/test()}} or {{URLPattern/exec()}}.)
	Username and password are also never inherited from a base URL when constructing a {{URLPattern}}. (They are, however, inherited from the base URL when parsing a URL from the string supplied as an argument to {{URLPattern/test()}} or {{URLPattern/exec()}}.)

Use "inherit left, wildcard right" behavior in base URL application and constructor string parsing. #198

Use "inherit left, wildcard right" behavior in base URL application and constructor string parsing. #198

Conversation

jeremyroman commented Oct 27, 2023 • edited by pr-preview bot Loading

jeremyroman commented Oct 27, 2023

jeremyroman commented Nov 1, 2023

domenic commented Nov 1, 2023

jeremyroman commented Nov 2, 2023

domenic left a comment

Choose a reason for hiding this comment

annevk commented Nov 6, 2023

domenic left a comment

Choose a reason for hiding this comment

domenic Nov 7, 2023

Choose a reason for hiding this comment

jeremyroman Nov 7, 2023

Choose a reason for hiding this comment

sisidovski Nov 7, 2023

Choose a reason for hiding this comment

jeremyroman Nov 7, 2023

Choose a reason for hiding this comment

sisidovski Nov 7, 2023

Choose a reason for hiding this comment

jeremyroman Nov 7, 2023

Choose a reason for hiding this comment

sisidovski commented Nov 7, 2023

jeremyroman commented Nov 7, 2023

jeremyroman commented Oct 27, 2023 •

edited by pr-preview bot

Loading