-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add router support for wildcard domains (*.foo.com) #11201
Conversation
[test] |
I assume now route admission has to check for prefix overlaps on route On Oct 3, 2016, at 9:38 PM, Ram Ranganathan notifications@github.com For trello card: Still need to beef up the tests + address route ownership/replacement (for @knobunc https://github.com/knobunc @rajatchopra preliminary feedback that would be greatly appreciated. ThxYou can view, comment on, or merge this pull request online at: #11201
File Changes
Patch Links:
— |
Can we separate out the cert map bit? There's a chance we'll want to backport that (along with enabling cert validation in the API by default) |
@smarterclayton yes route admission now has to check for ownership of a @liggitt yeah, I can break out the cert validation bits tomorrow (along with a separate simpler/just enable extended validation PR so that its easier to backport). |
@@ -100,15 +100,36 @@ frontend public | |||
acl secure_redirect base,map_beg(/var/lib/haproxy/conf/os_edge_http_redirect.map) -m found | |||
redirect scheme https if secure_redirect | |||
|
|||
{{ if matchPattern "true|TRUE" (env "ROUTER_ALLOW_WILDCARD_ROUTES" "")}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will also meet this condition when the value is 1?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could do a 1 in that set as well if need be but true|TRUE
follows the other parameters in this file.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, let's just support true for now. The SYN eater I added got it backwards and added '1' so I fixed that later and support 'true' and '1', but 'true' is preferred.
|
||
func hostInDomainList(host string, domains []string) bool { | ||
for _, dom := range domains { | ||
if strings.HasSuffix(host, fmt.Sprintf(".%s", dom)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this seems like a really expensive way to check this
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do you recommend using sort.Search
? For a small set in the list, it might not be as performant though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is called for every host. we shouldn't be sprintfing inside a loop, the allocations will kill us.
instead, I'd expect a sets.String
object containing the domains, and to iterate over the host, finding locations of .
and checking shorter and shorter substrings against the domain set.
func hostInDomainList(host string, domains sets.String) bool {
if domains.Has(host) {
return true
}
if i := strings.IndexRune(host, '.'); i >= 0 {
return hostInDomainList(host[i+1:], domains)
}
return false
}
@@ -43,6 +44,14 @@ type RouterSelection struct { | |||
ProjectLabels labels.Selector | |||
|
|||
IncludeUDP bool | |||
|
|||
DeniedDomains string | |||
BlacklistedDomains []string |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you just need the []string
var, bind it to a StringSliceVar flag
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
will change to that - thx
@@ -107,6 +107,8 @@ func NewTemplatePlugin(cfg TemplatePluginConfig, lookupSvc ServiceLookup) (*Temp | |||
"matchPattern": matchPattern, //anchors provided regular expression and evaluates against given string | |||
"isInteger": isInteger, //determines if a given variable is an integer | |||
"matchValues": matchValues, //compares a given string to a list of allowed strings | |||
|
|||
"genDomainWildcardRegexp": genDomainWildcardRegexp, //generates a regular expression matching wildcard hosts (and paths) for a [sub]domain |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
do we have tests in place to make sure we don't break the "API" we're surfacing for custom template files to use?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still adding tests - would be with the unique_host checks.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should add the tests against the template struct as it exists prior to this PR
func genDomainWildcardRegexp(hostname, path string, exactPath bool) string { | ||
route := &routeapi.Route{Spec: routeapi.RouteSpec{Host: hostname}} | ||
host, _ := validation.NormalizeWildcardRoute(route) | ||
expr := regexp.QuoteMeta(fmt.Sprintf(".%s%s", host, path)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
does go regex quoting exactly match haproxy config file regex quoting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The quoting should but its RE2 vs PCRE. That said though, the only meta character we are using here is *
and .
- which is ok.
[test] |
this is now ready for review. will add a couple more tests tomorrow. PTAL thx Spent a while debugging this today - there was a big issue this found with: And this is also waiting on #11217 to merge - one of the commits (for the cert list) would then drop off this PR. Updated commit sha |
396ccb5
to
14ec738
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nothing functional to note. My main suggestion would be simpler testing to make it easier to evaluate test quality.
if route1.UID < route2.UID { | ||
return true | ||
} | ||
if route1.Namespace < route2.Namespace { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry about that, this bug was my fault.
As you say, comparisons with other fields only make sense if the timestamps are equal. Given that UIDs can never be equal, though, what would be the point of comparing other fields? Consider removing namespace and name comparison:
if route1.CreationTimestamp == route2.CreationTimestamp {
if route1.UID < route2.UID {
return true
}
}
return false
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will do - have to check implications to the other tests (plugin_tests
). Should be ok I think - just need to ensure that UIDs are not the same in any of tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll keep this for now as a few other tests use it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Was this only in 1.4 or also in 1.3? IF so we need to backport.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
|
||
// NormalizeWildcardRoute tests if a route host is wildcarded and returns | ||
// the "normalized" (domain name currently) form of the route host. | ||
func NormalizeWildcardRoute(route *routeapi.Route) (string, bool) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider simplifying this function according to the Law of Demeter:
func NormalizeWildcardRoute(host string) (string, bool) {...}
This would simplify testing since a route object would no longer be required.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess if we do that - might be better to pull the function into helper.go
.
return nil | ||
} | ||
|
||
func getRouteSubdomain(route *routeapi.Route) string { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider passing the host instead of the route (Demeter).
@@ -0,0 +1,399 @@ | |||
package controller |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider simplifying these tests by validating each function exhaustively in isolation but minimally in combination. This would allow fewer test cases to provide more coverage. At the least I suggest testing addRoute and HandleRoute separately.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ack
errors bool | ||
}{ | ||
{ | ||
name: "No Host", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider simplifying by creating route objects in the test rather than the test data, e.g.
tests := []struct {
name string
routeName string
host string
errors bool
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The created routes are different though as they test for variations in the name
/namespace
and CreationTimestamp
and the Spec.Host
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This particular function doesn't vary the namespace but even if it did it would be simpler to include namespace as a variable:
tests := []struct {
name string
routeName string
namespace string
host string
errors bool
}{
{
name: "Blocked Host",
routeName: "blocked",
namespace: "host",
host: "host.domain.blocked.test",
errors: true,
},
}
for _, tc := range tests {
route := &routeapi.Route{
ObjectMeta: kapi.ObjectMeta{
Name: tc.routeName
Namespace: tc.namespace,
},
Spec: routeapi.RouteSpec{
Host: tc.host,
}
}
err := admitter.HandleRoute(watch.Added, route)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The ordinal relationship of tests's CreationTimestamp would be the same, since the routes would still be created in the same order.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I want to explicitly add routes with different timestamps out of band to check - rather than do a sequential check.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
never mind, I saw you meant the previous test TestHostAdmit
.
r.rejections[r.rejectionKey(route)] = reason | ||
} | ||
|
||
func admissionChecker(denyDomain string, allowWildcards bool) RouteAdmissionFunc { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given that the only point of the admitter function in this file's testing is to ensure that HandleRoute executes route rejection, consider using an anonymous function that statically returns nil
or err
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The parameters to the admissionChecker
function are different for the different tests and NewHostAdmitter
takes that generated function as an input parameter.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get that, but these are unit tests. The only thing the checker does is indicate to HandleRoute whether a route should be rejected or not, so there's no point in adding logic that enables more than that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have a strong opinion either way and will do it but to create 3/4 copies of this anonymized (or named) with a small variation seemed a bit too excessive.
[test] |
[test] |
1 similar comment
[test] |
@smarterclayton / @liggitt could you PTAL thx |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @ramr
@@ -100,15 +100,36 @@ frontend public | |||
acl secure_redirect base,map_beg(/var/lib/haproxy/conf/os_edge_http_redirect.map) -m found | |||
redirect scheme https if secure_redirect | |||
|
|||
{{ if matchPattern "true|TRUE" (env "ROUTER_ALLOW_WILDCARD_ROUTES" "")}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, let's just support true for now. The SYN eater I added got it backwards and added '1' so I fixed that later and support 'true' and '1', but 'true' is preferred.
} | ||
|
||
if hostInDomainList(route.Spec.Host, o.BlacklistedDomains) { | ||
glog.V(4).Infof("host %s in list of denied domains", route.Spec.Host) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we put quotes around the hostnames? Or is that not the pattern elsewhere?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think most/majority of the logs don't have quotes. Probably more readable without it I think.
// for a [sub]domain. | ||
func genDomainWildcardRegexp(hostname, path string, exactPath bool) string { | ||
route := &routeapi.Route{Spec: routeapi.RouteSpec{Host: hostname}} | ||
host, _ := routeapi.NormalizeWildcardHost(route.Spec.Host) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It feels a little odd to generate wildcards even if the host doesn't have a leading wildcard. But I don't know how to handle the error path, and we are checking in the config file that they do... so it should be safe.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Its a helper function for the template. Yeah its called from the template which calls it for wildcard routes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldn't harm if we explicitly just checked the boolean value returned by NormalizeWildcardHost. Its a function exposed in the template, who knows how/where field use may call it with. If I just call it without a leading '*' for example, outside the scope of $cfg.IsWildcard check.
@@ -20,11 +20,16 @@ func RouteLessThan(route1, route2 *Route) bool { | |||
if route1.CreationTimestamp.Before(route2.CreationTimestamp) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please change the commit description to be more descriptive of the change :) It massively helps me when I'm trying to draft release notes and need to read through 1000+ commits if the subject of each commit reflects the actual change.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ack
Fix that exposed genDomainWildcardRegexp function for cases where it is called without check on $cfg.IsWildcard. |
This is the same criteria every API change for the last two years has gone through. We've made a few, very rare exceptions, and they have been almost completely for the following:
|
Jordan's arguing for a net new field (on the side) - that would be:
That's potentially valid and would be equivalent to the
We'd probably have to have a master flag to change the defaulting behavior, or likewise require a cluster admin to opt in. |
This is really no different than TLS termination policy (the other example of adding a policy that changed the default). Opt-in behavior that new routers understood (and behaved differently) and old routers ignored. |
What is |
I think it's worth considering just how much flexibility we want to build into routes with an impending move towards upstream ingress looming. |
@@ -0,0 +1,204 @@ | |||
package controller |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is it desirable to add a new plugin vs updating UniqueHost to support wildcard routes? I don't understand why it wouldn't be simpler and less error prone to reuse the existing logic with normalized host values.
Jordan to insert his clever suggestion that preserves compatibility (needs feedback on the original use case and the outcome) HERE: |
@smarterclayton - what about using an annotation then (for The route says I want a host
subdomain == nested in your previous comments. Older routers would ignore that annotation (well it would give you a host Thoughts? |
we need to keep in mind how route.spec.host and route.status is used by all clients. that includes:
we can maintain compatibility with all of those clients if the host field continues to contain a prototypical dns name, and we make the wildcard request a separate field that indicates wildcarding of the first segment is requested:
would get you |
And if you didn't want the name to be valid, you could always choose "zzzzzzzz.foo.com" or something to indicate it's not real. |
I think i'd be ok with this for the primary concerns I had earlier. |
@liggitt suggestion in #11201 (comment) is the least offensive thing I've heard. I do wonder if 'wildcard' should be a |
that would be Ram, how much disruption to the current code is this? |
@smarterclayton shouldn't be that much hopefully. Will need to adjust the flow upto the admission controller and remove some code (validation etc) - beyond the admission controllers, it would be the same - maybe a minor change to the existing function to get the subdomain part from the host for generating the regexp. |
no concerns, just a few questions:
|
@liggitt so in response to your questions, I'd say:
|
don't we want status to reflect what the router actually admitted? I'd expect wildcardPolicy to be echoed somehow in it |
Was trying to get you some sample output - but am running into some issues. Please see: #11550 - maybe you have some ideas why the persisted object is missing the wildcardPolicy field. Anyway, the intended result is the status of the route has an message that indicates what the router admitted. Routers can still override the intention that the route specifies - ala I want this route to support wildcards for the subdomain but a router with no wildcard route support would reject it but admit the host. |
Clayton, if the old router images just ignore the wildcard hostnames -ben On Mon, Oct 24, 2016 at 2:52 PM, Clayton Coleman notifications@github.com
|
EDIT: wildly time delayed by google The latter form isn't backwards compatible with existing servers, which is |
BTW that last comment was an email from yesterday, just arriving now (thanks github). Ben, to your question, changing validation broke the meaning of host, and changing the meaning of host (even if our router code tolerates it) can result in security risks for our customers. So we at minimum need the new field that adds behavior (but doesn't change existing behavior) |
I'd like to see clear documentation and tests around the following
|
EDIT: wildly delayed a. Ram's suggestion of "Subdomain" worked for me. On Mon, Oct 24, 2016 at 5:12 PM, Jordan Liggitt notifications@github.com
|
For trello card: https://trello.com/c/cX42kXSA/222-7-subdomain-wildcard-router-ingress-demo
Still need to beef up the tests + address route ownership/replacement (for wildcard to overlapping routes) but this has support for the card via an environment variable + allows a {white,black}listing of host names. The route ownership/replacement might make this PR a bit too big - its big enough as is.
@knobunc @rajatchopra @marun @smarterclayton if you have some preliminary feedback that would be greatly appreciated. Thx