Skip to content
This repository has been archived by the owner on Mar 19, 2024. It is now read-only.

http-route-controller: watch ReferencePolicy lifecycle #156

Merged
merged 47 commits into from
Apr 26, 2022

Conversation

mikemorris
Copy link
Contributor

@mikemorris mikemorris commented Apr 21, 2022

Changes proposed in this PR:

Adds a Watches clause to the HTTPRouteReconciler to watch for changes to ReferencePolicy objects and trigger reconciliation and re-validation for any HTTPRoutes that may be affected by changes.

(Please don't view by commit, it's too messy to bother trying to interactive rebase cleanly 🙈)

How I've tested this PR:

  • Added a unit test for GatewayClient.GetHTTPRoutesInNamespace
  • Added a unit test for HTTPRouteReconciler.referencePolicyToRouteRequests
  • Add end-to-end tests for adding, updating and deleting a ReferencePolicy

How I expect reviewers to test this PR:

  • Validate if we actually want to release with this behavior. Triggering a reconciliation with our current HTTPRoute validation logic will evict a Route entirely from a Gateway if any BackendRefs are unpermitted due to the lack of a ReferencePolicy. If I understand correctly we're blocked on accepting a partially-invalid Route and programming Envoy to send 503 responses for the invalid rules until we have more direct control over xDS generation.

Reference Documentation:

For unimplemented GroupKind matching on ReferencePolicyFrom instead of assuming HTTPRoute, and BackendRef filtering on ReferencePolicyTo:

Checklist:

  • Tests added
  • CHANGELOG entry added

    HashiCorp engineers only, community PRs should not add a changelog entry.
    Entries should use imperative present tense (e.g. Add support for...)

Comment on lines -1034 to -1035
if condition.Type == "Accepted" ||
condition.Status == "True" {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactor uncovered that this should've been &&, so this check was erroneously passing on any condition with Status: "True", and was being called in a few places where conditionReady should have been called instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good find!

@mikemorris mikemorris marked this pull request as ready for review April 22, 2022 21:44
@mikemorris mikemorris requested a review from a team April 22, 2022 22:02
Co-authored-by: Nathan Coleman <nathan.coleman@hashicorp.com>
internal/k8s/gatewayclient/gatewayclient.go Show resolved Hide resolved
Comment on lines +166 to +167
// TODO: search by from.Group and from.Kind instead of assuming HTTPRoute
routes, err := r.Client.GetHTTPRoutesInNamespace(r.Context, string(from.Namespace))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sarahalsmiller could you look into how best to do this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a few hours of digging through the dynamic and unstructured packages, I don't think it's strictly possible to search for multiple kinds in a single API call with the go-client (would love to be proven wrong here though). The most generic thing I can figure out how to do is potentially use a mapper to search for the routes that are available on the cluster and list each of those found group version kinds and apply our logic from there, but I'm not sure that even makes sense at the moment when we're only supporting two kinds.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking it's not worth more digging vs. just having r.Client.GetTCPRoutesInNamespace( ... ). Could wait and dig more if we wind up hitting rule of three

Copy link
Contributor Author

@mikemorris mikemorris Apr 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 to revisiting this when we get to implementing TLSRoute or UDPRoute - something more generic might be possible with unstructured.Unstructured struct and scheme.Runtime using reflection to convert from kind strings to types as described in https://iximiuz.com/en/posts/kubernetes-api-go-types-and-common-machinery/ (and likely adding a RouteController interface), but definitely not straightforward.

Implemented as GetTCPRoutesInNamespace in #162

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than doing things with strings and unstructured, if you really want you could just make a generic List/ListInNamespace function on the interface and pass in the object you want retrieved directly. The k8s client uses reflection for retrieving lists/objects anyway. It'd handle something like client.ListInNamespace(ctx, &gateway.HTTPRouteList{}, namespace). Which is a lot closer to the vanilla k8s client List method. TCPRoute would just then be client.ListInNamespace(ctx, &gateway.TCPRouteList{}, namespace). The function signature would just specify a runtimeclient.ObjectList.

The main two reasons you'll still want to wrap the k8s client call, even if it's a really thin wrapper over List are for:

  1. K8sError wrapping for our retry middleware
  2. Easy mocking for error injection for tests

I don't necessarily have much of a preference but think the per-type signatures are kind of nice in that you can just having the initialized structures passed back to you. Per-type also follow more of the pattern throughout most of the gatewayclient interface. But either way is fine with me. If you do change it though, I'd do that in a follow-up PR at this point, no need to re-block.

Copy link
Member

@nathancoleman nathancoleman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few comments above but nothing I'd block merge for

Comment on lines +166 to +167
// TODO: search by from.Group and from.Kind instead of assuming HTTPRoute
routes, err := r.Client.GetHTTPRoutesInNamespace(r.Context, string(from.Namespace))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than doing things with strings and unstructured, if you really want you could just make a generic List/ListInNamespace function on the interface and pass in the object you want retrieved directly. The k8s client uses reflection for retrieving lists/objects anyway. It'd handle something like client.ListInNamespace(ctx, &gateway.HTTPRouteList{}, namespace). Which is a lot closer to the vanilla k8s client List method. TCPRoute would just then be client.ListInNamespace(ctx, &gateway.TCPRouteList{}, namespace). The function signature would just specify a runtimeclient.ObjectList.

The main two reasons you'll still want to wrap the k8s client call, even if it's a really thin wrapper over List are for:

  1. K8sError wrapping for our retry middleware
  2. Easy mocking for error injection for tests

I don't necessarily have much of a preference but think the per-type signatures are kind of nice in that you can just having the initialized structures passed back to you. Per-type also follow more of the pattern throughout most of the gatewayclient interface. But either way is fine with me. If you do change it though, I'd do that in a follow-up PR at this point, no need to re-block.

@mikemorris mikemorris merged commit 028dc0f into main Apr 26, 2022
@mikemorris mikemorris deleted the reference-policy-lifecycle branch April 26, 2022 16:11
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants