From 8ef8f61d1f580302c0ca28b731dab1c232677429 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Thu, 6 May 2021 11:36:18 -0700 Subject: [PATCH 01/17] Update (rewrite) overview, motivation, and data plane --- specs/eventing/data-plane.md | 260 +++++++++++++++++++++-------------- specs/eventing/motivation.md | 58 +++----- specs/eventing/overview.md | 135 +++++++++--------- 3 files changed, 237 insertions(+), 216 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 81e1aa3fc..abb5e6069 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -1,125 +1,171 @@ -# Knative Eventing Data Plane Contracts +# Knative Eventing Data Plane Contract ## Introduction -Developers using Knative Eventing need to know what is supported for delivery to -user provided components that receive events. Knative Eventing defines contract -for data plane components and we have listed them here. - -## Conformance +Late-binding event senders and receivers (composing applications using +configuration) only works when all event senders and recipients speak a common +protocol. In order to enable wide support for senders and receivers, Knative +Eventing extends the [CloudEvents HTTP +bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) +with additional semantics for the following reasons: + +- Knative Eventing aims to enable highly-reliable event processing workflows. As + such, it prefers duplicate delivery to discarded events. The CloudEvents spec + does not take a stance here. + +- The CloudEvents HTTP bindings provide a relatively simple and efficient + network protocol which can easily be supported in a wide variety of + programming languages leveraging existing library investments in HTTP. + +- Knative Eventing assumes a sender-driven (push) event delivery system. That + is, each event processor is actively responsible for an event until it is + handled (or affirmatively delivered to all following recipients). + +- Knative Eventing aims to make writing [event + sources](./overview.md#event-source) and event-processing software easier to + write; as such, it imposes higher standards on system components like + [brokers](./overview.md#broker) and [channels](./overview.md#channel) than on + edge components. + +This contract defines a mechanism for a single event sender to reliably deliver +a single event to a single recipient. Building from this primitive, chains of +reliable event delivery and event-driven applications can be built. + +## Background The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119. -## Data plane contract for Sinks +When not specified in this document, the [CloudEvents HTTP bindings, version +1](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) and +[HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be +followed (with the CloudEvents bindings preferred in the case of conflict). -A **Sink** MUST be able to handle duplicate events. +The current version of this document does not describe protocol negotiation or +the ability to upgrade an HTTP 1.1 event delivery into a more efficient protocol +such as GRPC, AMQP, or the like. It is expected that a future compatible version +of this specification may describe a protocol negotiation mechanism. -A **Sink** is an [_addressable_](./interfaces.md#addressable) resource that -takes responsibility for the event. A Sink could be a consumer of events, or -middleware. A Sink MUST be able to receive CloudEvents over HTTP and HTTPS. +## Event Delivery -A **Sink** MAY be [_callable_](./interfaces.md#callable) resource that -represents an Addressable endpoint which receives an event as input and -optionally returns an event to forward downstream. +To provide simpler support for event sources which may be translating events +from existing systems, some data plane requirements for senders are relaxed in +the general case. In the case of Knative Eventing provided resources (Channels +and Brokers) which implement these roles, requirements may be increased from +SHOULD to MUST. These cases are called out as they occur. -Almost every component in Knative Eventing may be a Sink providing -composability. +### Minimum supported protocol -Every Sink MUST support HTTP Protocol Binding for CloudEvents -[version 1.0](https://github.com/cloudevents/spec/blob/v1.0/http-protocol-binding.md) +All senders and recipients MUST support the CloudEvents 1.0 protocol and the +[binary](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#31-binary-content-mode) and -[version 0.3](https://github.com/cloudevents/spec/blob/v0.3/http-transport-binding.md) -with restrictions and extensions specified below. - -### HTTP Support - -This section adds restrictions on -[requirements in HTTP Protocol Binding for CloudEvents](https://github.com/cloudevents/spec/blob/v1.0/http-protocol-binding.md#12-relation-to-http). - -Sinks MUST accept HTTP requests with POST method and MAY support other HTTP -methods. If a method is not supported Sink MUST respond with HTTP status code -`405 Method Not Supported`. Non-event requests (e.g. health checks) are not -constrained. - -The URL used by a Sink MUST correspond to a single, unique endpoint at any given -moment in time. This MAY be done via the host, path, query string, or any -combination of these. This mapping is handled exclusively by the -[Addressable control-plane](./interfaces.md#control-plane) exposed via the -`status.address.url`. - -If an HTTP request's URL does not correspond to an existing endpoint, then the -Sink MUST respond with `404 Not Found`. - -Every non-Callable Sink MUST respond with `202 Accepted` if the request is -accepted. - -If Sink is Callable it MAY respond with `200 OK` and a single event in the HTTP -response. A returned event is not required to be related to the received event. -The Callable should return a successful response if the event was processed -successfully. If there is no event to send back then Callable Sink MUST respond -with 2xx HTTP and with empty body. - -If a Sink receives a request and is unable to parse a valid CloudEvent, then it -MUST respond with `400 Bad Request`. - -### Content Modes Supported - -A Sink MUST support `Binary Content Mode` and `Structured Content Mode` as -described in -[HTTP Message Mapping section of HTTP Protocol Binding for CloudEvents](https://github.com/cloudevents/spec/blob/master/http-protocol-binding.md#3-http-message-mapping) - -A Sink MAY support `Batched Content Mode` but that mode is not used in Knative -Eventing currently (that may change in future). - -### Retries - -Sinks should expect that retries and accept possibility that duplicate events -may be delivered. - -### Error handling - -If Sink is not returning HTTP success header (200 or 202) then the event may be -sent again. If the event can not be delivered then some sources of events (such -as Knative sources, brokers or channels) MAY support -[dead letter sink or channel](https://github.com/knative/eventing/blob/main/docs/delivery/README.md) for events that can not be -delivered. +[structured](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#32-structured-content-mode) +content modes of the CloudEvetns HTTP binding. Senders MUST support both +cleartext (`http`) and TLS (`https`) URLs as event delivery destinations. + +### HTTP Verbs + +In the absence of specific delivery preferences, the sender MUST initiate +delivery of the event to the recipient using the HTTP POST verb, using either +the structured or binary encoding of the event (sender's choice). This delivery +SHOULD be performed using the CloudEvents HTTP Binding, version 1. + +Senders MAY probe the recipient with an [HTTP OPTIONS +request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if implemented, the +recipent MUST indicate support for the POST verb using the [`Allow` +header](https://tools.ietf.org/html/rfc7231#section-7.4.1). Senders which +receive an error when probing with HTTP OPTIONS SHOULD proceed using the HTTP +POST mechanism. + +### Event Acknowledgement and Repeat Delivery + +Event recipients MUST use the HTTP response code to indicate acceptance of an +event. The recipient MUST NOT return a response accepting the event until it has +handled event (processed the event or stored it in stable storage). The +following response codes are explicitly defined; event recipients MAY also +respond with other response codes. A response code not in this table SHOULD be +treated as a retriable error. + +| Response code | Meaning | Retry | Delivery completed | Error | +| ------------- | --------------------------- | ----- | ------------------ | ----- | +| `1xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| `200` | [Event reply](#event-reply) | No | Yes | No | +| `202` | Event accepted | No | Yes | No | +| other `2xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| other `3xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| `400` | Unparsable event | No | No | Yes | +| `404` | Endpoint does not exist | No | No | Yes | +| other `4xx` | Error | Yes | No | Yes | +| other `5xx` | Error | Yes | No | Yes | + +\* Unspecified `1xx`, `2xx`, and `3xx` response codes are **reserved for future +extension**. Event recipients SHOULD NOT send these response codes in this spec +version, but event senders MUST handle these response codes as errors and +implement appropriate failure behavior. + + + +Recipients MUST be able to handle duplicate delivery of events and MUST accept +delivery of duplicate events, as the event acknowledgement could have been lost +in return to the sender. Event recipients MUST use the [`source` and `id` +attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) +to detect duplicated events (see [observability](#observability) for an example +case where other event attributes may vary from one delivery attempt to +another). + +Where possible, event senders SHOULD re-attempt delivery of events where the +HTTP request failed or returned a retriable status code. It is RECOMMENDED that +event senders implement some form of congestion control (such as exponential +backoff) when managing retry timing. This specification does not document any +specific congestion control algorithm or +parameters. [Brokers](./overview.md#broker) and +[Channels](./overview.md#channel) MUST implement congestion control and MUST +implement retries. ### Observability -CloudEvents received by Sink MAY have -[Distributed Tracing Extension Attribute](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md). - -### Event reply contract - -An event sender supporting event replies SHOULD include a `Prefer: reply` header -in delivery requests to indicate to the sink that event reply is supported. An -event sender MAY ignore an event reply in the delivery response if the -`Prefer: reply` header was not included in the delivery request. - -An example is that a Broker supporting event reply sends events with an -additional header `Prefer: reply` so that the sink connected to the Broker knows -event replies will be accepted. While a source sends events without the header, -in which case the sink may assume that any event reply will be dropped without -error or retry attempt. If a sink wishes to ensure the reply events will be -delivered, it can check for the existence of the `Prefer: reply` header in the -delivery request and respond with an error code if the header is not present. - -### Data plane contract for Sources - -See [Source Delivery specification](sources.md#source-event-delivery) -for details. - -### Data plane contract for Channels - -See [Channel Delivery specification](channel.md#data-plane) for details. - -### Data plane contract for Brokers - -See [Broker Delivery specification](broker.md) - -## Changelog +Event senders MAY add or update CloudEvents attributes before sending to +implement observability features such as tracing; in particular, the +[`traceparent` and `tracestate` distributed tracing +attributes](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) +may be modified in this way for each delivery attempt of the same event. + +This specification does not mandate any particular logging or metrics +aggregtion, nor a method of exposing observability information to users +configuring the resources. Platform administrators SHOULD expose event-delivery +telemetry to users through platform-specific interfaces, but such interfaces are +beyond the scope of this document. + + + +### Derived (Reply) Events + +In some applications, an event receiver might emit an event in reaction to a +received event. An event sender MAY document support for this pattern by +including a `Prefer: reply` header in the HTTP POST request. This header +indicates to the event receiver that the caller will accept a [`200` +response](#event-acknowledgement-and-repeat-delivery) which includes a +CloudEvent encoded using the binary or structured formats. +[Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST +indicate support for replies using the `Prefer: reply` header. + +The sender SHOULD NOT assume that a received reply event is directly related to +the event sent in the HTTP request. + +A recipient MAY reply to any HTTP POST with a `200` response to indicate that +the event was processed successfully, with or without a response payload. If the +recipient will _never_ provide a response payload, the `202` response code is +likely a better choice. + +If a recipient chooses to reply to a sender with a `200` response code and a +reply event in the absence of a `Prefer: reply` header, the sender SHOULD treat +the event as accepted, and MAY log an error about the unexpected payload. The +sender MUST NOT process the reply event if it did not advertise the `Prefer: +reply` capability. -- 2020-04-20: `0.13.x release`: initial version that documents common contract - for sinks, sources, channels and brokers. diff --git a/specs/eventing/motivation.md b/specs/eventing/motivation.md index 9d2b48653..42492f8b5 100644 --- a/specs/eventing/motivation.md +++ b/specs/eventing/motivation.md @@ -1,49 +1,33 @@ # Motivation -The goal of Knative Eventing is to define common, composable primitives to -enable late-binding event sources and event consumers. +The goal of the Knative Eventing project is to define common primitives to +enable composing event-processing applications through configuration, rather +than application code. - +Building by combining independent components provides a number of benefits for +application designers: -Knative Eventing has following principles: - -1. Services are loosely coupled during development and deployed independently on - a variety of platforms (Kubernetes, VMs, SaaS or FaaS). +1. Services are loosely coupled during development and may be deployed + independently on a variety of platforms (Kubernetes, VMs, SaaS or FaaS). This + composability allows re-use of common patterns and building blocks, even + across programming language and tooling boundaries. 1. A producer can generate events before a consumer is listening, and a consumer can express an interest in an event or class of events that is not yet being - produced. + produced. This allows event-driven applications to be evolve over time + without needing to closely coordinate changes. -1. Services can be connected to create new applications - - without modifying producer or consumer. +1. Services can be connected to create new applications: + - without modifying producer or consumer - with the ability to select a specific subset of events from a particular producer -These primitives enable producing and consuming events adhering to the -[CloudEvents Specification](https://github.com/cloudevents/spec), in a decoupled -way. - -Kubernetes has no primitives related to event processing, yet this is an -essential component in serverless workloads. Eventing introduces high-level -primitives for event production and delivery with an initial focus on push over -HTTP. If a new event source or type is required of your application, the effort -required to plumb them into the existing eventing framework will be minimal and -will integrate with CloudEvents middleware and message consumers. - -Knative eventing implements common components of an event delivery ecosystem: -enumeration and discovery of event sources, configuration and management of -event transport, and declarative binding of events (generated either by storage -services or earlier computation) to further event processing and persistence. - -The Knative Eventing API is intended to operate independently, and interoperate -well with the [Serving API](https://github.com/knative/serving) and -[Build API](https://github.com/knative/build). - ---- - -_Navigation_: +In order to enable loose coupling and late-binding of event producers and +consumers, Knative Eventing utilizes and extends the [CloudEvents +specification](https:?/github.com/cloudevents/spec) as the data plane protocol +between components. Knative Eventing prioritizes at-least-once delivery +semantics, using the CloudEvents HTTP POST (push) transport as a minimum common +transport between components. -- **Motivation and goals** -- [Resource type overview](overview.md) -- [Interface contracts](interfaces.md) -- [Object model specification](spec.md) +Knative Eventing also defines patterns to simplify the construction and usage of +event senders and recipients. diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index 3fafca4d4..e0b63de5f 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -1,97 +1,88 @@ # Resource Types -The API defines and provides a complete implementation for -[Trigger](spec.md#kind-trigger), [Broker](spec.md#kind-broker), -[Subscription](spec.md#kind-subscription) and abstract resource definitions for -[Channels](spec.md#kind-channel). +The Knative Eventing API provides primitives for two common event-processing +patterns (credit to James Urquhart for the formulation): -With extensibility and composability as a goal of Knative Eventing, the eventing -API defines several resources that can be reduced down to well understood -contracts. These eventing resource interfaces may be fulfilled by other -Kubernetes objects and then composed in the same way as the concrete objects. -The interfaces are ([Addressable](interfaces.md#addressable), -[Callable](interfaces.md#callable)). For more details, see -[Interface Contracts](interfaces.md). +* Point-to-point asynchronous communication ([`messaging.knative.dev`](#messaging)) -- A **Trigger** describes a filter on event attributes which should be delivered - to an _Addressable_. +* Content-based event routing ([`eventing.knative.dev`](#eventing)) -- A **Broker** provides a bucket of events which can be selected by attribute. +The other two patterns James identifies are log-stream processing and complex +workflows; these are not currently addressed by Knative Eventing. - +In addition to the primitives needed to express the above patterns, Knative +Eventing defines two [_interface contracts_](#interface-contracts) to allow +connecting multiple types of Kubernetes objects as event senders and recipients +to the core primitives. -![Broker Trigger Overview](images/broker-trigger-overview.svg) + -The above diagram shows a _Broker_ ingesting events and delivering to a -_Service_ only when the _Trigger_ filter matches. +## Eventing -- A **Subscription** describes the transformation of an event and optional - forwarding of a returned event. +### Broker -- A **Channel** provides event persistence and fanout of events from a - well-known input address to multiple outputs described by _Subscriptions_. +**Broker** provides a central event-routing hub which exposes a URL address +which event senders may use to submit events to the router. A Broker may be +implemented using many different underlying event-forwarding mechanisms; the +broker provides a small set of common event-delivery configuration options and +may reference additional implementation-specific configuration options via a +reference to an external object; the format of the external objects is not +standardized. - +### Trigger -![Resource Types Overview](images/resource-types-overview.svg) +**Trigger** defines a filtered delivery option to extract events delivered to a +**Broker** and route them to an **Addressable** destination. Trigger implements +uniform event filtering based on the CloudEvents attributes associated with the +event, ignoring the payload (which might be large and/or binary and need not be +parsed during event routing). The addressable interface contract allows Triggers +to deliver events to a variety of different destinations, including external +resources such as a virtual machine or SaaS service. -Channels as well as Sources are defined by independent CRDs that can be -installed into a cluster. Both Sources and Channels implementations can be -created directly. A Channel however offers also a way to create the backing -implementation as part of the generic Channel (using a _channelTemplate_). +## Messaging -## Trigger +### Channel -**Trigger** describes a registration of interest on a filter set of events -delivered to a _Broker_ which should be delivered to an _Addressable_. Events -selected by a _Trigger_ are buffered independently from other _Triggers_, even -if they deliver to the same _Addressable_. +**Channel** provides an abstract interface which may be fulfiled by several +concrete implementations of a backing asynchronous fan-out queue. The common +abstraction provided by channel allows both the composition of higher-level +constructs for chained or parallel processing of events, and the replacement of +particular messaging technologies (for example, allowing a development +environment to use a lower-reliability channel compared with the production +environment). -For more details, see [Kind: Trigger](spec.md#kind-trigger). +### Subscription -## Broker +**Subscription** defines a delivery destination for all events sent to a +**Channel**. Events sent to a channel are delivered to _each_ subscription +_independently_ -- a subscription maintains its own list of undelivered events +and will manage retry indpendently of any other subscriptions to the same +channel. Like **Trigger**, subscriptions use the **Addressable** interface +contract to support event delivery to many different destination types. -**Broker** provides an eventing mesh. This allows producers to deliver events to -a single endpoint and not need to worry about the routing details for individual -consumers. +## Interface Contracts -For more details, see [Kind: Broker](spec.md#kind-broker). +In addition to the concrete types described above in the `messaging.knative.dev` +and `eventing.knative.dev` API groups, Knative Eventing supports referencing +objects in other API groups as destinations for event delivery. This is done by +defining partial schemas which the other resources must support. The following +interface contracts define a set of expected resource fields on an referenced +resource. -## Subscription +### Addressable -**Subscriptions** describe a flow of events from one _Channel_ to the next -Channel\* through transformations (such as a Knative Service which processes -CloudEvents over HTTP). A _Subscription_ controller resolves the addresses of -transformations (`subscriber`) and destination storage (`reply`) through the -_Callable_ and _Addressable_ interface contracts, and writes the resolved -addresses to the _Channel_ in the `channel` reference. _Subscriptions_ do not -need to specify both a transformation and a storage destination, but at least -one must be provided. +**Addressable** resources expose a resource address (HTTP URL) in their `status` +object. The URL is used as a destination for delivery of events to the resource; +the exposed URL must implement the [data plane contract](data-plane.md) for +receiving events. -All event delivery linkage from a **Subscription** is 1:1 – only a single -`channel`, `subscriber`, and `reply` may be provided. +**Broker** and **Channel** both implement **Addressable**. -For more details, see [Kind: Subscription](spec.md#kind-subscription). +### Event Source -## Channel +**Event Sources** are resources which generate events and may be configured to +deliver the events to an **Addressable** resource designated by a `sink` object +in the resource's `spec`. The Knative Eventing spec does not define any specific +event sources, but does define common interfaces for discovering and managing +event sources. -**Channel** provides an event delivery mechanism which can fan out received -events to multiple destinations via _Subscriptions_. A _Channel_ has a single -inbound _Addressable_ interface which may accept events delivered directly or -forwarded from multiple _Subscriptions_. Different _Channels_ may implement -different degrees of persistence. Event delivery order is dependent on the -backing implementation of the _Channel_. - -Event selection on a _Channel_ is 1:N – a single _Channel_ may fan out to -multiple _Subscriptions_. - -See [Kind: Channel](spec.md#kind-channel). - ---- - -_Navigation_: - -- [Motivation and goals](motivation.md) -- **Resource type overview** -- [Interface contracts](interfaces.md) -- [Object model specification](spec.md) From 0a33307584c20c3e71969daccb0e35ff2c95e9b2 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Sun, 9 May 2021 14:46:02 -0700 Subject: [PATCH 02/17] Updates from n3wscott review --- specs/eventing/data-plane.md | 10 +++++----- specs/eventing/motivation.md | 2 +- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index abb5e6069..dd16b5f83 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -45,14 +45,14 @@ followed (with the CloudEvents bindings preferred in the case of conflict). The current version of this document does not describe protocol negotiation or the ability to upgrade an HTTP 1.1 event delivery into a more efficient protocol such as GRPC, AMQP, or the like. It is expected that a future compatible version -of this specification may describe a protocol negotiation mechanism. +of this specification might describe a protocol negotiation mechanism. ## Event Delivery -To provide simpler support for event sources which may be translating events +To provide simpler support for event sources which might be translating events from existing systems, some data plane requirements for senders are relaxed in the general case. In the case of Knative Eventing provided resources (Channels -and Brokers) which implement these roles, requirements may be increased from +and Brokers) which implement these roles, requirements are increased from SHOULD to MUST. These cases are called out as they occur. ### Minimum supported protocol @@ -69,7 +69,7 @@ cleartext (`http`) and TLS (`https`) URLs as event delivery destinations. In the absence of specific delivery preferences, the sender MUST initiate delivery of the event to the recipient using the HTTP POST verb, using either the structured or binary encoding of the event (sender's choice). This delivery -SHOULD be performed using the CloudEvents HTTP Binding, version 1. +SHOULD be performed using the CloudEvents HTTP Binding, version 1.0. Senders MAY probe the recipient with an [HTTP OPTIONS request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if implemented, the @@ -95,7 +95,7 @@ treated as a retriable error. | other `2xx` | (Unspecified) | Yes\* | No\* | Yes\* | | other `3xx` | (Unspecified) | Yes\* | No\* | Yes\* | | `400` | Unparsable event | No | No | Yes | -| `404` | Endpoint does not exist | No | No | Yes | +| `404` | Endpoint does not exist | Yes | No | Yes | | other `4xx` | Error | Yes | No | Yes | | other `5xx` | Error | Yes | No | Yes | diff --git a/specs/eventing/motivation.md b/specs/eventing/motivation.md index 42492f8b5..44439748c 100644 --- a/specs/eventing/motivation.md +++ b/specs/eventing/motivation.md @@ -24,7 +24,7 @@ application designers: In order to enable loose coupling and late-binding of event producers and consumers, Knative Eventing utilizes and extends the [CloudEvents -specification](https:?/github.com/cloudevents/spec) as the data plane protocol +specification](https://github.com/cloudevents/spec) as the data plane protocol between components. Knative Eventing prioritizes at-least-once delivery semantics, using the CloudEvents HTTP POST (push) transport as a minimum common transport between components. From 62169690a04675569d8950dce292b42ff59342bf Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Mon, 10 May 2021 10:35:00 -0700 Subject: [PATCH 03/17] Update data-plane language per Ville's suggestions. --- specs/eventing/data-plane.md | 67 +++++++++++++++++------------------- 1 file changed, 32 insertions(+), 35 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index dd16b5f83..8c92bffa7 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -5,8 +5,8 @@ Late-binding event senders and receivers (composing applications using configuration) only works when all event senders and recipients speak a common protocol. In order to enable wide support for senders and receivers, Knative -Eventing extends the [CloudEvents HTTP -bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) +Eventing extends the +[CloudEvents HTTP bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) with additional semantics for the following reasons: - Knative Eventing aims to enable highly-reliable event processing workflows. As @@ -20,12 +20,11 @@ with additional semantics for the following reasons: - Knative Eventing assumes a sender-driven (push) event delivery system. That is, each event processor is actively responsible for an event until it is handled (or affirmatively delivered to all following recipients). - -- Knative Eventing aims to make writing [event - sources](./overview.md#event-source) and event-processing software easier to - write; as such, it imposes higher standards on system components like - [brokers](./overview.md#broker) and [channels](./overview.md#channel) than on - edge components. +- Knative Eventing aims to make writing + [event sources](./overview.md#event-source) and event-processing software + easier to write; as such, it imposes higher standards on system components + like [brokers](./overview.md#broker) and [channels](./overview.md#channel) + than on edge components. This contract defines a mechanism for a single event sender to reliably deliver a single event to a single recipient. Building from this primitive, chains of @@ -37,9 +36,9 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC2119. -When not specified in this document, the [CloudEvents HTTP bindings, version -1](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) and -[HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be +When not specified in this document, the +[CloudEvents HTTP bindings, version 1](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) +and [HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be followed (with the CloudEvents bindings preferred in the case of conflict). The current version of this document does not describe protocol negotiation or @@ -52,8 +51,9 @@ of this specification might describe a protocol negotiation mechanism. To provide simpler support for event sources which might be translating events from existing systems, some data plane requirements for senders are relaxed in the general case. In the case of Knative Eventing provided resources (Channels -and Brokers) which implement these roles, requirements are increased from -SHOULD to MUST. These cases are called out as they occur. +and Brokers) which implement these roles, requirements are increased from SHOULD +to MUST. Exceptions with stronger requirements for Channels and Brokers are +called out as they occur. ### Minimum supported protocol @@ -61,7 +61,7 @@ All senders and recipients MUST support the CloudEvents 1.0 protocol and the [binary](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#31-binary-content-mode) and [structured](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#32-structured-content-mode) -content modes of the CloudEvetns HTTP binding. Senders MUST support both +content modes of the CloudEvents HTTP binding. Senders MUST support both cleartext (`http`) and TLS (`https`) URLs as event delivery destinations. ### HTTP Verbs @@ -71,12 +71,12 @@ delivery of the event to the recipient using the HTTP POST verb, using either the structured or binary encoding of the event (sender's choice). This delivery SHOULD be performed using the CloudEvents HTTP Binding, version 1.0. -Senders MAY probe the recipient with an [HTTP OPTIONS -request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if implemented, the -recipent MUST indicate support for the POST verb using the [`Allow` -header](https://tools.ietf.org/html/rfc7231#section-7.4.1). Senders which -receive an error when probing with HTTP OPTIONS SHOULD proceed using the HTTP -POST mechanism. +Senders MAY probe the recipient with an +[HTTP OPTIONS request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if +implemented, the recipent MUST indicate support for the POST verb using the +[`Allow` header](https://tools.ietf.org/html/rfc7231#section-7.4.1). Senders +which receive an error when probing with HTTP OPTIONS SHOULD proceed using the +HTTP POST mechanism. ### Event Acknowledgement and Repeat Delivery @@ -111,9 +111,9 @@ be retried? What about `405` (Method Not Allowed), 413 (Payload Too Large), 414 Recipients MUST be able to handle duplicate delivery of events and MUST accept delivery of duplicate events, as the event acknowledgement could have been lost -in return to the sender. Event recipients MUST use the [`source` and `id` -attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) -to detect duplicated events (see [observability](#observability) for an example +in return to the sender. Event recipients MUST use the +[`source` and `id` attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) +to determine duplicate events (see [observability](#observability) for an example case where other event attributes may vary from one delivery attempt to another). @@ -121,21 +121,19 @@ Where possible, event senders SHOULD re-attempt delivery of events where the HTTP request failed or returned a retriable status code. It is RECOMMENDED that event senders implement some form of congestion control (such as exponential backoff) when managing retry timing. This specification does not document any -specific congestion control algorithm or -parameters. [Brokers](./overview.md#broker) and -[Channels](./overview.md#channel) MUST implement congestion control and MUST -implement retries. +specific congestion control algorithm or parameters. +[Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST +implement congestion control and MUST implement retries. ### Observability Event senders MAY add or update CloudEvents attributes before sending to implement observability features such as tracing; in particular, the -[`traceparent` and `tracestate` distributed tracing -attributes](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) +[`traceparent` and `tracestate` distributed tracing attributes](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) may be modified in this way for each delivery attempt of the same event. This specification does not mandate any particular logging or metrics -aggregtion, nor a method of exposing observability information to users +aggregation, nor a method of exposing observability information to users configuring the resources. Platform administrators SHOULD expose event-delivery telemetry to users through platform-specific interfaces, but such interfaces are beyond the scope of this document. @@ -149,8 +147,8 @@ https://github.com/knative/specs/blob/main/specs/eventing/channel.md#observabili In some applications, an event receiver might emit an event in reaction to a received event. An event sender MAY document support for this pattern by including a `Prefer: reply` header in the HTTP POST request. This header -indicates to the event receiver that the caller will accept a [`200` -response](#event-acknowledgement-and-repeat-delivery) which includes a +indicates to the event receiver that the caller will accept a +[`200` response](#event-acknowledgement-and-repeat-delivery) which includes a CloudEvent encoded using the binary or structured formats. [Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST indicate support for replies using the `Prefer: reply` header. @@ -166,6 +164,5 @@ likely a better choice. If a recipient chooses to reply to a sender with a `200` response code and a reply event in the absence of a `Prefer: reply` header, the sender SHOULD treat the event as accepted, and MAY log an error about the unexpected payload. The -sender MUST NOT process the reply event if it did not advertise the `Prefer: -reply` capability. - +sender MUST NOT process the reply event if it did not advertise the +`Prefer: reply` capability. From e776f367659a9b81d740a728042ebe4bd84793c4 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Mon, 10 May 2021 11:39:13 -0700 Subject: [PATCH 04/17] Overview revisions based on vaikas feedback. --- specs/eventing/overview.md | 95 ++++++++++++++++++++++---------------- 1 file changed, 54 insertions(+), 41 deletions(-) diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index e0b63de5f..3e159d326 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -1,14 +1,17 @@ # Resource Types The Knative Eventing API provides primitives for two common event-processing -patterns (credit to James Urquhart for the formulation): +patterns: -* Point-to-point asynchronous communication ([`messaging.knative.dev`](#messaging)) +- Topology-based event routing ([`messaging.knative.dev`](#messaging)) -* Content-based event routing ([`eventing.knative.dev`](#eventing)) +- Content-based event routing ([`eventing.knative.dev`](#eventing)) -The other two patterns James identifies are log-stream processing and complex -workflows; these are not currently addressed by Knative Eventing. +Knative Eventing does not directly specify mechanisms for other event-processing +models, including multi-stage workflows, corellated request-reply, and +sequential (windowed) event processing; these models could be built using the +primitives provided by Knative, or Knative could deliver events to an external +system that implements these models. In addition to the primitives needed to express the above patterns, Knative Eventing defines two [_interface contracts_](#interface-contracts) to allow @@ -17,6 +20,42 @@ to the core primitives. +## Interface Contracts + +In addition to the concrete types described below in the `messaging.knative.dev` +and `eventing.knative.dev` API groups, Knative Eventing supports referencing +objects in other API groups as destinations for event delivery. This is done by +defining partial schemas which the other resources must support. The following +interface contracts define object fragments and partial schemas (required fields +on an arbitrary API object) which form a basis for Knative Eventing. + +### Addressable + +**Addressable** resources expose a resource address (HTTP URL) in their `status` +object. The URL is used as a destination for delivery of events to the resource; +the exposed URL must implement the [data plane contract](data-plane.md) for +receiving events. + +[**Broker**](#broker) and [**Channel**](#channel) both implement **Addressable** +to receive events from other components. + +### Destination + +**Destination** is an interface (object fragment) which is used consistently +through Knative Eventing to reference a message delivery destination. A +destination is typically a +[tagged union](https://en.wikipedia.org/wiki/Tagged_union) of different +addressing models; at a minimum, it supports a direct URL, a reference to an +**Addressable** object, or a Kubernetes Service (as a special case). + +### Event Source + +**Event Sources** are resources which generate events and may be configured to +deliver the events to a **Destiantion** designated by a `sink` object in the +resource's `spec`. The Knative Eventing spec does not define any specific event +sources, but does define common interfaces for discovering and managing event +sources. + ## Eventing ### Broker @@ -26,17 +65,18 @@ which event senders may use to submit events to the router. A Broker may be implemented using many different underlying event-forwarding mechanisms; the broker provides a small set of common event-delivery configuration options and may reference additional implementation-specific configuration options via a -reference to an external object; the format of the external objects is not +reference to an external object (either a kubernetes built-in like ConfigMap or +a custom object); the format of the external objects is intentionally not standardized. ### Trigger -**Trigger** defines a filtered delivery option to extract events delivered to a -**Broker** and route them to an **Addressable** destination. Trigger implements -uniform event filtering based on the CloudEvents attributes associated with the -event, ignoring the payload (which might be large and/or binary and need not be -parsed during event routing). The addressable interface contract allows Triggers -to deliver events to a variety of different destinations, including external +**Trigger** defines a filtered delivery option to select events delivered to a +**Broker** and route them to a **Destination**. Trigger implements uniform event +filtering based on the CloudEvents attributes associated with the event, +ignoring the payload (which might be large and/or binary and need not be parsed +during event routing). The addressable interface contract allows Triggers to +deliver events to a variety of different destinations, including external resources such as a virtual machine or SaaS service. ## Messaging @@ -57,32 +97,5 @@ environment). **Channel**. Events sent to a channel are delivered to _each_ subscription _independently_ -- a subscription maintains its own list of undelivered events and will manage retry indpendently of any other subscriptions to the same -channel. Like **Trigger**, subscriptions use the **Addressable** interface -contract to support event delivery to many different destination types. - -## Interface Contracts - -In addition to the concrete types described above in the `messaging.knative.dev` -and `eventing.knative.dev` API groups, Knative Eventing supports referencing -objects in other API groups as destinations for event delivery. This is done by -defining partial schemas which the other resources must support. The following -interface contracts define a set of expected resource fields on an referenced -resource. - -### Addressable - -**Addressable** resources expose a resource address (HTTP URL) in their `status` -object. The URL is used as a destination for delivery of events to the resource; -the exposed URL must implement the [data plane contract](data-plane.md) for -receiving events. - -**Broker** and **Channel** both implement **Addressable**. - -### Event Source - -**Event Sources** are resources which generate events and may be configured to -deliver the events to an **Addressable** resource designated by a `sink` object -in the resource's `spec`. The Knative Eventing spec does not define any specific -event sources, but does define common interfaces for discovering and managing -event sources. - +channel. Like **Trigger**, subscriptions use the **Destination** interface to +support event delivery to many different destination types. From 5d0c5b4f476a179a1a6ede3abaffb34dd27a0fd1 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Mon, 10 May 2021 16:58:32 -0700 Subject: [PATCH 05/17] Updates from comments in #25 --- specs/eventing/data-plane.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 8c92bffa7..4eedd6c64 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -37,7 +37,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", interpreted as described in RFC2119. When not specified in this document, the -[CloudEvents HTTP bindings, version 1](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) +[CloudEvents HTTP bindings, version 1.0](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) and [HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be followed (with the CloudEvents bindings preferred in the case of conflict). @@ -81,7 +81,7 @@ HTTP POST mechanism. ### Event Acknowledgement and Repeat Delivery Event recipients MUST use the HTTP response code to indicate acceptance of an -event. The recipient MUST NOT return a response accepting the event until it has +event. The recipient SHOULD NOT return a response accepting the event until it has handled event (processed the event or stored it in stable storage). The following response codes are explicitly defined; event recipients MAY also respond with other response codes. A response code not in this table SHOULD be From 0a54751d2fab0c4251b5e10caa5e97250beaa663 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Thu, 13 May 2021 12:06:12 -0700 Subject: [PATCH 06/17] Updates from #25 review. --- specs/eventing/data-plane.md | 19 +++++++++++-------- specs/eventing/overview.md | 17 ++++++++--------- 2 files changed, 19 insertions(+), 17 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 4eedd6c64..24ecc1bc3 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -61,8 +61,11 @@ All senders and recipients MUST support the CloudEvents 1.0 protocol and the [binary](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#31-binary-content-mode) and [structured](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#32-structured-content-mode) -content modes of the CloudEvents HTTP binding. Senders MUST support both -cleartext (`http`) and TLS (`https`) URLs as event delivery destinations. +content modes of the CloudEvents HTTP binding. Senders which do not advertise +the ability t o accept [reply events](#derived-reply-events) MAY implement only +one content mode, as the recipient is not allowed to negotiate the content mode. +Senders MUST support both cleartext (`http`) and TLS (`https`) URLs as event +delivery destinations. ### HTTP Verbs @@ -73,7 +76,7 @@ SHOULD be performed using the CloudEvents HTTP Binding, version 1.0. Senders MAY probe the recipient with an [HTTP OPTIONS request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if -implemented, the recipent MUST indicate support for the POST verb using the +implemented, the recipient MUST indicate support for the POST verb using the [`Allow` header](https://tools.ietf.org/html/rfc7231#section-7.4.1). Senders which receive an error when probing with HTTP OPTIONS SHOULD proceed using the HTTP POST mechanism. @@ -81,8 +84,8 @@ HTTP POST mechanism. ### Event Acknowledgement and Repeat Delivery Event recipients MUST use the HTTP response code to indicate acceptance of an -event. The recipient SHOULD NOT return a response accepting the event until it has -handled event (processed the event or stored it in stable storage). The +event. The recipient SHOULD NOT return a response accepting the event until it +has handled event (processed the event or stored it in stable storage). The following response codes are explicitly defined; event recipients MAY also respond with other response codes. A response code not in this table SHOULD be treated as a retriable error. @@ -113,12 +116,12 @@ Recipients MUST be able to handle duplicate delivery of events and MUST accept delivery of duplicate events, as the event acknowledgement could have been lost in return to the sender. Event recipients MUST use the [`source` and `id` attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) -to determine duplicate events (see [observability](#observability) for an example -case where other event attributes may vary from one delivery attempt to +to determine duplicate events (see [observability](#observability) for an +example case where other event attributes may vary from one delivery attempt to another). Where possible, event senders SHOULD re-attempt delivery of events where the -HTTP request failed or returned a retriable status code. It is RECOMMENDED that +HTTP request failed or returned a retryable status code. It is RECOMMENDED that event senders implement some form of congestion control (such as exponential backoff) when managing retry timing. This specification does not document any specific congestion control algorithm or parameters. diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index 3e159d326..476cb5148 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -8,7 +8,7 @@ patterns: - Content-based event routing ([`eventing.knative.dev`](#eventing)) Knative Eventing does not directly specify mechanisms for other event-processing -models, including multi-stage workflows, corellated request-reply, and +models, including multi-stage workflows, correlated request-reply, and sequential (windowed) event processing; these models could be built using the primitives provided by Knative, or Knative could deliver events to an external system that implements these models. @@ -42,16 +42,15 @@ to receive events from other components. ### Destination **Destination** is an interface (object fragment) which is used consistently -through Knative Eventing to reference a message delivery destination. A -destination is typically a -[tagged union](https://en.wikipedia.org/wiki/Tagged_union) of different -addressing models; at a minimum, it supports a direct URL, a reference to an -**Addressable** object, or a Kubernetes Service (as a special case). +through Knative Eventing to reference a event delivery destination. A +Destination eventually resolves the supplied information to a URL, and may be a +simple URL or relative to an **Addressable** object reference; it also supports +a Kubernetes Service object reference (as a special case). ### Event Source **Event Sources** are resources which generate events and may be configured to -deliver the events to a **Destiantion** designated by a `sink` object in the +deliver the events to a **Destination** designated by a `sink` object in the resource's `spec`. The Knative Eventing spec does not define any specific event sources, but does define common interfaces for discovering and managing event sources. @@ -83,7 +82,7 @@ resources such as a virtual machine or SaaS service. ### Channel -**Channel** provides an abstract interface which may be fulfiled by several +**Channel** provides an abstract interface which may be fulfilled by several concrete implementations of a backing asynchronous fan-out queue. The common abstraction provided by channel allows both the composition of higher-level constructs for chained or parallel processing of events, and the replacement of @@ -96,6 +95,6 @@ environment). **Subscription** defines a delivery destination for all events sent to a **Channel**. Events sent to a channel are delivered to _each_ subscription _independently_ -- a subscription maintains its own list of undelivered events -and will manage retry indpendently of any other subscriptions to the same +and will manage retry independently of any other subscriptions to the same channel. Like **Trigger**, subscriptions use the **Destination** interface to support event delivery to many different destination types. From 07fcbdecf89ca6eaa5e78d7c45975d60a593a279 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Wed, 19 May 2021 14:03:02 -0700 Subject: [PATCH 07/17] Update with matezw / slinkydeveloper feedback --- specs/eventing/data-plane.md | 57 ++++++++++++++++++------------------ specs/eventing/overview.md | 9 +++--- 2 files changed, 34 insertions(+), 32 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 24ecc1bc3..95d36f973 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -9,17 +9,20 @@ Eventing extends the [CloudEvents HTTP bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) with additional semantics for the following reasons: -- Knative Eventing aims to enable highly-reliable event processing workflows. As - such, it prefers duplicate delivery to discarded events. The CloudEvents spec - does not take a stance here. +- Knative Eventing aims to enable at least once event processing; hence it + prefers duplicate delivery to discarded events. The CloudEvents spec does not + take a stance here. - The CloudEvents HTTP bindings provide a relatively simple and efficient network protocol which can easily be supported in a wide variety of - programming languages leveraging existing library investments in HTTP. + programming languages leveraging existing library investments in HTTP. The + CloudEvents project has already written these libraries for many popular + languages. - Knative Eventing assumes a sender-driven (push) event delivery system. That is, each event processor is actively responsible for an event until it is handled (or affirmatively delivered to all following recipients). + - Knative Eventing aims to make writing [event sources](./overview.md#event-source) and event-processing software easier to write; as such, it imposes higher standards on system components @@ -62,7 +65,7 @@ All senders and recipients MUST support the CloudEvents 1.0 protocol and the and [structured](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#32-structured-content-mode) content modes of the CloudEvents HTTP binding. Senders which do not advertise -the ability t o accept [reply events](#derived-reply-events) MAY implement only +the ability to accept [reply events](#derived-reply-events) MAY implement only one content mode, as the recipient is not allowed to negotiate the content mode. Senders MUST support both cleartext (`http`) and TLS (`https`) URLs as event delivery destinations. @@ -81,7 +84,7 @@ implemented, the recipient MUST indicate support for the POST verb using the which receive an error when probing with HTTP OPTIONS SHOULD proceed using the HTTP POST mechanism. -### Event Acknowledgement and Repeat Delivery +### Event Acknowledgement and Delivery Retry Event recipients MUST use the HTTP response code to indicate acceptance of an event. The recipient SHOULD NOT return a response accepting the event until it @@ -90,28 +93,25 @@ following response codes are explicitly defined; event recipients MAY also respond with other response codes. A response code not in this table SHOULD be treated as a retriable error. -| Response code | Meaning | Retry | Delivery completed | Error | -| ------------- | --------------------------- | ----- | ------------------ | ----- | -| `1xx` | (Unspecified) | Yes\* | No\* | Yes\* | -| `200` | [Event reply](#event-reply) | No | Yes | No | -| `202` | Event accepted | No | Yes | No | -| other `2xx` | (Unspecified) | Yes\* | No\* | Yes\* | -| other `3xx` | (Unspecified) | Yes\* | No\* | Yes\* | -| `400` | Unparsable event | No | No | Yes | -| `404` | Endpoint does not exist | Yes | No | Yes | -| other `4xx` | Error | Yes | No | Yes | -| other `5xx` | Error | Yes | No | Yes | +| Response code | Meaning | Retry | Delivery completed | Error | +| ------------- | --------------------------------- | ----- | ------------------ | ----- | +| `1xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| `200` | [Event reply](#event-reply) | No | Yes | No | +| `202` | Event accepted | No | Yes | No | +| other `2xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| other `3xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| `400` | Unparsable event | No | No | Yes | +| `404` | Endpoint does not exist | Yes | No | Yes | +| `409` | Conflict / Processing in progress | Yes | No | Yes | +| `429` | Too Many Requests / Overloaded | Yes | No | Yes | +| other `4xx` | Error | No | No | Yes | +| other `5xx` | Error | Yes | No | Yes | \* Unspecified `1xx`, `2xx`, and `3xx` response codes are **reserved for future extension**. Event recipients SHOULD NOT send these response codes in this spec version, but event senders MUST handle these response codes as errors and implement appropriate failure behavior. - - Recipients MUST be able to handle duplicate delivery of events and MUST accept delivery of duplicate events, as the event acknowledgement could have been lost in return to the sender. Event recipients MUST use the @@ -148,17 +148,18 @@ https://github.com/knative/specs/blob/main/specs/eventing/channel.md#observabili ### Derived (Reply) Events In some applications, an event receiver might emit an event in reaction to a -received event. An event sender MAY document support for this pattern by -including a `Prefer: reply` header in the HTTP POST request. This header -indicates to the event receiver that the caller will accept a +received event. Components MAY to choose to support this pattern by accepting an +encoded CloudEvent in the HTTP response. The sender SHOULD NOT assume that a +received reply event is directly related to the event sent in the HTTP request. + +An event sender MAY document support for this pattern by including a +`Prefer: reply` header in the HTTP POST request. This header indicates to the +event receiver that the caller will accept a [`200` response](#event-acknowledgement-and-repeat-delivery) which includes a CloudEvent encoded using the binary or structured formats. [Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST indicate support for replies using the `Prefer: reply` header. -The sender SHOULD NOT assume that a received reply event is directly related to -the event sent in the HTTP request. - A recipient MAY reply to any HTTP POST with a `200` response to indicate that the event was processed successfully, with or without a response payload. If the recipient will _never_ provide a response payload, the `202` response code is diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index 476cb5148..112d063d4 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -45,7 +45,9 @@ to receive events from other components. through Knative Eventing to reference a event delivery destination. A Destination eventually resolves the supplied information to a URL, and may be a simple URL or relative to an **Addressable** object reference; it also supports -a Kubernetes Service object reference (as a special case). +a Kubernetes Service object reference (as a special case). An absolute URL in a +Destination may be used to reference cluster-external resources such as a +virtual machine or SaaS service. ### Event Source @@ -74,9 +76,8 @@ standardized. **Broker** and route them to a **Destination**. Trigger implements uniform event filtering based on the CloudEvents attributes associated with the event, ignoring the payload (which might be large and/or binary and need not be parsed -during event routing). The addressable interface contract allows Triggers to -deliver events to a variety of different destinations, including external -resources such as a virtual machine or SaaS service. +during event routing). The destination interface contract allows Triggers to +deliver events both cluster-local objects or external resources. ## Messaging From 917eac3ae902c3702885a895ab57f6e8cfc3b71b Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Thu, 20 May 2021 12:27:07 -0700 Subject: [PATCH 08/17] Updates from slinky, travis, ville, matzew --- specs/eventing/data-plane.md | 33 ++++++++++++++++++++------------- specs/eventing/overview.md | 4 ++-- 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 95d36f973..b54d1105b 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -45,9 +45,10 @@ and [HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be followed (with the CloudEvents bindings preferred in the case of conflict). The current version of this document does not describe protocol negotiation or -the ability to upgrade an HTTP 1.1 event delivery into a more efficient protocol -such as GRPC, AMQP, or the like. It is expected that a future compatible version -of this specification might describe a protocol negotiation mechanism. +any delivery mechanism other than HTTP 1.1. Future versions may define protocol +negotiation to optimize delivery; compliant implementations SHOULD aim to +interoperate by ignoring unrecognized negotiation options (such as HTTP Upgrade +headers). ## Event Delivery @@ -95,11 +96,11 @@ treated as a retriable error. | Response code | Meaning | Retry | Delivery completed | Error | | ------------- | --------------------------------- | ----- | ------------------ | ----- | -| `1xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| `1xx` | (Unspecified) | No\* | No\* | Yes\* | | `200` | [Event reply](#event-reply) | No | Yes | No | | `202` | Event accepted | No | Yes | No | -| other `2xx` | (Unspecified) | Yes\* | No\* | Yes\* | -| other `3xx` | (Unspecified) | Yes\* | No\* | Yes\* | +| other `2xx` | (Unspecified) | No\* | No\* | Yes\* | +| other `3xx` | (Unspecified) | No\* | No\* | Yes\* | | `400` | Unparsable event | No | No | Yes | | `404` | Endpoint does not exist | Yes | No | Yes | | `409` | Conflict / Processing in progress | Yes | No | Yes | @@ -112,13 +113,17 @@ extension**. Event recipients SHOULD NOT send these response codes in this spec version, but event senders MUST handle these response codes as errors and implement appropriate failure behavior. -Recipients MUST be able to handle duplicate delivery of events and MUST accept -delivery of duplicate events, as the event acknowledgement could have been lost -in return to the sender. Event recipients MUST use the +Recipients MUST be able to handle duplicate delivery of events (for example, via +idempotency or tracking event delivery state) and MUST accept delivery of +duplicate events, as the event acknowledgement could have been lost in return to +the sender. As specified in the +[CloudEvents specification](https://github.com/cloudevents/spec/blob/v1.0.1/primer.md#id), +event recipients MUST use the [`source` and `id` attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) -to determine duplicate events (see [observability](#observability) for an -example case where other event attributes may vary from one delivery attempt to -another). +to determine duplicate events if needed. This specification does not describe +state requirements for clients which need to detect duplicate events. (see +[observability](#observability) for an example case where other event attributes +may vary from one delivery attempt to another). Where possible, event senders SHOULD re-attempt delivery of events where the HTTP request failed or returned a retryable status code. It is RECOMMENDED that @@ -132,7 +137,9 @@ implement congestion control and MUST implement retries. Event senders MAY add or update CloudEvents attributes before sending to implement observability features such as tracing; in particular, the -[`traceparent` and `tracestate` distributed tracing attributes](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) +`traceparent` and `tracestate` distributed tracing attributes defined by +[W3C](https://www.w3.org/TR/trace-context/) and +[CloudEvents](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) may be modified in this way for each delivery attempt of the same event. This specification does not mandate any particular logging or metrics diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index 112d063d4..f3cc4b689 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -26,7 +26,7 @@ In addition to the concrete types described below in the `messaging.knative.dev` and `eventing.knative.dev` API groups, Knative Eventing supports referencing objects in other API groups as destinations for event delivery. This is done by defining partial schemas which the other resources must support. The following -interface contracts define object fragments and partial schemas (required fields +interface contracts define resource fragments and partial schemas (required fields on an arbitrary API object) which form a basis for Knative Eventing. ### Addressable @@ -41,7 +41,7 @@ to receive events from other components. ### Destination -**Destination** is an interface (object fragment) which is used consistently +**Destination** is an interface (resource fragment) which is used consistently through Knative Eventing to reference a event delivery destination. A Destination eventually resolves the supplied information to a URL, and may be a simple URL or relative to an **Addressable** object reference; it also supports From 7bd01619805c442665bd4e93825b1cca031317f2 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Thu, 20 May 2021 14:24:24 -0700 Subject: [PATCH 09/17] Link to upgrade header. --- specs/eventing/data-plane.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index b54d1105b..19dc4c999 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -47,8 +47,8 @@ followed (with the CloudEvents bindings preferred in the case of conflict). The current version of this document does not describe protocol negotiation or any delivery mechanism other than HTTP 1.1. Future versions may define protocol negotiation to optimize delivery; compliant implementations SHOULD aim to -interoperate by ignoring unrecognized negotiation options (such as HTTP Upgrade -headers). +interoperate by ignoring unrecognized negotiation options (such as +[HTTP `Upgrade` headers](https://datatracker.ietf.org/doc/html/rfc7230#section-6.7)). ## Event Delivery From fae7a876fb4f81993b5157ff9cd0d025b52c9805 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Tue, 25 May 2021 11:52:22 -0700 Subject: [PATCH 10/17] Finish up TODOs, fix some typos. --- specs/eventing/data-plane.md | 10 +-- specs/eventing/images/eventing-overview.svg | 87 +++++++++++++++++++++ specs/eventing/overview.md | 34 +++++++- 3 files changed, 123 insertions(+), 8 deletions(-) create mode 100644 specs/eventing/images/eventing-overview.svg diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 19dc4c999..6b906d217 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -121,7 +121,7 @@ the sender. As specified in the event recipients MUST use the [`source` and `id` attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) to determine duplicate events if needed. This specification does not describe -state requirements for clients which need to detect duplicate events. (see +state requirements for clients which need to detect duplicate events. (See [observability](#observability) for an example case where other event attributes may vary from one delivery attempt to another). @@ -140,7 +140,7 @@ implement observability features such as tracing; in particular, the `traceparent` and `tracestate` distributed tracing attributes defined by [W3C](https://www.w3.org/TR/trace-context/) and [CloudEvents](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md) -may be modified in this way for each delivery attempt of the same event. +MAY be modified in this way for each delivery attempt of the same event. This specification does not mandate any particular logging or metrics aggregation, nor a method of exposing observability information to users @@ -148,14 +148,10 @@ configuring the resources. Platform administrators SHOULD expose event-delivery telemetry to users through platform-specific interfaces, but such interfaces are beyond the scope of this document. - - ### Derived (Reply) Events In some applications, an event receiver might emit an event in reaction to a -received event. Components MAY to choose to support this pattern by accepting an +received event. Components MAY choose to support this pattern by accepting an encoded CloudEvent in the HTTP response. The sender SHOULD NOT assume that a received reply event is directly related to the event sent in the HTTP request. diff --git a/specs/eventing/images/eventing-overview.svg b/specs/eventing/images/eventing-overview.svg new file mode 100644 index 000000000..4ba72413f --- /dev/null +++ b/specs/eventing/images/eventing-overview.svg @@ -0,0 +1,87 @@ +eventing.knative.devmessaging.knative.devBrokerTriggerTriggerChannelSubscriptionSubscription«Knative Service»Addressable«Kubernetes Service»Addressablebrokerbrokersubscriberchannelchannelsubscriberreply \ No newline at end of file diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index f3cc4b689..f21fbe83f 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -18,7 +18,39 @@ Eventing defines two [_interface contracts_](#interface-contracts) to allow connecting multiple types of Kubernetes objects as event senders and recipients to the core primitives. - +![Overview of objects](images/eventing-overview.svg) + ## Interface Contracts From 9e33332dd3117a79ceb24081829aaa509c602b49 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Tue, 25 May 2021 15:05:55 -0700 Subject: [PATCH 11/17] Address a number of duglin comments --- specs/eventing/data-plane.md | 76 ++++++++++++++++++++---------------- specs/eventing/motivation.md | 2 +- specs/eventing/overview.md | 2 +- 3 files changed, 44 insertions(+), 36 deletions(-) diff --git a/specs/eventing/data-plane.md b/specs/eventing/data-plane.md index 6b906d217..d69dcfb8d 100644 --- a/specs/eventing/data-plane.md +++ b/specs/eventing/data-plane.md @@ -1,10 +1,23 @@ # Knative Eventing Data Plane Contract +## Terminology + +This document discusses communication between two parties: + +- **Event Senders** initiate an HTTP POST to deliver a CloudEvent. +- **Event Recipients** receive an HTTP POST and accept (or reject) a CloudEvent. + +Additionally, these roles may be combined in different ways: + +- **Event Processors** may be event senders, event recipients, or both. +- **Event Sources** are exclusively event senders, and never act as recipients. +- **Event Sinks** are exclusively event recipients, and never act as senders. + ## Introduction -Late-binding event senders and receivers (composing applications using +Late-binding event senders and recipients (composing applications using configuration) only works when all event senders and recipients speak a common -protocol. In order to enable wide support for senders and receivers, Knative +protocol. In order to enable wide support for senders and recipients, Knative Eventing extends the [CloudEvents HTTP bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) with additional semantics for the following reasons: @@ -20,14 +33,13 @@ with additional semantics for the following reasons: languages. - Knative Eventing assumes a sender-driven (push) event delivery system. That - is, each event processor is actively responsible for an event until it is - handled (or affirmatively delivered to all following recipients). + is, each recipient is actively responsible for an event until it is handled + (or affirmatively delivered to all following recipients). -- Knative Eventing aims to make writing - [event sources](./overview.md#event-source) and event-processing software - easier to write; as such, it imposes higher standards on system components - like [brokers](./overview.md#broker) and [channels](./overview.md#channel) - than on edge components. +- Knative Eventing aims to make [event sources](./overview.md#event-source) and + event-processing software easier to write; as such, it imposes higher + standards on system components like [brokers](./overview.md#broker) and + [channels](./overview.md#channel) than on edge components. This contract defines a mechanism for a single event sender to reliably deliver a single event to a single recipient. Building from this primitive, chains of @@ -42,23 +54,17 @@ interpreted as described in RFC2119. When not specified in this document, the [CloudEvents HTTP bindings, version 1.0](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) and [HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be -followed (with the CloudEvents bindings preferred in the case of conflict). +followed (with the CloudEvents bindings taking precedence in the case of +conflict). The current version of this document does not describe protocol negotiation or -any delivery mechanism other than HTTP 1.1. Future versions may define protocol -negotiation to optimize delivery; compliant implementations SHOULD aim to -interoperate by ignoring unrecognized negotiation options (such as +any delivery mechanism other than HTTP 1.1. Future versions might define +protocol negotiation to optimize delivery; compliant implementations SHOULD aim +to interoperate by ignoring unrecognized negotiation options (such as [HTTP `Upgrade` headers](https://datatracker.ietf.org/doc/html/rfc7230#section-6.7)). ## Event Delivery -To provide simpler support for event sources which might be translating events -from existing systems, some data plane requirements for senders are relaxed in -the general case. In the case of Knative Eventing provided resources (Channels -and Brokers) which implement these roles, requirements are increased from SHOULD -to MUST. Exceptions with stronger requirements for Channels and Brokers are -called out as they occur. - ### Minimum supported protocol All senders and recipients MUST support the CloudEvents 1.0 protocol and the @@ -89,7 +95,7 @@ HTTP POST mechanism. Event recipients MUST use the HTTP response code to indicate acceptance of an event. The recipient SHOULD NOT return a response accepting the event until it -has handled event (processed the event or stored it in stable storage). The +has handled the event (processed the event or stored it in stable storage). The following response codes are explicitly defined; event recipients MAY also respond with other response codes. A response code not in this table SHOULD be treated as a retriable error. @@ -100,13 +106,13 @@ treated as a retriable error. | `200` | [Event reply](#event-reply) | No | Yes | No | | `202` | Event accepted | No | Yes | No | | other `2xx` | (Unspecified) | No\* | No\* | Yes\* | -| other `3xx` | (Unspecified) | No\* | No\* | Yes\* | +| `3xx` | (Unspecified) | No\* | No\* | Yes\* | | `400` | Unparsable event | No | No | Yes | | `404` | Endpoint does not exist | Yes | No | Yes | | `409` | Conflict / Processing in progress | Yes | No | Yes | | `429` | Too Many Requests / Overloaded | Yes | No | Yes | | other `4xx` | Error | No | No | Yes | -| other `5xx` | Error | Yes | No | Yes | +| `5xx` | Error | Yes | No | Yes | \* Unspecified `1xx`, `2xx`, and `3xx` response codes are **reserved for future extension**. Event recipients SHOULD NOT send these response codes in this spec @@ -121,17 +127,19 @@ the sender. As specified in the event recipients MUST use the [`source` and `id` attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes) to determine duplicate events if needed. This specification does not describe -state requirements for clients which need to detect duplicate events. (See +state requirements for recipients which need to detect duplicate events. (See [observability](#observability) for an example case where other event attributes may vary from one delivery attempt to another). Where possible, event senders SHOULD re-attempt delivery of events where the HTTP request failed or returned a retryable status code. It is RECOMMENDED that event senders implement some form of congestion control (such as exponential -backoff) when managing retry timing. This specification does not document any -specific congestion control algorithm or parameters. -[Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST -implement congestion control and MUST implement retries. +backoff) and delivery throttling when managing retry timing. Congestion control +MAY cause event delivery to fail or retry attempts to be skipped. This +specification does not document any specific congestion control algorithm or +parameters. [Brokers](./overview.md#broker) and +[Channels](./overview.md#channel) MUST implement congestion control and MUST +implement retries. ### Observability @@ -150,14 +158,14 @@ beyond the scope of this document. ### Derived (Reply) Events -In some applications, an event receiver might emit an event in reaction to a +In some applications, an event recipient MAY emit an event in reaction to a received event. Components MAY choose to support this pattern by accepting an encoded CloudEvent in the HTTP response. The sender SHOULD NOT assume that a received reply event is directly related to the event sent in the HTTP request. An event sender MAY document support for this pattern by including a `Prefer: reply` header in the HTTP POST request. This header indicates to the -event receiver that the caller will accept a +event recipient that the caller will accept a [`200` response](#event-acknowledgement-and-repeat-delivery) which includes a CloudEvent encoded using the binary or structured formats. [Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST @@ -169,7 +177,7 @@ recipient will _never_ provide a response payload, the `202` response code is likely a better choice. If a recipient chooses to reply to a sender with a `200` response code and a -reply event in the absence of a `Prefer: reply` header, the sender SHOULD treat -the event as accepted, and MAY log an error about the unexpected payload. The -sender MUST NOT process the reply event if it did not advertise the -`Prefer: reply` capability. +reply event in the absence of a `Prefer: reply` header from the sender, the +sender SHOULD treat the event as accepted, and MAY log an error about the +unexpected payload. The sender MUST NOT process the reply event if the sender +did not advertise the `Prefer: reply` capability. diff --git a/specs/eventing/motivation.md b/specs/eventing/motivation.md index 44439748c..961a6b57e 100644 --- a/specs/eventing/motivation.md +++ b/specs/eventing/motivation.md @@ -30,4 +30,4 @@ semantics, using the CloudEvents HTTP POST (push) transport as a minimum common transport between components. Knative Eventing also defines patterns to simplify the construction and usage of -event senders and recipients. +event producers and consumers. diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index f21fbe83f..1007d7637 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -63,7 +63,7 @@ on an arbitrary API object) which form a basis for Knative Eventing. ### Addressable -**Addressable** resources expose a resource address (HTTP URL) in their `status` +**Addressable** resources expose a resource `address` (HTTP URL) in their `status` object. The URL is used as a destination for delivery of events to the resource; the exposed URL must implement the [data plane contract](data-plane.md) for receiving events. From ea4f14c6f727b4870258f300480bf10dd6e71a43 Mon Sep 17 00:00:00 2001 From: Evan Anderson Date: Wed, 26 May 2021 14:54:59 -0700 Subject: [PATCH 12/17] Update overview with Doug's suggestions. --- specs/eventing/overview.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/specs/eventing/overview.md b/specs/eventing/overview.md index 1007d7637..7a8c7301f 100644 --- a/specs/eventing/overview.md +++ b/specs/eventing/overview.md @@ -5,8 +5,19 @@ patterns: - Topology-based event routing ([`messaging.knative.dev`](#messaging)) + Events are routed based on _connections between objects_ (in particular, + events flow along a [channel](#channel) to all + [subscriptions](#subscription)). This model can be thought of as "event + plumbing" in that events are managed like flows of water through pipes. + - Content-based event routing ([`eventing.knative.dev`](#eventing)) + Events are selected for routing based on the event _attributes_ rather than + primarily by object connections (a [broker](#broker) provides a stream of + events which can be selected by a [trigger](#trigger)). This model is more + akin to picking parts off a conveyor belt, where each event is considered + separately for processing. + Knative Eventing does not directly specify mechanisms for other event-processing models, including multi-stage workflows, correlated request-reply, and sequential (windowed) event processing; these models could be built using the @@ -19,6 +30,7 @@ connecting multiple types of Kubernetes objects as event senders and recipients to the core primitives. ![Overview of objects](images/eventing-overview.svg) +