Update (rewrite) overview, motivation, and data plane

specs/eventing/data-plane.md

@@ -1,125 +1,171 @@
-# Knative Eventing Data Plane Contracts
+# Knative Eventing Data Plane Contract
 
 ## Introduction
 
-Developers using Knative Eventing need to know what is supported for delivery to
-user provided components that receive events. Knative Eventing defines contract
-for data plane components and we have listed them here.
-
-## Conformance
+Late-binding event senders and receivers (composing applications using
+configuration) only works when all event senders and recipients speak a common
+protocol. In order to enable wide support for senders and receivers, Knative
+Eventing extends the [CloudEvents HTTP
+bindings](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md)
+with additional semantics for the following reasons:
+
+- Knative Eventing aims to enable highly-reliable event processing workflows. As
+  such, it prefers duplicate delivery to discarded events. The CloudEvents spec
+  does not take a stance here.
+
+- The CloudEvents HTTP bindings provide a relatively simple and efficient
+  network protocol which can easily be supported in a wide variety of
+  programming languages leveraging existing library investments in HTTP.
+
+- Knative Eventing assumes a sender-driven (push) event delivery system. That
+  is, each event processor is actively responsible for an event until it is
+  handled (or affirmatively delivered to all following recipients).
+
+- Knative Eventing aims to make writing [event
+  sources](./overview.md#event-source) and event-processing software easier to
+  write; as such, it imposes higher standards on system components like
+  [brokers](./overview.md#broker) and [channels](./overview.md#channel) than on
+  edge components.
+
+This contract defines a mechanism for a single event sender to reliably deliver
+a single event to a single recipient. Building from this primitive, chains of
+reliable event delivery and event-driven applications can be built.
+
+## Background
 
 The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
 "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be
 interpreted as described in RFC2119.
 
-## Data plane contract for Sinks
+When not specified in this document, the [CloudEvents HTTP bindings, version
+1](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md) and
+[HTTP 1.1 protocol](https://tools.ietf.org/html/rfc7230) standards should be
+followed (with the CloudEvents bindings preferred in the case of conflict).
 
-A **Sink** MUST be able to handle duplicate events.
+The current version of this document does not describe protocol negotiation or
+the ability to upgrade an HTTP 1.1 event delivery into a more efficient protocol
+such as GRPC, AMQP, or the like. It is expected that a future compatible version
+of this specification might describe a protocol negotiation mechanism.
 
-A **Sink** is an [_addressable_](./interfaces.md#addressable) resource that
-takes responsibility for the event. A Sink could be a consumer of events, or
-middleware. A Sink MUST be able to receive CloudEvents over HTTP and HTTPS.
+## Event Delivery
 
-A **Sink** MAY be [_callable_](./interfaces.md#callable) resource that
-represents an Addressable endpoint which receives an event as input and
-optionally returns an event to forward downstream.
+To provide simpler support for event sources which might be translating events
+from existing systems, some data plane requirements for senders are relaxed in
+the general case. In the case of Knative Eventing provided resources (Channels
+and Brokers) which implement these roles, requirements are increased from
+SHOULD to MUST. These cases are called out as they occur.
 
-Almost every component in Knative Eventing may be a Sink providing
-composability.
+### Minimum supported protocol
 
-Every Sink MUST support HTTP Protocol Binding for CloudEvents
-[version 1.0](https://github.com/cloudevents/spec/blob/v1.0/http-protocol-binding.md)
+All senders and recipients MUST support the CloudEvents 1.0 protocol and the
+[binary](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#31-binary-content-mode)
 and
-[version 0.3](https://github.com/cloudevents/spec/blob/v0.3/http-transport-binding.md)
-with restrictions and extensions specified below.
-
-### HTTP Support
-
-This section adds restrictions on
-[requirements in HTTP Protocol Binding for CloudEvents](https://github.com/cloudevents/spec/blob/v1.0/http-protocol-binding.md#12-relation-to-http).
-
-Sinks MUST accept HTTP requests with POST method and MAY support other HTTP
-methods. If a method is not supported Sink MUST respond with HTTP status code
-`405 Method Not Supported`. Non-event requests (e.g. health checks) are not
-constrained.
-
-The URL used by a Sink MUST correspond to a single, unique endpoint at any given
-moment in time. This MAY be done via the host, path, query string, or any
-combination of these. This mapping is handled exclusively by the
-[Addressable control-plane](./interfaces.md#control-plane) exposed via the
-`status.address.url`.
-
-If an HTTP request's URL does not correspond to an existing endpoint, then the
-Sink MUST respond with `404 Not Found`.
-
-Every non-Callable Sink MUST respond with `202 Accepted` if the request is
-accepted.
-
-If Sink is Callable it MAY respond with `200 OK` and a single event in the HTTP
-response. A returned event is not required to be related to the received event.
-The Callable should return a successful response if the event was processed
-successfully. If there is no event to send back then Callable Sink MUST respond
-with 2xx HTTP and with empty body.
-
-If a Sink receives a request and is unable to parse a valid CloudEvent, then it
-MUST respond with `400 Bad Request`.
-
-### Content Modes Supported
-
-A Sink MUST support `Binary Content Mode` and `Structured Content Mode` as
-described in
-[HTTP Message Mapping section of HTTP Protocol Binding for CloudEvents](https://github.com/cloudevents/spec/blob/master/http-protocol-binding.md#3-http-message-mapping)
-
-A Sink MAY support `Batched Content Mode` but that mode is not used in Knative
-Eventing currently (that may change in future).
-
-### Retries
-
-Sinks should expect that retries and accept possibility that duplicate events
-may be delivered.
-
-### Error handling
-
-If Sink is not returning HTTP success header (200 or 202) then the event may be
-sent again. If the event can not be delivered then some sources of events (such
-as Knative sources, brokers or channels) MAY support
-[dead letter sink or channel](https://github.com/knative/eventing/blob/main/docs/delivery/README.md) for events that can not be
-delivered.
+[structured](https://github.com/cloudevents/spec/blob/v1.0.1/http-protocol-binding.md#32-structured-content-mode)
+content modes of the CloudEvetns HTTP binding. Senders MUST support both
+cleartext (`http`) and TLS (`https`) URLs as event delivery destinations.
+
+### HTTP Verbs
+
+In the absence of specific delivery preferences, the sender MUST initiate
+delivery of the event to the recipient using the HTTP POST verb, using either
+the structured or binary encoding of the event (sender's choice). This delivery
+SHOULD be performed using the CloudEvents HTTP Binding, version 1.0.
+
+Senders MAY probe the recipient with an [HTTP OPTIONS
+request](https://tools.ietf.org/html/rfc7231#section-4.3.7); if implemented, the
+recipent MUST indicate support for the POST verb using the [`Allow`
+header](https://tools.ietf.org/html/rfc7231#section-7.4.1). Senders which
+receive an error when probing with HTTP OPTIONS SHOULD proceed using the HTTP
+POST mechanism.
+
+### Event Acknowledgement and Repeat Delivery
+
+Event recipients MUST use the HTTP response code to indicate acceptance of an
+event. The recipient MUST NOT return a response accepting the event until it has
+handled event (processed the event or stored it in stable storage). The
+following response codes are explicitly defined; event recipients MAY also
+respond with other response codes. A response code not in this table SHOULD be
+treated as a retriable error.
+
+| Response code | Meaning                     | Retry | Delivery completed | Error |
+| ------------- | --------------------------- | ----- | ------------------ | ----- |
+| `1xx`         | (Unspecified)               | Yes\* | No\*               | Yes\* |
+| `200`         | [Event reply](#event-reply) | No    | Yes                | No    |
+| `202`         | Event accepted              | No    | Yes                | No    |
+| other `2xx`   | (Unspecified)               | Yes\* | No\*               | Yes\* |
+| other `3xx`   | (Unspecified)               | Yes\* | No\*               | Yes\* |
+| `400`         | Unparsable event            | No    | No                 | Yes   |
+| `404`         | Endpoint does not exist     | Yes   | No                 | Yes   |
+| other `4xx`   | Error                       | Yes   | No                 | Yes   |
+| other `5xx`   | Error                       | Yes   | No                 | Yes   |
+
+\* Unspecified `1xx`, `2xx`, and `3xx` response codes are **reserved for future
+extension**. Event recipients SHOULD NOT send these response codes in this spec
+version, but event senders MUST handle these response codes as errors and
+implement appropriate failure behavior.
+
+<!-- TODO: Should 3xx redirects and 401 (Unauthorized) or 403 (Forbidden) errors
+be retried? What about `405` (Method Not Allowed), 413 (Payload Too Large), 414
+(URI Too Long), 426 (Upgrade Required), 431 (Header Fields Too Large), 451
+(Unavailable for Legal Reasons)? -->
+
+Recipients MUST be able to handle duplicate delivery of events and MUST accept
+delivery of duplicate events, as the event acknowledgement could have been lost
+in return to the sender. Event recipients MUST use the [`source` and `id`
+attributes](https://github.com/cloudevents/spec/blob/v1.0.1/spec.md#required-attributes)
+to detect duplicated events (see [observability](#observability) for an example
+case where other event attributes may vary from one delivery attempt to
+another).
+
+Where possible, event senders SHOULD re-attempt delivery of events where the
+HTTP request failed or returned a retriable status code. It is RECOMMENDED that
+event senders implement some form of congestion control (such as exponential
+backoff) when managing retry timing. This specification does not document any
+specific congestion control algorithm or
+parameters. [Brokers](./overview.md#broker) and
+[Channels](./overview.md#channel) MUST implement congestion control and MUST
+implement retries.
 
 ### Observability
 
-CloudEvents received by Sink MAY have
-[Distributed Tracing Extension Attribute](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md).
-
-### Event reply contract
-
-An event sender supporting event replies SHOULD include a `Prefer: reply` header
-in delivery requests to indicate to the sink that event reply is supported. An
-event sender MAY ignore an event reply in the delivery response if the
-`Prefer: reply` header was not included in the delivery request.
-
-An example is that a Broker supporting event reply sends events with an
-additional header `Prefer: reply` so that the sink connected to the Broker knows
-event replies will be accepted. While a source sends events without the header,
-in which case the sink may assume that any event reply will be dropped without
-error or retry attempt. If a sink wishes to ensure the reply events will be
-delivered, it can check for the existence of the `Prefer: reply` header in the
-delivery request and respond with an error code if the header is not present.
-
-### Data plane contract for Sources
-
-See [Source Delivery specification](sources.md#source-event-delivery)
-for details.
-
-### Data plane contract for Channels
-
-See [Channel Delivery specification](channel.md#data-plane) for details.
-
-### Data plane contract for Brokers
-
-See [Broker Delivery specification](broker.md)
-
-## Changelog
+Event senders MAY add or update CloudEvents attributes before sending to
+implement observability features such as tracing; in particular, the
+[`traceparent` and `tracestate` distributed tracing
+attributes](https://github.com/cloudevents/spec/blob/v1.0/extensions/distributed-tracing.md)
+may be modified in this way for each delivery attempt of the same event.
+
+This specification does not mandate any particular logging or metrics
+aggregtion, nor a method of exposing observability information to users
+configuring the resources. Platform administrators SHOULD expose event-delivery
+telemetry to users through platform-specific interfaces, but such interfaces are
+beyond the scope of this document.
+
+<!-- TODO: should we mention RECOMMENDED spans or RECOMMENDED metrics like in
+https://github.com/knative/specs/blob/main/specs/eventing/channel.md#observability?
+-->
+
+### Derived (Reply) Events
+
+In some applications, an event receiver might emit an event in reaction to a
+received event. An event sender MAY document support for this pattern by
+including a `Prefer: reply` header in the HTTP POST request. This header
+indicates to the event receiver that the caller will accept a [`200`
+response](#event-acknowledgement-and-repeat-delivery) which includes a
+CloudEvent encoded using the binary or structured formats.
+[Brokers](./overview.md#broker) and [Channels](./overview.md#channel) MUST
+indicate support for replies using the `Prefer: reply` header.
+
+The sender SHOULD NOT assume that a received reply event is directly related to
+the event sent in the HTTP request.
+
+A recipient MAY reply to any HTTP POST with a `200` response to indicate that
+the event was processed successfully, with or without a response payload. If the
+recipient will _never_ provide a response payload, the `202` response code is
+likely a better choice.
+
+If a recipient chooses to reply to a sender with a `200` response code and a
+reply event in the absence of a `Prefer: reply` header, the sender SHOULD treat
+the event as accepted, and MAY log an error about the unexpected payload. The
+sender MUST NOT process the reply event if it did not advertise the `Prefer:
+reply` capability.
 
-- 2020-04-20: `0.13.x release`: initial version that documents common contract
-  for sinks, sources, channels and brokers.

specs/eventing/motivation.md

@@ -1,49 +1,33 @@
 # Motivation
 
-The goal of Knative Eventing is to define common, composable primitives to
-enable late-binding event sources and event consumers.
+The goal of the Knative Eventing project is to define common primitives to
+enable composing event-processing applications through configuration, rather
+than application code.
 
-<!-- TODO(n3wscott): [Why late-binding] -->
+Building by combining independent components provides a number of benefits for
+application designers:
 
-Knative Eventing has following principles:
-
-1. Services are loosely coupled during development and deployed independently on
-   a variety of platforms (Kubernetes, VMs, SaaS or FaaS).
+1. Services are loosely coupled during development and may be deployed
+   independently on a variety of platforms (Kubernetes, VMs, SaaS or FaaS). This
+   composability allows re-use of common patterns and building blocks, even
+   across programming language and tooling boundaries.
 
 1. A producer can generate events before a consumer is listening, and a consumer
    can express an interest in an event or class of events that is not yet being
-   produced.
+   produced. This allows event-driven applications to be evolve over time
+   without needing to closely coordinate changes.
 
-1. Services can be connected to create new applications
-   - without modifying producer or consumer.
+1. Services can be connected to create new applications:
+   - without modifying producer or consumer
    - with the ability to select a specific subset of events from a particular
      producer
 
-These primitives enable producing and consuming events adhering to the
-[CloudEvents Specification](https://github.com/cloudevents/spec), in a decoupled
-way.
-
-Kubernetes has no primitives related to event processing, yet this is an
-essential component in serverless workloads. Eventing introduces high-level
-primitives for event production and delivery with an initial focus on push over
-HTTP. If a new event source or type is required of your application, the effort
-required to plumb them into the existing eventing framework will be minimal and
-will integrate with CloudEvents middleware and message consumers.
-
-Knative eventing implements common components of an event delivery ecosystem:
-enumeration and discovery of event sources, configuration and management of
-event transport, and declarative binding of events (generated either by storage
-services or earlier computation) to further event processing and persistence.
-
-The Knative Eventing API is intended to operate independently, and interoperate
-well with the [Serving API](https://github.com/knative/serving) and
-[Build API](https://github.com/knative/build).
-
----
-
-_Navigation_:
+In order to enable loose coupling and late-binding of event producers and
+consumers, Knative Eventing utilizes and extends the [CloudEvents
+specification](https://github.com/cloudevents/spec) as the data plane protocol
+between components. Knative Eventing prioritizes at-least-once delivery
+semantics, using the CloudEvents HTTP POST (push) transport as a minimum common
+transport between components.
 
-- **Motivation and goals**
-- [Resource type overview](overview.md)
-- [Interface contracts](interfaces.md)
-- [Object model specification](spec.md)
+Knative Eventing also defines patterns to simplify the construction and usage of
+event senders and recipients.