Decide on pubsub topic and sub deletion strategy for Broker and Trigger

[grantr]

When a Broker or Trigger is deleted, we also need to clean up their pubsub resources. Deleting immediately may not be the best approach for the user, since they may lose messages that were already acknowledged by the Broker.

This issue is about deciding what the deletion strategy is. We can create a new issue for implementation.

Pubsub deletion scenarios

In pubsub, the deletion scenarios work like this:

Delete a topic, sub still exists: sub continues delivering events until deleted, references _deleted-topic_.

Create sub with name of previously deleted topic: ? Haven't tried this yet, but probably fails

Create a topic with same name as deleted topic: A new topic is created. Any subs to the deleted topic continue delivering events from the deleted topic. Internally the sub is still attached to the deleted topic.

Create sub with name of recreated topic: sub attaches to new topic, not deleted topic.

Set sub expiration policy: Set a ttl (1-31 days). The sub is automatically deleted by PubSub when there is no "activity" for ttl period.

[grantr]

In my opinion, we should aim for the same behavior as pubsub. In this context the Broker is equivalent to the topic and the Trigger is equivalent to the pull subscription. Deleting a broker means no more ingress, but any accepted events are still delivered to existing triggers. If a trigger is deleted, no more events will be delivered to the subscriber, and any undelivered events will be dropped.

In concrete terms this could be implemented like this:
Delete Broker:

1. Delete decoupling topic.
2. Set expiration policy of N days on decoupling sub. The sub will be deleted by PubSub when it goes idle, so the Broker doesn't have to track it anymore.
3. Remove finalizer.

Delete Trigger:

1. delete retry topic and sub.
2. Remove finalizer.

The data plane will need to expect that pubsub resources may be deleted at any time and handle that gracefully.

[grantr]

@yolocs notes that when the broker is deleted, reply is no longer supported.

[grantr]

Because support for reply is necessary for the Broker to remain conformant, it seems that the decoupling topic must remain available for publish as long as triggers are delivering events. But triggers aren't required to continue delivering events, since the conformance spec doesn't specify what happens to triggers when a broker is deleted. This seems like a gap in the spec.

If we do want to support pubsub-like behavior for triggers, I can think of a couple strategies:

• Block broker deletion while triggers still exist and have message to consume. In this case, should the broker continue to be available for new messages? If it isn't, that potentially leads to a long downtime for that broker's address because a new Broker can't be created until all triggers consume all their events. But if the broker is available for new messages, sources would continue sending messages to it, so when will it ever be idle? This basically requires all source and trigger objects to also be deleted.
• Delete the broker object (so no more events can come in), but leave the decoupling topic in place for the data plane to continue using. The deleted broker's targets config would remain in place. This makes the fanout and retry services more complex as they must deal with a trigger being associated with multiple brokers at once (1+ deleted brokers and the current broker with that name). It also means we need a special controller to notice when deleted broker resources are idle and delete them. Creating Kubernetes resources for Topics and PullSubscriptions may make tracking these Pubsub resources easier.

These are just initial thoughts; I may be completely wrong about all of this. 😁

[grantr]

We talked a bit more about this recently. Notes:

One use case for triggers continuing to operate after a broker is deleted is drain: similar to graceful shutdown, we want to stop accepting new events and process any events that were already accepted.

If we force-delete triggers when a broker is deleted, that could be a surprising experience for users who created the triggers. Brokers are intended to be mostly behind the scenes.

If the broker/trigger has a dead letter queue, we could dead-letter replies instead of attempting to deliver them, but this would also be a surprising experience for users.

We agreed that the broker should not accept new events while it is being deleted (allowing for eventual consistency, since this is Kubernetes).

If we block broker deletion until all triggers are deleted AND wait for users to delete the triggers or for the triggers to become idle, then the broker might be blocked on deletion for a long time. Combining that with the above decision means that a broker blocked on deletion will also make that broker's address unable to receive events until all triggers are deleted. Is this acceptable to operators? Is there an operator use case for replacing an existing broker with a new one with the same name relatively quickly?

[grantr]

Current strategy for R1 is to delete all pubsub resource on broker deletion.