Create a notifier plugin for AWS

[esweiss]

Create a notifier plugin for AWS for non-K8S usage, similar to the gcsbundle notifier plugin.

[martincapello]

I have been struggling trying to figure it out how to create a notifier plugin based on Amazon's S3.
The issue I'm facing is that S3 seems to not support the required functionality to make a notifier work properly.
For instance, for the gcsbundle, when preparing the SPIRE server bundle to be saved on a GCS bucket it uses the storage.Conditions struct to set the conditions that GCS must enforce in order to write/overwrite the object (in this case, the server bundle) successfully. In other words, it uses a sort of optimistic locking when saving the bundle on the GCS bucket.
On S3 it is the lack of this capability the thing that is preventing me to use it, as long as it is not able to enforce any conditions before writing the object on the bucket, I cannot implement any optimistic locking-like logic.

Some context
This is only a problem when SPIRE server is running as HA, so several servers are running behind a load balancer and a shared datastore is used by them.
Then, for any SPIRE server, when a rotation of keys happens, a notifier's logic goes like this:

1. Fetch the current object version for the server bundle on the remote storage system (i.e. GCS).
2. Load the local bundle from the shared datastore.
3. Upload the local bundle only if the object version has not changed in the remote storage system compared with the version number we got on step 1.
4. If the version has changed, it would indicate that other server won the race and updated the bundle first, so the cycle restarts with step 1.

So, to properly implement step 3, I need some support from the remote storage, which S3 doesn't seem to provide.

All this bundle uploading to a remote storage is to enable the agents to boot using the trust bundle retrieved from a specified URL (as described here).

Some possibilities/ideas

1. Use Amazon's DynamoDB: It seems to provide the functionality needed, but it wouldn't provide a nice user experience given the added complexity involved in handling the json response it returns.
2. Introduce some coordination mechanism into SPIRE server (in core or in a plugin maybe?) that allows it to upload the bundle by only one cluster's server at a time.
   I've been researching a bit how could this be implemented, and it seems that nats would be a good fit, because it already supports clustering out-of-the-box using the RAFT consensus algorithm. The downside of this approach is the amount of work it involves, besides adding another moving part to an already complex piece of software. But I believe this could be mitigated if we could introduce a plugin that does all the heavy lifting, then the impact would be quite limited.
3. Evaluate the actual cost/benefit in using S3 for this job and discard this issue if it doesn't worth it. For this it would be awesome to have as many community members opinions as possible.

[evan2645]

The notifier interface was originally created to satisfy the bundle update use case, with the thinking that more "notification" use cases would bubble up in the future. Unfortunately, we haven't really seen that happen.

In the meantime, we modeled this bundle update use case against the notifier interface, which is event-driven. There are a lot of things that can cause this bundle update event to be fired... and if we are going to handle it appropriately, then consistency and concurrency become big concerns.

I think the problem here is that our use case doesn't really require us to worry about concurrency or consistency. What we need is a bundle that is sufficiently up-to-date for agent bootstrapping. If I was an operator using SPIRE today, and I wanted a bundle on S3, I'd probably just write a cron job that pulls a copy of the bundle and uploads it every 5 minutes.... and, I bet that would work in most cases

It makes me wonder what other options we have. It feels like the notifier interface isn't a good fit.

[azdagron]

Yeah, I concur. The notifier interface really hasn't turned out to be a good fit, even for bundle notifications, where we currently don't even use the bundle in the event payload because of the requisite read-modify-write conflict detection cycle the plugins have to employ so that servers don't overwrite recently published changes with stale results.

[evan2645]

@azdagron what would your preferred solution here be? Something inside SPIRE Server that knows how to do this, or something in a different binary?

[azdagron]

Good question. I do think this job can be performed satisfactorily from an outside binary consuming the Bundle.GetBundle endpoint. We've already got one of those (the oidc-discovery-provider), so there is precedent for building a sidecar binary to provide API-enabled behavior.
However, while the Notifier interface hasn't been the easiest from the implementation standpoint, having something built in the SPIRE server to perform this function has been convenient. Requiring another binary complicates the deployment story a little.

I probably need to think about it. If it was implemented in SPIRE server, would we still want it pluggable or would we just implement various bundle "sources" directly?

[evan2645]

If it was implemented in SPIRE server, would we still want it pluggable or would we just implement various bundle "sources" directly?

I don't know... we are getting uncomfortably close to SVIDStore here. Does SPIRE Server get a BundleStore? Does SPIRE Server get an SVIDStore? @amartinezfayo and @MarcosDY curious to hear your thoughts on this...

[evan2645]

This work never completed due to consistency model mismatch between the notifier plugin and S3. I don't think it will ever complete due to this reason. Rather than keep this issue, I opened a new issue (#2909) suggesting we deprecate the notifier plugin in favor of a new plugin that is built for this specific purpose. If we take this route, I think we can get our S3 bundles.

Thanks very much to @martincapello for doing the research here 🤗