-
Notifications
You must be signed in to change notification settings - Fork 163
Watcher Braindump
This is just a random brain dump of ideas about watchers.
New watchers are registered via the standard REST api.
Probably will provide:
- pattern
- endpoint
- lifetime?
- autogenerated client id of some kind? (SHA1 enough to avoid collisions?)
Patterns would obviously be redis psubscribe patterns.
Endpoint would be a uri scheme endpoint. I.e.
* amqp://user:password@rabbitmqserver:port/vhost
* http://user:password@webhookserver:port/
* xmpp://node@domain/?query
* redis://password@server:port/db
* mysql://user:password@server:port/db/table ?
* websocket?
* ??????
Essentially I'm looking for the ability to conceptualize an endpoint in a parseable uri string.
This is something I'm debating in a forward looking sense. With ZooKeeper, watches are one-shot deals. You register a watch and once you get a "hit", your watch is deleted and you must reregister. This works fine for the stateful/persistent connection model of ZooKeeper clients. It's all in-band. For the stateless model Noah uses, it doesn't make sense. The endpoint is probably disconnected from the client that registered the watch.
So the problem becomes how to expire watches that are no longer valid? Is it neccesary? In a "cloud" environment of dynamically growing and shrinking resources, you might have a minimum poolsize of 10 servers. During capacity events, the pool size will grow and shrink never having less than 10. Obviously we don't want to keep 10 additional endpoint registrations when those nodes are ephemeral. The (optional?) lifetime would allow a user to set a value that matches with whatever policy they have defined for autoscaling. E.g.
In a scaling event, we allocate resources in blocks of ten. Allocated blocks are persisted through the life of the event + four hours beyond the event. At this point deallocation happens.
Dynamically allocated resources are no different than persistent resources except in the area of lifetime. Should the user have to externally track dynamic resources and deregister the commensurate watches?
Optionally, Noah processing agents could simply remove any failed endpoint after some defined period of time for unreachability.
As a watcher processor, I should attempt to notify a registered endpoint 3 times over a period of 4 hours before deregistering the endpoint.
This leads to the last item...
Optionally in watcher endpoint registration, a server-side SHA1(256?) key is generated for the endpoint. The user is free to track said key and use for managing ephemeral watches or other purposes.
Some method is needed for identifying duplicate watch registrations. Possibly it's as simple as an index on pattern+endpoint.
This deals primarily with the architecture behind processing the published messages from Redis and distributing them to endpoints.
In an effort to ensure that notifications are handled as quickly and efficiently as possible, it may make sense to aggregate watches by pattern and process all registered endpoints for said pattern at once. How will this play out in EventMachine land? EM::Iter? How does this scale to hundreds or thousands of registered endpoints for a given pattern? Should Noah "autoshard" patterns. Example:
given a threshold of 10 endpoints per distinct pattern, watcher processors will narrow the pattern namespace as needed to maintain the ratio of endpoints per pattern
Another option is to simply say that a processor will handle N registrations. The C10K notes in the EM documentation are primarily geared to servicing inbound requests as opposed to operating as a client.
This variation on Fixed Registrations simply groups processors by the type of endpoint. This would allow for optimization based on endpoint. Webhook endpoints have a different profile than AMQP. Assuming we have multiple patterns to the same AMQP endpoint could all be proccessed in-band. Same would apply for multiple patterns to the same redis server.
There needs to be a way to ensure, without fail, that no notification is sent by multiple processors to the same endpoint. Additionally, notifications MUST be in order. This is absolutely critical to the point that the whole project should be scrapped if it can't be ensured.
Just some more random thoughts
I'm considering implementing a long-polling in-band to the API endpoint somehow. Possibly similar to the WebSocket example.
Given an API endpoint of http://servername.domain:XXXX/, an optional in-band notification service is available at http://servername.domain:YYYY/
One way that you can use ZooKeeper is as a distributed lock manager
in progress