"Building your own storage adapter" info?

[thearchduke]

I've been working on a system that will be using postgres for our event database. However, the page for how to build your own adapter seems to have been deleted. Is the information in the linked commit outdated? It's still listed when you search the https://castore-dev.github.io/ docs, but then gives a 404. Not sure the right way to proceed here.

Thanks for the help!

[dvxam]

Hello there!

First and foremost, I want to extend my heartfelt congratulations to all the contributors behind this outstanding library. It truly stands as a remarkable tool.

Regarding the issue at hand, I've been working on crafting an EventStorage adapter for EventStoreDB, and I'm also considering creating event bus and queue adapters down the line.

To answer your question @thearchduke , up to this point, I've been shaping my adapter's behavior by drawing inspiration from existing implementations and unit tests for the DynamoDB and InMemory adapters. However, I suspect I might be overlooking some undocumented features, which has prompted a few questions, targeted to this project's main contributors (@ThomasAribart it seems given source history :) ):

• Performance expectations for certain operations: Some operations would need to be implemented by filtering events in-memory adapter-side instead of database-side. Take, for instance, the listAggregateIds function, which includes options initialEventAfter and initialEventBefore. In the case of EventStoreDB, listing aggregateIds would require to list all events from the beginning, potentially negating the expected performance boost from using the initialEventAfter option. While I've noticed this isn't employed in the core code but rather in tools like "dam",
  My question is whether this approach of in-memory filtering on the application side aligns with the library's planned design?

• the atomicity behaviour of pushEventGroup and GroupedEvent feature: while EventStoreDB provides transactional and atomic write of multiple event to a single stream (meaning, to a single aggregateId), it does not offer the same warranty when writing to multiple threads (for multiple different aggregateIds). And as EventStoreDB streams are append-only, there is no clean way to transactionally rollback if a event write went to fail.
  My question here is: In which limit this would impact the validity of EventStoreDB as a adapter for castore's EventStorage ?

I plan to open source the adapter, would you be open to include it in this codebase?

[thearchduke]

Regarding the issue at hand, I've been working on crafting an EventStorage adapter for EventStoreDB, and I'm also considering creating event bus and queue adapters down the line.

I wrote an (I'm sure hackish) implementation of a redis-backed command queue using bullmq that wasn't the hardest thing in the world. I also ended up building a whole edifice around using commands in a different way, which I'm not sure I'm able to open source, but would probably be a valuable snippet if nothing else. I'll check about that.

[ThomasAribart]

@thearchduke Indeed, good catch, the page has disappeared when we migrated the docs! Will re-publish it next week, thanks for the heads up 👍

Meanwhile, you can follow the implementation based on DynamoDB. The TypeScript interface does 90% of the job. The only small catch is that the adapter should throw an error that implements the EventAlreadyExists error when, well, attempting to write an event that already exists (without the force: true option). This allows for commands to be retried down the line in case of race condition: https://castore-dev.github.io/castore/docs/event-sourcing/pushing-events/

@dvxam thanks for the feedback! Always appreciated 🤗

Of course, I would be happy to include the EventStoreDB adapter into Castore!

Regarding the initialEventAfter, you are right. It is mainly there as a way to paginate the listAggregateIds query, which could otherwise return problematically large amounts of data. This method is exposed but not really used internally, except for the pourEventStoreCollectionEvents util in dam which is thought of as a long running task anyway. Potential solutions:

• Tolerate the bad performances and warn users not to use the initialEventAfter/Before options (through the README and/or console.warns)
• Throw an error when users use those options (and find a workaround to still have pourEventStoreCollectionEvents work).
• If EventStoreDB offers another pagination pattern, I'm open to considering it.
• Keep a special aggregate (with a reserved id), and duplicate any initial event there as well (keeping only the aggregateId, timestamp and eventStoreId), and query those events instead of all the db.

Regarding pushEventGroup, that's a bummer. Potential solutions:

• Throw an error when users use this method 🤷‍♂️ It's sad but that's life
• Implement transactions yourself, like pushing two "technical" events on each aggregates like AGGREGATE_LOCKED and AGGREGATE_UNLOCKED respectively before and after the transactions take place. The pushEvent method could then transactionally push events only if aggregates are in an unlocked state (i.e. if the previous event is not AGGREGATE_LOCKED). This is hacky and raises a lot of questions (like what versions those events should even use ?), but seems doable.

Last thought: The immutability of EventStoreDB makes the force: true option of the pushEvent method unimplementable. I don't see any option there other than throwing a clear error if anyone uses it (but they will probably already be aware of that anyway).