Rework sequencer storage & add handle update #711
Conversation
dholms
changed the title
| if (tx.writes.length > 200) { | ||
| throw new InvalidRequestError('Two many writes. Max: 200') | ||
| } |
There was a problem hiding this comment.
I assume this check lives here because you don't want it to be a hard limit in the lexicon (i.e. for all PDS implementations), is that right?
There was a problem hiding this comment.
Yea exactly. There are cases where you may want to do more than 200 writes & i feel that we shouldn't put this at the lexicon level. There's a good chance we'll remove this check at some point when we have a better handle on the repercussions of large mutations
| .addColumn('sequencedAt', 'varchar', (col) => col.notNull()) | ||
| .execute() | ||
|
|
||
| await db.schema |
There was a problem hiding this comment.
Having trouble recalling which types of queries these indices support. I kinda like the comments we've added in some of the other migrations documenting this, have found it useful!
There was a problem hiding this comment.
yup i'll add some comments 👍
| if (dialect === 'pg') { | ||
| builder = builder | ||
| .addColumn('seq', 'bigserial', (col) => col.primaryKey()) | ||
| .addColumn('invalidatedBy', 'bigint') |
There was a problem hiding this comment.
I dig the invalidatedBy idea! I wonder if we should also make it a foreign key.
There was a problem hiding this comment.
ah yeah I'd be game 👍
| const justRoot = new BlockMap() | ||
| justRoot.add(commitData.blocks.get(commitData.commit)) | ||
| carSlice = await blocksToCar(commitData.commit, justRoot) |
| .innerJoin('repo_commit_history', (join) => | ||
| join | ||
| .onRef('repo_commit_history.creator', '=', 'repo_seq.did') | ||
| .onRef('repo_commit_history.commit', '=', 'repo_seq.commit'), | ||
| ) | ||
| .select([ | ||
| 'repo_seq.seq as seq', | ||
| 'repo_seq.did as did', | ||
| 'repo_seq.commit as commit', | ||
| 'repo_seq.sequencedAt as sequencedAt', | ||
| 'repo_commit_history.prev as prev', | ||
| ]) | ||
| .selectAll() | ||
| .orderBy('seq', 'asc') | ||
| .where('invalidatedBy', 'is', null) |
There was a problem hiding this comment.
🤤 so nice, this thing is gonna whiz.
Co-authored-by: devin ivy <devinivy@gmail.com>
This reworks our sequencer storage so that we do the work upfront instead of collating it when we serve.
This does cause some duplicated data however, I do not think it's a concern for several reasons:
This allows our
repo_seqtable to hold heterogenous event types. It will also speed up the subscription route.This PR also adds in support for handle update events.
repo_optable as we weren't using it anymoretooBigevents I chose a limit of 200 ops or 1MB of block data.batchWriteinvalidatedBycolumn to seq which tracks if an event is invalidated by a later event namely: