Internal architecture: adjust DataStore & DataType responsibilities

[gmaclennan]

Description

As the codebase has evolved along with the proposed public API, some changes to the responsibilities of the internal modules DataStore and DataType can simplify integration and reduce dependencies between different parts of the codebase.

Proposed changes:

DataStore

ADataStore should be an abstraction over a particular "Mapeo Store", e.g. a collection of hypercores that store a particular datatype. Responsibilities:

• Write a document to the write core of the store.
• Read a document based on versionId.
• Manage read indexing and return current indexing state.
• Handle decoding/encoding of data

With this we can isolate the boundary with hypercore / core store to this module (apart from within the blob store), and largely isolate indexing: The write method can resolve when the document is indexed.

const store = new DataStore({
  coreManager,
  namespace: 'auth' | 'data',
  // DataStore will consider a block "indexed" once this function resolves.
  indexDocs: (blocks: Buffer[]) => Promise<void>
})

// Guarantee that `doc` is indexed once this resolves
store.write(doc: Exclude<MapeoDoc, 'versionId'>): Promise<MapeoDoc>

store.read(versionId: string): Promise<MapeoDoc>

store.getIndexState() // return whether is currently indexing, and estimate of indexing remaining to be done

store.on('index-state') // same as above.

DataType

Implements the MapeoDoc methods from the public API. Needs a DataStore instance, probably also needs a way to get permissions. One dataType instance for each MapeoDoc type, but not all will be exposed in the public API. Needs a way to define what data types it will handle, possibly a Drizzle schema.

const dataType = new DataType({
  dataStore,
  schema,
  getPermissions // function to return current permissions (read/write, own vs. others)
})

dataType.create()
dataType.getByDocId()
dataType.getByVersionId()
dataType.getMany()
dataType.update()
dataType.delete()

IndexWriter

One instance that does all writing to the indexes. Only place where SQLite is written to (all other places are read-only SQLite). Just exports one method batch() which writes one or more blocks to the index.

• Decodes blocks from buffers
• Ignores invalid blocks
• Migrates previous schema versions if required
• Writes each data type to a separate sqlite table
• batch() resolves when documents are written to the index
• If batch errors, should return details about any docs that were written

const indexer = new Indexer({
  schemaNames: [] // Array of schema names to process
})
indexer.batch(blocks: Buffer[]): Promise<void>

Tasks

• Create sqlite index tables from JSONSchema definitions in @mapeo/schema #122
• Updated DataStore class #142
• new IndexWriter for any block #143
• Updated DataType class #145
• Integrate into new DataStore and DataType into Mapeo class #148

[gmaclennan]

Done in #149