WIP: propose the document level ownership #271

imotai · 2023-01-10T02:58:05Z

Motivations

solve the problem of data exchange between accounts

Would you like to implement this feature? [y/n]

Solution

jingchen2222 · 2023-01-15T03:55:30Z

Discussion Kick off

goal: record level permission management @jingchen2222 @imotai
architecture - data model @imotai @jingchen2222
primary key generator (logic, size limit, order)
- index key generator @jingchen2222
- ~~consider supporting aggregation by address~~ @jingchen2222
- additional schema: txid
- implementation @jingchen2222
- unified address design @imotai
key optimization (future work)

jingchen2222 · 2023-01-29T18:48:06Z

Design Doc: Document Database Storage

Background

Goal:

solve the problem of data exchange between accounts
record level permission management.
design primary key generator (logic, size limit, order)
design index key generator
support CreateDocument/UpdateDocument/DeleteDocument API based on KV operations
key optimization (future work)

Issue:

[propose the document level ownership #271](#271)

Structure

Database → Collection → Document

flowchart LR

A[Developers] --> B[database]
B --> C[colection 1]
C --> E[document 1]
C --> F[document 2]
C --> G[document 3]
B --> D[colection 2]
D --> H[document 4]
D --> I[document 5]
J[User] --> |"(Document, Signature, PublicKey)"|D

ID

ID Type	Size	Description
DbId	20	DbId := DB3Address
TxId	32	transaction id
AccountId	20	AccountId := DB3Address
BlockID	8	Current block height
MutationID	4	Index of mutation in the current block.
PutEntryIdx	4	Index of document in current database mutation
PutEntryID	16	PutEntryID := BlockID + MutationID + PutEntryIdx
CollectionID	16	CollectionID := PutEntryID
DocumentEntryID	16	DocumentEntryID := PutEntryID
DocumentID	33	DocumenID := DocumentEntryType+CollectionID + DocumentKeyID
IndexFiledID	4	Index of the index filed in the collection
IndexID	KeyBytes + 54	IndexId := IndexEntryType+CollectionID + IndexFiledID + KeyBytes + DocumentID

Document serialization

rust https://docs.rs/bson/latest/bson/
typescript https://github.com/mongodb/js-bson

Database & Collection Op

#297

Document OP

1. CreateDocument

sequenceDiagram
    Client->>+Node: 1.1 CreateDocument(document)
    Node->>+DocumentImpl:1.3 SubmitMutation(WRITE_DOCUMENT, document)
    Node-->>-Client: 1.4 Response(status, msg)
    DocumentImpl->>DocumentImpl: 2.1 documentId = GenerateDocumentId(blockId, autoId)
    DocumentImpl->>+KVStore: 2.2 InsertKV(documentId, txid, document)
    KVStore-->>-DocumentImpl: 2.3 Response(status, msg)
    DocumentImpl-->>DocumentImpl: 2.4 index_key_pairs = GetIndexKeyPairs(db, collection, document)
    loop Create Indexes: index_key_pairs.foreach
    DocumentImpl-->>DocumentImpl: 2.5.1 GenerateIndexId(db, collection, documentId, index, key)
    DocumentImpl->>+KVStore: 2.5.2 InsertKV(indexId, txid, documentId)
    KVStore-->>-DocumentImpl: 2.5.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.6 Response(status, msg, documentId)

2. UpdateDocument

sequenceDiagram
    Client->>+Node: 1.1 UpdateDocument(documentId, document)
    Node->>+DocumentImpl:1.2 SubmitMutation(UpdateDocument, documentId, document)
    Node-->>-Client: 1.3 Response(status, msg)
    DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, address, old_document)
    DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, address)
    DocumentImpl-->>DocumentImpl: 2.4 modified_index_key_pairs = GetModifiedIndexKeyPairs(db, collection, old_document, document)
    DocumentImpl->>+KVStore: 2.5 Update documet: InsertKV(documentId, txid, document)
    KVStore-->>-DocumentImpl: 2.6 Response(status, msg)
    DocumentImpl->>DocumentImpl: 2.7 (indexes, keys) = CheckModifiedIndexs(document, index_keys)
    loop Update Indexes: modified_index_key_pairs.foreach(index, key)
    DocumentImpl-->>DocumentImpl: 2.8.1 indexId = GenerateIndexId(db, collection, index, key, documentId)
    DocumentImpl->>+KVStore: 2.8.2 InsertKV(indexId, txid, documentId)
    KVStore-->>-DocumentImpl: 2.8.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.9 Response(status, msg)

2. DeleteDocument

sequenceDiagram
    Client->>+Node: 1.1 DeleteDocument(documentId)
    Node->>+DocumentImpl:1.2 SubmitMutation(DeleteDocument, documentId)
    Node-->>-Client: 1.3 Response(status, msg)
    DocumentImpl->>+KVStore: 2.1 GetKV(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, index_keys, txid)
    DocumentImpl-->>DocumentImpl: 2.3 CheckOwnership(Signature, PublicKey, txid)
    DocumentImpl->>+KVStore: 2.4 Delete documet: DeleteKV(documentId)
    KVStore-->>-DocumentImpl: 2.5 Response(status, msg)
    DocumentImpl-->>DocumentImpl: 2.6 index_key_pairs = GetIndexKeyPairs(db, collection, document)
    loop Delete Indexes:  index_key_pairs.foreach(index, key)
    DocumentImpl-->>DocumentImpl: 2.7.1 GenerateIndexId(db, collection, index, key, documentId)
    DocumentImpl->>+KVStore: 2.7.2 DeleteKV(indexId, txid)
    KVStore-->>-DocumentImpl: 2.7.3 Response(status, msg)
    end
    DocumentImpl-->>-Node: 2.8 Response(status, msg)

4. GetDocument

sequenceDiagram
    Client->>+Node: 1.1 GetDocumentById(documentId)
    Node->>+DocumentImpl:1.3 GetDocumentById(documentId)
    DocumentImpl->>+KVStore: 2.1 BatchGetKey(documentId)
    KVStore-->>-DocumentImpl: 2.2 Response(status, msg, document)
    DocumentImpl-->>-Node: 2.5 Response(status, msg)
    Node-->>-Client: 1.4 Response(status, msg)
    Client->>+Node: 2.1 GetDocuments(index, key)
    Node->>+DocumentImpl:2.2 GetDocuments(index, key)
    DocumentImpl->>DocumentImpl: 2.3 index_range = GenerateIndexRange(db, collection, index, key)
    DocumentImpl->>+KVStore: 2.3 GetRange(index_range)
    KVStore-->>-DocumentImpl: 2.4 Response(status, msg, documents)
    DocumentImpl-->>-Node: 2.5 Response(status, msg, documents)
    Node-->>-Client: 2.6 Response(status, msg, documents)

Implementation

jingchen2222 · 2023-01-30T23:43:55Z

Discussion

@imotai

1. What is the relationship between collection and documents? 1:N or N:M?

Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?

2. Will we support Project, Namespace, Collection Group, in db3.

I find those concepts are used as defined in the google firestore.

imotai · 2023-01-31T00:30:25Z

Discussion

@imotai

1. What is the relationship between collection and documents? 1:N or N:M?

Regarding to the rpc proto definition mentioned here, one document might be under multiple collections? Will we define and implement collection in the same way?

2. Will we support Project, Namespace, Collection Group, in db3.

I find those concepts are used as defined in the google firestore.

1:n
db3 just support database , collection and document

jingchen2222 · 2023-01-31T00:33:06Z

RPC Interface

DB3 will design and implement a set of APIs which are similar to google firestore so that the developers who are familiar to firestore can transfer to DB3 easily.

Reference:

DB3 Admin API

This API provides several administrative services for DB3 store.

P0

API	Description
CreateDatabase	Create database and linked with address
CreateCollection	Define indexes
ListIndexes	Lists composite indexes.
GetIndex	Gets a composite index.
GetDatabase	Gets information about a database.
GetField	Gets the metadata and configuration for a Field.
ListFields	Lists the field configuration and metadata for this database.

P1:

API	Description
UpdateDatabase	Updates a database.
UpdateField	Updates a field configuration. Currently, field updates apply only to single field index configuration.
ExportDocuments	Exports a copy of all or a subset of documents from DB3 store
ImportDocuments	Imports documents into DB3 store. Existing documents with the same name are overwritten.
ListDatabases	List all the databases in the project.
CreateIndex	Creates a composite index. This returns a db.longrunninbg.Operation (similar to google.longrunning.Operation) which may be used to track the status of the creation. The metadata for the operation will be the type IndexOperationMetadata
DeleteIndex	Deletes a composite index.

DB3 Document Op API

P0: API

API	Description
CreateDocument	Creates a new document.
GetDocument	Gets a single document.
ListDocuments	List documents
UpdateDocument	Updates or inserts a document.
DeleteDocument	Delete a document

P1: Document API

API	Description
BatchGetDocuments	Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested.
ListCollectionIds	List documents
RunQuery	Runs a query.
RunAggregationQuery	Return the number of documents in table given a filter. `SELECT COUNT() FROM ( SELECT FROM k where a = true );`
PartitionQuery	Partitions a query by returning partition cursors that can be used to run the query in parallel.

DB3 Entity Proto

Entity	Description
Database	Gets multiple documents. Documents returned by this method are not guaranteed to be returned in the same order that they were requested.
Field	Represents a single field in the database. Fields are grouped by their "Collection Group", which represent all collections in the database with the same id.
Index	indexes enable simple and complex queries against documents in a database.
Document	A db3 document. Must not exceed 1 MiB - 4 bytes.

jingchen2222 · 2023-01-31T00:40:23Z

Milestone and resource plan

Milestore	Task	Owner	Status
Design POC	Document Operation workflow design	@cj	Completed
Design POC	DB3 admin opertation workflow design	@wtz	Completed
Feature Implementation	Entity Proto implement P0	@wtz	In Progress
Feature Implementation	DB3 Admin P0 Op Implementation	@wtz	In Progress
Feature Implementation	DB document P0 Op implementation	@cj	Not started
QA	Unit test & integartion test	@cj	Not started
QA	Benchmark	@cj	Not started
JS SDK API	Implement JS SDK for db3 admin and document operation	@zhaojun	Not started

jingchen2222 · 2023-02-02T03:11:45Z

jingchen2222 · 2023-02-05T01:27:34Z

Discussion

@imotai 🔔 👀 ♨️
I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().

IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|

imotai · 2023-02-05T01:35:15Z

Discussion

@imotai 🔔 👀 ♨️ I wanna introduce CollectionID (16 bytes) as a part of IndexID instead of using colecction_name varchar().

IndexId = |DbID(20)|CollectionId(16)|BlockID(16)|

agree

jingchen2222 · 2023-02-14T23:23:33Z

Propose index encode solution

Goal

index key encode into/decode from bytes
index bytes can be sorted
index bytes order should be consistent with the original value order
support Null encode
support Emptry String encode
define the order between types. Reference: https://www.mongodb.com/docs/manual/reference/bson-type-comparison-order/

Reference

https://www.mongodb.com/docs/manual/reference/bson-type-comparison-order/

Further work will be taken care by #321

imotai added the discussion-required label Jan 10, 2023

jingchen2222 mentioned this issue Jan 23, 2023

WIP: propose the auto-increment document id generator #262

Closed

jingchen2222 self-assigned this Jan 29, 2023

jingchen2222 added documentation Improvements or additions to documentation metastore D-kvstore document protocol labels Jan 30, 2023

imotai pinned this issue Feb 2, 2023

jingchen2222 mentioned this issue Feb 3, 2023

Add DB3Document and support add document operation and new-db #302

Merged

6 tasks

jingchen2222 linked a pull request Feb 7, 2023 that will close this issue

feat: support list documents api and show-doc command cli #306

Merged

5 tasks

imotai closed this as completed in #306 Feb 8, 2023

jingchen2222 linked a pull request Feb 9, 2023 that will close this issue

support get-doc command cli and GetDocument API #311

Merged

2 tasks

jingchen2222 reopened this Feb 9, 2023

jingchen2222 mentioned this issue Feb 9, 2023

support get-doc command cli and GetDocument API #311

Merged

2 tasks

imotai closed this as completed in #311 Feb 10, 2023

jingchen2222 reopened this Feb 12, 2023

jingchen2222 linked a pull request Feb 13, 2023 that will close this issue

Feat/delete doc #314

Merged

2 tasks

jingchen2222 removed a link to a pull request Feb 13, 2023

Feat/delete doc #314

Merged

2 tasks

imotai unpinned this issue Feb 16, 2023

imotai closed this as completed Apr 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: propose the document level ownership #271

WIP: propose the document level ownership #271

imotai commented Jan 10, 2023

jingchen2222 commented Jan 15, 2023 •

edited

Loading

jingchen2222 commented Jan 29, 2023 •

edited

Loading

jingchen2222 commented Jan 30, 2023 •

edited

Loading

imotai commented Jan 31, 2023 •

edited

Loading

Discussion

1. What is the relationship between collection and documents? 1:N or N:M?

2. Will we support Project, Namespace, Collection Group, in db3.

jingchen2222 commented Jan 31, 2023 •

edited

Loading

jingchen2222 commented Jan 31, 2023 •

edited

Loading

jingchen2222 commented Feb 2, 2023 •

edited

Loading

jingchen2222 commented Feb 5, 2023 •

edited

Loading

imotai commented Feb 5, 2023

Discussion

jingchen2222 commented Feb 14, 2023 •

edited

Loading

WIP: propose the document level ownership #271

WIP: propose the document level ownership #271

Comments

imotai commented Jan 10, 2023

Motivations

Solution

jingchen2222 commented Jan 15, 2023 • edited Loading

Discussion Kick off

jingchen2222 commented Jan 29, 2023 • edited Loading

Design Doc: Document Database Storage

Background

Goal:

Issue:

Structure

ID

Document serialization

Database & Collection Op

Document OP

1. CreateDocument

2. UpdateDocument

2. DeleteDocument

4. GetDocument

Implementation

jingchen2222 commented Jan 30, 2023 • edited Loading

Discussion

1. What is the relationship between collection and documents? 1:N or N:M?

2. Will we support Project, Namespace, Collection Group, in db3.

imotai commented Jan 31, 2023 • edited Loading

Discussion

1. What is the relationship between collection and documents? 1:N or N:M?

2. Will we support Project, Namespace, Collection Group, in db3.

jingchen2222 commented Jan 31, 2023 • edited Loading

RPC Interface

Reference:

DB3 Admin API

P0

P1:

DB3 Document Op API

P0: API

P1: Document API

DB3 Entity Proto

jingchen2222 commented Jan 31, 2023 • edited Loading

Milestone and resource plan

jingchen2222 commented Feb 2, 2023 • edited Loading

Implementation

Basic operation

Advance feature

jingchen2222 commented Feb 5, 2023 • edited Loading

Discussion

imotai commented Feb 5, 2023

Discussion

jingchen2222 commented Feb 14, 2023 • edited Loading

Propose index encode solution

Goal

Reference

jingchen2222 commented Jan 15, 2023 •

edited

Loading

jingchen2222 commented Jan 29, 2023 •

edited

Loading

jingchen2222 commented Jan 30, 2023 •

edited

Loading

imotai commented Jan 31, 2023 •

edited

Loading

jingchen2222 commented Jan 31, 2023 •

edited

Loading

jingchen2222 commented Jan 31, 2023 •

edited

Loading

jingchen2222 commented Feb 2, 2023 •

edited

Loading

jingchen2222 commented Feb 5, 2023 •

edited

Loading

jingchen2222 commented Feb 14, 2023 •

edited

Loading