Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

datastore: support kind schema. fixes #85 #86 #157

Closed

Conversation

stephenplusplus
Copy link
Contributor

Hopefully this is what you had in mind 😊

@stephenplusplus
Copy link
Contributor Author

Probably could have done a better job explaining how this works.

The basic idea is, a schema is registered as a hidden "__proto" object on the Kind. It is in the format of an entity proto, without any actual values. When an entity is compared against a kind, the entity is converted to an entity proto, and matched up against the "ghost protocol".

@ryanseys
Copy link
Contributor

a schema is registered as a hidden "__proto" object on the Kind

Can you explain this further? I don't quite understand why this needs to be done and what the purpose of __proto is. I've never actually used this property or pattern in the past so I don't have a clue what its purpose is. Googling for "__proto" just returns results on "__proto__" and that "The __proto__ property is deprecated and should not be used." which sounds sketchy.

@stephenplusplus
Copy link
Contributor Author

I'll start this post off the way I started my last one... Probably could have done a better job explaining how this works! There's actually nothing special about __proto, I just used the double underscore to mark it pseudo-private. If it's more clear, I could re-structure the fieldMeta object to look more like:

{
  name: {
    metadata: {
      type: String,
      indexed: false
    },
    protocol: {
      string_value: null
    }
  },
  age: {
    metadata: {
      type: datastore.int
    },
    protocol: {
      integer_value: null
    }
  }
}

It's currently:

{
  name: {
    type: String,
    indexed: false,
    __proto: {
      string_value: null
    }
  },
  age: {
    type: datastore.int,
    indexed: false,
    __proto: {
      integer_value: null
    }
  }
}

The basic idea is to store this "ghost protocol" (it sounds too cool not to call it that) to compare against the later completed protocol, whereby a "protocol" means: an entity in the format the JSON api expects. There are many conversion methods in entity.js, which handle taking something like: "Hello!" and converting it to the datastore format, { "string_value": "Hello!" }. I wanted to take the metadata schema format, such as:

var schema = {
  name: {
    type: String,
    indexed: false
  },
  tags: {
    type: String,
    multi: true
  },
  favArticles: {
    type: 'Key',
    multi: true
  },
  contact: {
    type: {
      telephone: { type: String },
      email: { type: String }
    }
  }
};

...and turn it into an empty (ghost) "protocol".

The whole process looks like this:

  1. User registers a Kind:

    dataset.registerKind('Person', {
      name: {
        type: 'String'
      }
    });

    Internally, we flesh out this object to the ghost protocol.

  2. User wants to save a new Person entity, but first wants to validate:

    dataset.validate('Person', {
      name: 'Stephen'
    });

    Internally, we convert the object passed in to an entity protocol, then compare it against the ghost protocol.

Hope this helps. Feel free to let me know if you (or anyone else, of course) have any ideas for simplifying the validation. 👍

@stephenplusplus
Copy link
Contributor Author

Updated to box in the metadata and protocol, as shown in #157 (comment)

@stephenplusplus
Copy link
Contributor Author

@rakyll friendly ping. I know this was going to be M2, but if you have any thoughts on this now, I'm happy to work on them so this makes the cut. Plus, since the change is rather large, the longer we wait, the more likely I'm stuck in rebase sad land. If there's anything I can do a better job of explaining, or you have any questions, let me know. If it makes it, yay! If not, at least we got to remove an outstanding PR :)

@rakyll
Copy link
Contributor

rakyll commented Sep 9, 2014

Could we keep it open for a while? Datastore has plans for an indexing/schema API. We should cover their needs before we publicize a client supports a schema.

@stephenplusplus stephenplusplus added this to the M2: bigquery & pubsub milestone Sep 19, 2014
@stephenplusplus stephenplusplus added status: blocked Resolving the issue is dependent on other work. and removed ready for review labels Sep 29, 2014
@jgeewax jgeewax added api: datastore Issues related to the Datastore API. enhancement labels Feb 2, 2015
@jgeewax jgeewax modified the milestones: Datastore Future, M2: bigquery & pubsub Feb 2, 2015
chingor13 pushed a commit that referenced this pull request Sep 12, 2022
- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 468735472

Source-Link: googleapis/googleapis@cfa1b37

Source-Link: googleapis/googleapis-gen@09b7666
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDliNzY2NjY1NjUxMGY1YjAwYjg5M2YwMDNhMGJhNTc2NmY5ZTI1MCJ9
chingor13 pushed a commit that referenced this pull request Sep 14, 2022
- [ ] Regenerate this pull request now.

PiperOrigin-RevId: 468735472

Source-Link: googleapis/googleapis@cfa1b37

Source-Link: googleapis/googleapis-gen@09b7666
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMDliNzY2NjY1NjUxMGY1YjAwYjg5M2YwMDNhMGJhNTc2NmY5ZTI1MCJ9
sofisl pushed a commit that referenced this pull request Oct 13, 2022
* updated CHANGELOG.md [ci skip]

* updated package.json [ci skip]

* updated samples/package.json

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Oct 13, 2022
* chore(main): release 2.0.0

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 11, 2022
* chore(main): release 3.1.0

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 11, 2022
* test: use fully qualified request type name in tests

PiperOrigin-RevId: 475685359

Source-Link: googleapis/googleapis@7a12973

Source-Link: googleapis/googleapis-gen@370c729
Copy-Tag: eyJwIjoiLmdpdGh1Yi8uT3dsQm90LnlhbWwiLCJoIjoiMzcwYzcyOWUyYmEwNjJhMTY3NDQ5YzI3ODgyYmE1ZjM3OWM1YzM0ZCJ9

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 11, 2022
🤖 I have created a release \*beep\* \*boop\*
---
## [2.13.0](https://www.github.com/googleapis/nodejs-dialogflow-cx/compare/v2.12.1...v2.13.0) (2021-08-17)


### Features

* add advanced settings for agent level feat: add rollout config, state and failure reason for experiment feat: add insights export settings for security setting feat: add language code for streaming recognition result and flow versions for query p... ([#156](https://www.github.com/googleapis/nodejs-dialogflow-cx/issues/156)) ([80d2f94](https://www.github.com/googleapis/nodejs-dialogflow-cx/commit/80d2f94bcfaa65ea442ee4f8133d850035be04ed))
* added support for DLP templates; expose `Locations` service to get/list avaliable locations of Dialogflow products ([#162](https://www.github.com/googleapis/nodejs-dialogflow-cx/issues/162)) ([b77bde9](https://www.github.com/googleapis/nodejs-dialogflow-cx/commit/b77bde9893cde0c9e766b8e844fa4cca61efb5aa))


### Bug Fixes

* **build:** migrate to using main branch ([#164](https://www.github.com/googleapis/nodejs-dialogflow-cx/issues/164)) ([092d69d](https://www.github.com/googleapis/nodejs-dialogflow-cx/commit/092d69d76716297aff6a671200088d5740b1fd47))
* **deps:** google-gax v2.24.1 ([#165](https://www.github.com/googleapis/nodejs-dialogflow-cx/issues/165)) ([6261c37](https://www.github.com/googleapis/nodejs-dialogflow-cx/commit/6261c37019b3ce94a9876bfa4634837132691888))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
sofisl pushed a commit that referenced this pull request Nov 11, 2022
🤖 I have created a release *beep* *boop*
---


## [1.1.0](googleapis/nodejs-service-management@v1.0.0...v1.1.0) (2022-06-30)


### Features

* support regapic LRO ([#154](googleapis/nodejs-service-management#154)) ([9871067](googleapis/nodejs-service-management@9871067))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
sofisl pushed a commit that referenced this pull request Nov 11, 2022
* chore(main): release 2.1.2

* 🦉 Updates from OwlBot post-processor

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>
sofisl pushed a commit that referenced this pull request Nov 16, 2022
[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [@types/node](https://togithub.com/DefinitelyTyped/DefinitelyTyped/tree/master/types/node) ([source](https://togithub.com/DefinitelyTyped/DefinitelyTyped)) | [`^16.0.0` -> `^18.0.0`](https://renovatebot.com/diffs/npm/@types%2fnode/16.18.3/18.11.9) | [![age](https://badges.renovateapi.com/packages/npm/@types%2fnode/18.11.9/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/npm/@types%2fnode/18.11.9/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/npm/@types%2fnode/18.11.9/compatibility-slim/16.18.3)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/npm/@types%2fnode/18.11.9/confidence-slim/16.18.3)](https://docs.renovatebot.com/merge-confidence/) |

---

### Configuration

📅 **Schedule**: Branch creation - "after 9am and before 3pm" (UTC), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/googleapis/nodejs-managed-identities).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzMi4yNDEuMTEiLCJ1cGRhdGVkSW5WZXIiOiIzNC4xMS4xIn0=-->
sofisl pushed a commit that referenced this pull request Nov 16, 2022
sofisl pushed a commit that referenced this pull request Nov 30, 2022
* remove unneeded doc links fix

* rerun synthtool
sofisl pushed a commit that referenced this pull request Nov 30, 2022
🤖 I have created a release \*beep\* \*boop\* 
---
## [2.1.0](https://www.github.com/googleapis/nodejs-datacatalog/compare/v2.0.0...v2.1.0) (2020-06-16)


### Features

* add order field to TagField and TagTemplateField ([#157](https://www.github.com/googleapis/nodejs-datacatalog/issues/157)) ([70c23c2](https://www.github.com/googleapis/nodejs-datacatalog/commit/70c23c2b609bd24901dc902090e23fbf92b55895))
* move ts target to es2018 from es2016 ([#172](https://www.github.com/googleapis/nodejs-datacatalog/issues/172)) ([e354dde](https://www.github.com/googleapis/nodejs-datacatalog/commit/e354ddeeeedd3e106c50ed579075f23c9d9c5bb5))
* promote library to GA ([#177](https://www.github.com/googleapis/nodejs-datacatalog/issues/177)) ([ac11090](https://www.github.com/googleapis/nodejs-datacatalog/commit/ac110906b723f362024318295e3de6743c905b7e))


### Bug Fixes

* proper fallback option handling ([51df672](https://www.github.com/googleapis/nodejs-datacatalog/commit/51df6721306ab6d2d9bf6919b0f3f432ca85b4ab))
* regenerate unit tests ([#163](https://www.github.com/googleapis/nodejs-datacatalog/issues/163)) ([4593117](https://www.github.com/googleapis/nodejs-datacatalog/commit/45931176d266d364e0508039cf6a4e284ad2923d))
* synth.py clean up for multiple version ([#167](https://www.github.com/googleapis/nodejs-datacatalog/issues/167)) ([c799604](https://www.github.com/googleapis/nodejs-datacatalog/commit/c799604b2d29269006ff39748afb78302ce46ca2))
---


This PR was generated with [Release Please](https://github.com/googleapis/release-please).
sofisl pushed a commit that referenced this pull request Jan 10, 2023
sofisl pushed a commit that referenced this pull request Jan 24, 2023
sofisl pushed a commit that referenced this pull request Jan 25, 2023
sofisl pushed a commit that referenced this pull request Sep 13, 2023
* Vision beta samples

* Vision beta samples

* Vision beta samples

* Vision beta samples

* fixing tests

* fixing tests

* fixing tests

* fixing tests

* fixing tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the Datastore API. status: blocked Resolving the issue is dependent on other work.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants