Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement datastore v1beta3 #1288

Closed
46 of 49 tasks
dhermes opened this issue Dec 15, 2015 · 26 comments
Closed
46 of 49 tasks

Implement datastore v1beta3 #1288

dhermes opened this issue Dec 15, 2015 · 26 comments
Assignees
Labels
api: datastore Issues related to the Datastore API.

Comments

@dhermes
Copy link
Contributor

dhermes commented Dec 15, 2015

Current plan of attack:

New Features:

  • Add GqlQuery support (Missing GQL support #304)
  • QueryResultBatch.skipped_cursor added (only set when skipped_results != 0)
  • EntityResult.cursor added (set by the backend when the entity result is part of a RunQueryResponse.batch.entity_results response)
  • Parse protobuf errors
@dhermes dhermes added the api: datastore Issues related to the Datastore API. label Dec 15, 2015
@pcostell
Copy link
Contributor

All those project IDs have already been whitelisted for the API.

Just double checking that this is a typo in the issue, but the correct url is:

https://datastore.googleapis.com/v1beta3/projects/omega-moonlight-697:runQuery

@dhermes
Copy link
Contributor Author

dhermes commented Dec 15, 2015

D'oh! Thought I changed it! Thanks for the keen eye

@dhermes
Copy link
Contributor Author

dhermes commented Dec 15, 2015

Culprit found, the constructor was relying on the connection base class (non-datastore specific)

from gcloud import connection
...
        if api_base_url is None:
            api_base_url = os.getenv(GCD_HOST,
                                     connection.API_BASE_URL)

@dhermes
Copy link
Contributor Author

dhermes commented Dec 16, 2015

For posterity, here is the "retired" original description:


pcostell I believe I am "done" updating for v1beta3 but am getting 404s when running the system tests with both

https://www.googleapis.com/v1beta3/projects/omega-moonlight-697:runQuery
https://www.googleapis.com/v1beta3/projects/s~omega-moonlight-697:runQuery

I think it requires a whitelist at this point to use v1beta3? I seem to remember a whitelist discussion but can't find any evidence of this.

Project IDs:

omega-moonlight-697
precise-truck-742
citric-celerity-697

/cc tseaver

dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 3, 2016
This is towards googleapis#1288 in preparation for the upgrade to `v1beta3`.
In particular, a single `Mutation` protobuf instance in `v1beta3` is not
sufficient to contain all changes to be committed, so we use the
container that is up one level in the hierarchy.
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 7, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 7, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 7, 2016
This is towards googleapis#1288 in preparation for the upgrade to `v1beta3`.
In particular, a single `Mutation` protobuf instance in `v1beta3` is not
sufficient to contain all changes to be committed, so we use the
container that is up one level in the hierarchy.
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 15, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Jan 15, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Feb 12, 2016
dhermes added a commit to dhermes/google-cloud-python that referenced this issue Feb 13, 2016
@dhermes
Copy link
Contributor Author

dhermes commented Feb 18, 2016

FYI @jgeewax @tseaver I just did a merge of master in the datastore-v1beta3 branch. I don't know of a better place to surface such comments. Just wanted to keep them in sync / catch any merge conflicts before the happen. Merged perfectly with no issues!

@dhermes
Copy link
Contributor Author

dhermes commented Feb 29, 2016

Another FYI, just did another merge. Only one merge conflict in system_tests/run_system_test.py (due to Bigtable modifying REQUIREMENTS and datastore-v1beta3 just removing it as a variable). Merge conflict easily resolved and merge completed.

@navreetaft
Copy link

Will this update change the user facing API for gcloud-python when interacting with datastore?

Also, will this help with ndb being implemented in gcloud-python? If so, any idea on an ETA for that? Also, @dhermes commented that

ndb saves take about 1/5 the amount of time the gcloud saves do because the RPCs happen directly in GAE instead of via the Cloud Datastore API

Will this performance gap be addressed by #557?

@dhermes
Copy link
Contributor Author

dhermes commented Mar 8, 2016

@navreetaft Essentially yes. The only interface change is Query.group_by to Query.distinct_on (to match the changed name in the protobuf). We also add support for geo points and for explicit null values stored and remove support for blob key values. We also plan to add support for GqlQuery.

You can check it out right now at the datastore-v1beta3 branch. It is 100% working (system tests even pass against the new version).

@dhermes
Copy link
Contributor Author

dhermes commented Mar 8, 2016

As for the performance gap, that is in reference to code running within App Engine.

@navreetaft
Copy link

I see, so the performance gap is due to ndb being able to talk directly to DataStore servers, whereas gcloud-python will have to go through some REST gateway (an additional hop)? Just want to make sure I understand.

@dhermes
Copy link
Contributor Author

dhermes commented Mar 8, 2016

Yup. The ndb library (inside GAE) talks to Google services via RPCs (remote-procedure calls) while gcloud-python (at least when I ran that test in November 2014) was using standard HTTPS requests to just call the datastore API.

@dhermes
Copy link
Contributor Author

dhermes commented Mar 8, 2016

@pcostell May have some thoughts / comments here

@pcostell
Copy link
Contributor

pcostell commented Mar 8, 2016

Yes, right now ndb inside GAE will still be faster than gcloud-python due to communication via RPC instead of HTTPS requests. gcloud-python on v1beta3 will be faster from GAE because v1beta3 is more performant than v1beta2, however it likely still won't be as fast as ndb in GAE for the same reasons above.

@dhermes
Copy link
Contributor Author

dhermes commented Mar 17, 2016

Another FYI, just did another merge. Only merge conflict was the delete of the demos.

/cc @pcostell

dhermes added a commit that referenced this issue Mar 28, 2016
@dhermes
Copy link
Contributor Author

dhermes commented Mar 28, 2016

Another FYI, just did another merge but rebased by mistake, d'oh!

@pcostell Both of my projects got 400's when running the system tests against v1beta3

@pcostell
Copy link
Contributor

@dhermes My guess this is related to some permissions changes -- now you have to explicitly enable the API (instead of just being part of the whitelist).

You can either login to your app with an @google.com account to do this (look for Cloud Datastore API - NEW in the API manager), or add me as an owner and I can do it for you.

@pcostell
Copy link
Contributor

@dhermes Are there any steps that need to happen before we can merge the v1beta3 branch into master and push to pypi?

@dhermes
Copy link
Contributor Author

dhermes commented Mar 29, 2016

  1. Merge the datastore-v1beta3 branch into master
  2. Send a PR to update the version in setup.py
  3. Push a new tag with the version (Travis pushes to PyPI on tag builds)

@pcostell
Copy link
Contributor

pcostell commented Apr 1, 2016

@dhermes We are ready for you to merge this in and push to pypi.

@dhermes
Copy link
Contributor Author

dhermes commented Apr 1, 2016

w00t.

@dhermes
Copy link
Contributor Author

dhermes commented Apr 1, 2016

Did my final merge of master into datastore-v1beta3. Only conflict was in run_system_tests.py (as predicted).

@tseaver
Copy link
Contributor

tseaver commented Apr 4, 2016

@pcostell I can't seem to enable the v1beta3 API. I have added you as an owner to my project: can you figure out what I screwed up?

@pcostell
Copy link
Contributor

pcostell commented Apr 4, 2016

@tseaver There is an (unfortunately timed) issue with the API service causing this issue. In particular if you disable Datastore v1beta2 (and potentially other APIs), you get left in state where you can't enable APIs. I'll follow up when I get more details.

@tseaver
Copy link
Contributor

tseaver commented Apr 4, 2016

@pcostell thanks for looking into it.

@pcostell
Copy link
Contributor

pcostell commented Apr 4, 2016

@tseaver
Copy link
Contributor

tseaver commented Apr 5, 2016

@pcostell Thank you -- whatever you did for the user in that issue seems to have worked for me, too.

parthea pushed a commit that referenced this issue Sep 22, 2023
* Add samples for Cloud Tasks [(#1068)](GoogleCloudPlatform/python-docs-samples#1068)

* Add samples for Cloud Tasks

* Respond to tasks sample review

* Update app engine queues samples

* Address review feedback

* Address review issues and convert pull queue sample to not use API key auth

* Reform pull queues to match appengine queues changes to auth, command line input, readme

* flake8 and fix comment

* Fix Tasks sample test issues.

* Remove queue.yaml, now unused

* Add required flag on certain arguments to command-line tools

* Auto-update dependencies. [(#1116)](GoogleCloudPlatform/python-docs-samples#1116)

* Auto-update dependencies. [(#1133)](GoogleCloudPlatform/python-docs-samples#1133)

* Auto-update dependencies.

* Fix missing http library

Change-Id: I99faa600f2f3f1f50f57694fc9835d7f35bda250

* Update Task Queue samples for beta

* Use full import to get client

* Auto-update dependencies. [(#1186)](GoogleCloudPlatform/python-docs-samples#1186)

* Auto-update dependencies. [(#1217)](GoogleCloudPlatform/python-docs-samples#1217)

* Added "Open in Cloud Shell" buttons to README files [(#1254)](GoogleCloudPlatform/python-docs-samples#1254)

* Add comments and region tags to Cloud Tasks samples [(#1271)](GoogleCloudPlatform/python-docs-samples#1271)

* Standardize on CamelCase, reword confusing endpoint name [(#1288)](GoogleCloudPlatform/python-docs-samples#1288)

* Auto-update dependencies. [(#1309)](GoogleCloudPlatform/python-docs-samples#1309)

* Rename pull to lease and fix name/parent confusion [(#1311)](GoogleCloudPlatform/python-docs-samples#1311)

* Updated region tags for pull queue samples [(#1329)](GoogleCloudPlatform/python-docs-samples#1329)

* Auto-update dependencies. [(#1320)](GoogleCloudPlatform/python-docs-samples#1320)

* Auto-update dependencies. [(#1355)](GoogleCloudPlatform/python-docs-samples#1355)

* Auto-update dependencies. [(#1359)](GoogleCloudPlatform/python-docs-samples#1359)

* Auto-update dependencies.

* Update Cloud Tasks Samples [(#1529)](GoogleCloudPlatform/python-docs-samples#1529)

* passing create task

* Passing tests

* updates to region tags

* update region tags [(#1532)](GoogleCloudPlatform/python-docs-samples#1532)

* update Tasks Sample for App Engine [(#1541)](GoogleCloudPlatform/python-docs-samples#1541)

* update gcloud command for creating queues

* deploys and runs

* update license

* passing tests

* Fix run command [(#1563)](GoogleCloudPlatform/python-docs-samples#1563)

* Updated library for TTS GA [(#1552)](GoogleCloudPlatform/python-docs-samples#1552)

* update gcloud command for Cloud Tasks [(#1566)](GoogleCloudPlatform/python-docs-samples#1566)

* update gcloud command

* update pull queue command

* update pull queue command

* Update Cloud Tasks Push Queue Sample [(#1698)](GoogleCloudPlatform/python-docs-samples#1698)

* deleted pull queues

* updated samples

* fix dependency versions

* [Cloud Tasks] Move samples to new folder [(#2114)](GoogleCloudPlatform/python-docs-samples#2114)

* Move samples to keep consistent with other langauges

* Ad system tests as well

* [Cloud Tasks] Add task with authentication sample [(#2113)](GoogleCloudPlatform/python-docs-samples#2113)

* Add task with authentication sample

* Fix linting

* Fix linting

* Fix spacing

* Update tests with service account

* Move samples and update READMEs

* Update version and linting

* Update task sample comments [(#2156)](GoogleCloudPlatform/python-docs-samples#2156)

* Update task comments

* Update readme

* Update queue name

* update gcloud [(#2208)](GoogleCloudPlatform/python-docs-samples#2208)

* Add protobuf dep and install instructions [(#2250)](GoogleCloudPlatform/python-docs-samples#2250)

* Add Migration Guide Snippets for Cloud Tasks [(#2316)](GoogleCloudPlatform/python-docs-samples#2316)

* Migration guide

* remove app and update migraitonn

* snippets for migration guide - tests added

* lint

* remove print statements

* Styling changes

* Travis trigger

* Update create_http_task.py [(#2187)](GoogleCloudPlatform/python-docs-samples#2187)

Updates `create_http_task.py` to have missing `in_seconds` variable

* Adds updates for samples profiler ... vision [(#2439)](GoogleCloudPlatform/python-docs-samples#2439)

* Update Cloud Tasks library version [(#2516)](GoogleCloudPlatform/python-docs-samples#2516)

* Update to new library

* update library version

* Adds Task name while creating Task [(#2543)](GoogleCloudPlatform/python-docs-samples#2543)

* Task Name in Creating HTTP Task with Token [(#2700)](GoogleCloudPlatform/python-docs-samples#2700)

* chore(deps): update dependency google-cloud-tasks to v1.5.0 [(#3168)](GoogleCloudPlatform/python-docs-samples#3168)

* chore(deps): update dependency googleapis-common-protos to v1.51.0 [(#3171)](GoogleCloudPlatform/python-docs-samples#3171)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [googleapis-common-protos](https://togithub.com/googleapis/googleapis) | minor | `==1.6.0` -> `==1.51.0` |

---

### Renovate configuration

:date: **Schedule**: At any time (no schedule defined).

:vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

:recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox.

:no_bell: **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

* Simplify noxfile setup. [(#2806)](GoogleCloudPlatform/python-docs-samples#2806)

* chore(deps): update dependency requests to v2.23.0

* Simplify noxfile and add version control.

* Configure appengine/standard to only test Python 2.7.

* Update Kokokro configs to match noxfile.

* Add requirements-test to each folder.

* Remove Py2 versions from everything execept appengine/standard.

* Remove conftest.py.

* Remove appengine/standard/conftest.py

* Remove 'no-sucess-flaky-report' from pytest.ini.

* Add GAE SDK back to appengine/standard tests.

* Fix typo.

* Roll pytest to python 2 version.

* Add a bunch of testing requirements.

* Remove typo.

* Add appengine lib directory back in.

* Add some additional requirements.

* Fix issue with flake8 args.

* Even more requirements.

* Readd appengine conftest.py.

* Add a few more requirements.

* Even more Appengine requirements.

* Add webtest for appengine/standard/mailgun.

* Add some additional requirements.

* Add workaround for issue with mailjet-rest.

* Add responses for appengine/standard/mailjet.

Co-authored-by: Renovate Bot <bot@renovateapp.com>

* Remove dependency googleapis-common-protos [(#3955)](GoogleCloudPlatform/python-docs-samples#3955)

* Update dependency googleapis-common-protos to v1.52.0

* Update requirements.txt

* Update requirements.txt

* Update requirements.txt

Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>

* Replace GCLOUD_PROJECT with GOOGLE_CLOUD_PROJECT. [(#4022)](GoogleCloudPlatform/python-docs-samples#4022)

* [tasks] testing: use fixtures for the queue [(#4049)](GoogleCloudPlatform/python-docs-samples#4049)

fixes #4045
fixes #4044

I don't know why these tests started to fail, but anyways we'd better
use fixtures and temporary queues.

* add python snippets and tests for creating, listing, and deleting queues [(#4012)](GoogleCloudPlatform/python-docs-samples#4012)

* add python snippets and tests for creating, listing, and deleting queues

* fix grammar

* update licenses

* apply suggested fixes and format with black

* refine delete_queue_test with fixture for setup

* utilize fixtures and match format of create_http_task_test

* utilize fixtures in list_queues_test and create_queue_test

* make create_queue_test call the right function

* still attempt to delete queue after test runs in case of failure

* attempt to delete queue in case of failure, using try/except approach

* add print when NotFound is caught

* fix import

Co-authored-by: Averi Kitsch <akitsch@google.com>
Co-authored-by: Takashi Matsuo <tmatsuo@google.com>

* docs(tasks): service_account_email parameter example [(#4183)](GoogleCloudPlatform/python-docs-samples#4183)

* docs(tasks): serviceAccountEmail parameter example

* docs(tasks): parameter name camel => snake case

* docs(tasks): comment order = function param order

* chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloudPlatform/python-docs-samples#4279)

* chore(deps): update dependency pytest to v5.4.3

* specify pytest for python 2 in appengine

Co-authored-by: Leah Cole <coleleah@google.com>

* Update dependency pytest to v6 [(#4390)](GoogleCloudPlatform/python-docs-samples#4390)

* tasks: added json content-type request [(#4473)](GoogleCloudPlatform/python-docs-samples#4473)

- added json payload compatibility
- fix imports and code block used on https://cloud.google.com/tasks/docs/creating-http-target-tasks#python

## Description

Fixes #<ISSUE-NUMBER>

Note: It's a good idea to open an issue first for discussion.

## Checklist
- [ ] I have followed [Sample Guidelines from AUTHORING_GUIDE.MD](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md)
- [ ] README is updated to include [all relevant information](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#readme-file)
- [ ] **Tests** pass:   `nox -s py-3.6` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup))
- [ ] **Lint** pass:   `nox -s lint` (see [Test Environment Setup](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/AUTHORING_GUIDE.md#test-environment-setup))
- [ ] These samples need a new **API enabled** in testing projects to pass (let us know which ones)
- [ ] These samples need a new/updated **env vars** in testing projects set to pass (let us know which ones)
- [ ] Please **merge** this PR for me once it is approved.
- [ ] This sample adds a new sample directory, and I updated the [CODEOWNERS file](https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/.github/CODEOWNERS) with the codeowners for this sample

* docs: add samples from python-docs-samples/tasks

* chore: update templates

Co-authored-by: Andrew Gorcester <andrew.gorcester@gmail.com>
Co-authored-by: Andrew Gorcester <gorcester@google.com>
Co-authored-by: DPE bot <dpebot@google.com>
Co-authored-by: michaelawyu <chenyumic@google.com>
Co-authored-by: ellenevans <35748459+ellenevans@users.noreply.github.com>
Co-authored-by: Averi Kitsch <akitsch@google.com>
Co-authored-by: michaelawyu <michael.a.w.yu@hotmail.com>
Co-authored-by: Noah Negrey <nnegrey@users.noreply.github.com>
Co-authored-by: Alex Voorhees <alex@cloudbakers.com>
Co-authored-by: Gus Class <gguuss@gmail.com>
Co-authored-by: Sarath Kaul <kaul.sarath@gmail.com>
Co-authored-by: Sarath Kaul <sarath.kaul@searce.com>
Co-authored-by: WhiteSource Renovate <bot@renovateapp.com>
Co-authored-by: Kurtis Van Gent <31518063+kurtisvg@users.noreply.github.com>
Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
Co-authored-by: Takashi Matsuo <tmatsuo@google.com>
Co-authored-by: Aaron Johnson <aaronmjohnson29@gmail.com>
Co-authored-by: Adam Ross <adamross@google.com>
Co-authored-by: Leah Cole <coleleah@google.com>
Co-authored-by: Joab Leite S. Neto <leitejoab@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the Datastore API.
Projects
None yet
Development

No branches or pull requests

4 participants