-
Notifications
You must be signed in to change notification settings - Fork 6.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add samples for Cloud Tasks #1068
Changes from 1 commit
87c721f
507a54b
1a43939
ee920fb
fa4270c
4d1f228
5e7a379
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,21 @@ | ||
# Google Cloud Tasks Samples | ||
|
||
Sample program for interacting with the Google Cloud Tasks API. | ||
|
||
The `pull_queues` directory contains command line samples for listing queues, creating | ||
tasks, pulling tasks, and acknowledging them. | ||
|
||
The `appengine_queues` directory contains an App Engine app and command line | ||
samples for creating tasks to be pushed to the App Engine app. | ||
|
||
See the respective READMEs for detailed instructions. | ||
|
||
## Contributing changes | ||
|
||
Contributions are not accepted during Alpha. | ||
|
||
## Licensing | ||
|
||
* See [LICENSE](LICENSE) | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
# Google Cloud Tasks App Engine Queue Samples | ||
|
||
Sample command-line program for interacting with the Cloud Tasks API | ||
using App Engine queues. | ||
|
||
App Engine queues push tasks to an App Engine HTTP target. This directory | ||
contains both the App Engine app to deploy, as well as the snippets to run | ||
locally to push tasks to it, which could also be called on App Engine. | ||
|
||
`app_engine_queue_snippets.py` is a simple command-line program to create tasks | ||
to be pushed to the App Engine app. | ||
|
||
`main.py` is the main App Engine app. This app serves as an endpoint to receive | ||
App Engine task attempts. | ||
|
||
`app.yaml` configures the App Engine app. | ||
|
||
|
||
## Prerequisites to run locally: | ||
|
||
The samples require a Python environment with | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you just link to (I really need to finish the autogenerator for web samples) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
[pip](https://pypi.python.org/pypi/pip) installed. | ||
[virtualenv](https://virtualenv.readthedocs.org/en/latest/) is also recommended. | ||
|
||
All samples require a Google Cloud Project whitelisted for the Cloud Tasks API. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is whitelisting required in beta? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
* Enable the Cloud Tasks API | ||
* Enable the Cloud Datastore API (used to demonstrate storing payloads) | ||
|
||
To create a project and enable the API, go to the [Google Developers | ||
Console](https://console.developer.google.com). You must also create an API key. | ||
This can be done under API Manager -> Credentials. | ||
|
||
To install the Python application dependencies, run the following commands: | ||
|
||
* pip install -r requirements.txt | ||
|
||
## Authentication | ||
|
||
To set up authentication locally, download the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you just link to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. On hold until I rewrite to use service account auth (shortly) |
||
[Cloud SDK](https://cloud.google.com/sdk), and run | ||
|
||
gcloud auth application-default login | ||
|
||
On App Engine, authentication credentials will be automatically detected. | ||
|
||
On Compute Engine and Container Engine, authenticatino credentials will be | ||
automatically detected, but the instances must have been created with the | ||
necessary scopes. | ||
|
||
In any other environment, for example Compute Engine instance without the | ||
necessary scopes, you should set `GOOGLE_APPLICATION_CREDENTIALS` environment | ||
variable to a JSON key file for a service account. | ||
|
||
See the [authentication guide](https://cloud.google.com/docs/authentication) | ||
for more information. | ||
|
||
## Creating a queue | ||
|
||
Queues can not currently be created by the API. To create a queue using the | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this still true in beta? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, it is not. Changing the samples to reflect this is coming up but for now I'll at least scrub this from the docs. Done |
||
Cloud SDK, use the provided queue.yaml: | ||
|
||
gcloud app deploy queue.yaml | ||
|
||
## Deploying the App Engine app | ||
|
||
First, vendor the dependencies into the project: | ||
|
||
pip install -r requirements.txt | ||
|
||
Next, deploy the App Engine app | ||
|
||
gcloud app deploy | ||
|
||
Verify the index page is serving: | ||
|
||
gcloud app browse | ||
|
||
The App Engine app serves as a target for the push requests. It has an | ||
endpoint `/set_payload` that that stores the payload from the HTTP POST data in | ||
Cloud Datastore. The payload can be accessed in your browser at the | ||
`/get_payload` endpoint with a GET request. | ||
|
||
## Running the Samples | ||
|
||
The project ID must be specified either as a command line argument using | ||
`--project-id`, or by editing `DEFAULT_PROJECT_ID` within `task_snippets.py`. | ||
|
||
Set the environment variables: | ||
|
||
export PROJECT_ID=my-project-id | ||
export LOCATION_ID=us-central1 | ||
export QUEUE_ID=my-appengine-queue # From queue.yaml | ||
|
||
View all queues: | ||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY list-queues --project_id=$PROJECT_ID --location_id=$LOCATION_ID | ||
|
||
Set the queue name as an environment variable: | ||
|
||
export QUEUE_NAME=projects/$PROJECT_ID/locations/$LOCATION_ID/queues/$QUEUE_ID | ||
|
||
Create a task, targeted at the `set_payload` endpoint with a payload specified: | ||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello | ||
|
||
Now view that the payload was received and verify the count and payload: | ||
|
||
http://your-app-id.appspot.com/get_payload | ||
|
||
Create a task that will be scheduled for a few seconds in the future using | ||
the `--in_seconds` flag: | ||
|
||
python app_engine_queue_snippets.py --api_key=$API_KEY create-task --queue_name=$QUEUE_NAME --payload=hello --in_seconds=30 | ||
|
||
Since `--in_seconds` was set to 30, it will take 30 seconds for the new | ||
payload to be pushed to the `/get_payload` endpoint, which can then be viewed at: | ||
|
||
http://your-app-id.appspot.com/get_payload | ||
|
||
It might also be helpful to view the request logs of your App Engine app: | ||
|
||
https://console.cloud.google.com/logs | ||
|
||
## Testing the Samples | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Drop this section There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
Install the testing dependencies: | ||
|
||
pip install -r requirements-test.txt | ||
|
||
Run pytest: | ||
|
||
pytest |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,6 @@ | ||
runtime: python | ||
env: flex | ||
entrypoint: gunicorn -b :$PORT main:app | ||
|
||
runtime_config: | ||
python_version: 3 |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,144 @@ | ||
# Copyright 2016 Google Inc. All Rights Reserved. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import argparse | ||
import base64 | ||
import datetime | ||
|
||
from googleapiclient import discovery | ||
|
||
|
||
def format_rfc3339(datetime_instance): | ||
"""Format a datetime per RFC 3339.""" | ||
return datetime_instance.isoformat("T") + "Z" | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. use single quotes. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
|
||
def get_seconds_from_now_rfc3339(seconds): | ||
"""Return seconds from the current time as a RFC 3339 string.""" | ||
d = datetime.datetime.utcnow() + datetime.timedelta(seconds=seconds) | ||
return format_rfc3339(d) | ||
|
||
|
||
def list_queues(api_key, project_id, location_id): | ||
"""List the queues in the location.""" | ||
client = get_client(api_key) | ||
parent = 'projects/{}/locations/{}'.format(project_id, location_id) | ||
queues = [] | ||
next_page_token = None | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The apiary client should automatically create a |
||
|
||
while True: | ||
response = client.projects().locations().queues().list( | ||
parent=parent, pageToken=next_page_token).execute() | ||
queues += response['queues'] | ||
if next_page_token is None: | ||
break | ||
|
||
print('Listing queues for location {}'.format(location_id)) | ||
|
||
for queue in response['queues']: | ||
print queue['name'] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. print is a function. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done (I also added from future import print_function as I prefer to include that for reasons I've mentioned previously) |
||
return response | ||
|
||
|
||
def create_task(api_key, queue_name, payload=None, in_seconds=None): | ||
"""Create a task for a given queue with an arbitrary payload.""" | ||
client = get_client(api_key) | ||
|
||
url = '/set_payload' | ||
task = { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Suggestion: use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
'task': { | ||
'app_engine_task_target': { | ||
'http_method': 'POST', | ||
'relative_url': url | ||
} | ||
} | ||
} | ||
|
||
if payload is not None: | ||
task['task']['app_engine_task_target']['payload'] = base64.b64encode( | ||
payload) | ||
|
||
if in_seconds is not None: | ||
scheduled_time = get_seconds_from_now_rfc3339(int(in_seconds)) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
task['task']['schedule_time'] = scheduled_time | ||
|
||
print('Sending task {}'.format(task)) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. use pprint so that this looks kind of pretty? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Done |
||
|
||
response = client.projects().locations().queues().tasks().create( | ||
parent=queue_name, body=task).execute() | ||
|
||
# By default CreateTaskRequest.responseView is BASIC, so not all | ||
# information is retrieved by default because some data, such as payloads, | ||
# might be desirable to return only when needed because of its large size | ||
# or because of the sensitivity of data that it contains. | ||
print('Created task {}'.format(response['name'])) | ||
return response | ||
|
||
|
||
def get_client(api_key): | ||
"""Build an authenticated http client.""" | ||
DISCOVERY_URL = 'https://cloudtasks.googleapis.com/$discovery/rest?version=v2beta2&key={}'.format( | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. is this discovery url needed any more for beta? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It's needed right at this moment, so this will have to be one of the last things to change. |
||
api_key) | ||
client = discovery.build('cloudtasks', 'v2beta2', | ||
discoveryServiceUrl=DISCOVERY_URL) | ||
return client | ||
|
||
|
||
if __name__ == '__main__': | ||
parser = argparse.ArgumentParser( | ||
description=__doc__, | ||
formatter_class=argparse.RawDescriptionHelpFormatter) | ||
|
||
parser.add_argument('--api_key', help='API Key', required=True) | ||
|
||
subparsers = parser.add_subparsers(dest='command') | ||
|
||
list_queues_parser = subparsers.add_parser( | ||
'list-queues', | ||
help=list_queues.__doc__) | ||
|
||
list_queues_parser.add_argument( | ||
'--project_id', | ||
help='Project ID you want to access.', | ||
required=True) | ||
|
||
list_queues_parser.add_argument( | ||
'--location_id', | ||
help='Location of the queues.', | ||
required=True) | ||
|
||
create_task_parser = subparsers.add_parser( | ||
'create-task', | ||
help=create_task.__doc__) | ||
create_task_parser.add_argument( | ||
'--queue_name', | ||
help='Fully qualified name of the queue to add the task to.' | ||
) | ||
|
||
create_task_parser.add_argument( | ||
'--payload', | ||
help='Optional payload to attach to the push queue.' | ||
) | ||
|
||
create_task_parser.add_argument( | ||
'--in_seconds', | ||
help='The number of seconds from now to schedule task attempt.' | ||
) | ||
|
||
args = parser.parse_args() | ||
|
||
if args.command == 'list-queues': | ||
list_queues(args.api_key, args.project_id, args.location_id) | ||
if args.command == 'create-task': | ||
create_task(args.api_key, args.queue_name, args.payload, args.in_seconds) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
# Copyright 2016 Google Inc. All Rights Reserved. | ||
# | ||
# Licensed under the Apache License, Version 2.0 (the "License"); | ||
# you may not use this file except in compliance with the License. | ||
# You may obtain a copy of the License at | ||
# | ||
# http://www.apache.org/licenses/LICENSE-2.0 | ||
# | ||
# Unless required by applicable law or agreed to in writing, software | ||
# distributed under the License is distributed on an "AS IS" BASIS, | ||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
# See the License for the specific language governing permissions and | ||
# limitations under the License. | ||
|
||
import mock | ||
|
||
import app_engine_queue_snippets | ||
|
||
TEST_PROJECT_ID = 'mock-project' | ||
TEST_LOCATION = 'us-central1' | ||
API_KEY = 'mock-api-key' | ||
TEST_QUEUE_NAME = 'projects/{}/locations/{}/queues/{}'.format( | ||
TEST_PROJECT_ID, TEST_LOCATION, 'my-push-queue') | ||
|
||
|
||
@mock.patch('app_engine_queue_snippets.get_client') | ||
def test_list_queues(get_client): | ||
locations = get_client.return_value.projects.return_value.locations | ||
queues = locations.return_value.queues | ||
execute_function = queues.return_value.list.return_value.execute | ||
execute_function.return_value = {'name': 'task_name', 'queues': []} | ||
app_engine_queue_snippets.list_queues(API_KEY, TEST_PROJECT_ID, | ||
TEST_LOCATION) | ||
assert execute_function.called | ||
|
||
|
||
@mock.patch('app_engine_queue_snippets.get_client') | ||
def test_create_task(get_client): | ||
projects = get_client.return_value.projects.return_value | ||
locations = projects.locations.return_value | ||
create_function = locations.queues.return_value.tasks.return_value.create | ||
execute_function = create_function.return_value.execute | ||
execute_function.return_value = {'name': 'task_name'} | ||
app_engine_queue_snippets.create_task(API_KEY, TEST_QUEUE_NAME) | ||
assert execute_function.called | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can drop this section and the next.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done