Nuagecron is designed to answer the "How do I schedule my cloud based tasks?" Question. While there are solutions out there right now none of them meet the "serverless first" approach.
AWS Cloudwatch events always seemed lacking. You couldn't investigate failures to invoke, couldn't invoke them in an adhoc manner and couldn't have more than 200 of them in an account. Airflow is a great tool for building DAGs, but it isn't really a scheduler at heart, it has scheduling functionality, but adding a schedule requires modifications of the DAG folder.
The idea behind nuagecron is to create a cloud native cron service, with execution tracking, namespace support (was call them project_stacks), support for hundreds of tasks and a low operational cost. This really boils down to "I want cron to run in the cloud and kick off the resources I put in there". So nuagecron is built to provide a "scheduler as a service" to teams, where you define and deploy your resources to your cloud and then submit a schedule to nuagecron to kick those tasks off and monitor them (Some assmebly required, to be detailed later). This solution is capable of supporting multiple teams without conflicts and will eventually provide a visualization solution in the form of a web page.
You will need a few things to install this:
poetry installyarn install
As of right now only AWS deployments are supported. In the future we will try to support multiple cloud providers and Kubernetes but we want to walk before we try to run here. At some point I will try to transition this to Terraform to help abstract some of this away.
- Set up your local AWS developer credentials
- Run
./deploy.sh(this should install things but serverless currently has an issue with spinning up multiple GSIs on Dynamo tables)
As of right now this is a WIP as I try to learn React. It is set up so that you may start the web UI using yarn start and the backend using flask run. The web page will try to load any schedules it finds in the database.
(Template from the folks at serverless-wsgi)
This template configures a single function, api, which is responsible for handling all incoming requests thanks to configured http events. To learn more about http event configuration options, please refer to http event docs. As the events are configured in a way to accept all incoming requests, Flask framework is responsible for routing and handling requests internally. The implementation takes advantage of serverless-wsgi, which allows you to wrap WSGI applications such as Flask apps. To learn more about serverless-wsgi, please refer to corresponding GitHub repository. The template also relies on serverless-python-requirements plugin for packaging dependencies from requirements.txt file. For more details about serverless-python-requirements configuration, please refer to corresponding GitHub repository.
Additionally, the template also handles provisioning of a DynamoDB database that is used for storing data about users. The Flask application exposes two endpoints, POST /users and GET /user/{userId}, which allow to create and retrieve users.
In order to package your dependencies locally with serverless-python-requirements, you need to have Python3.8 installed locally. You can create and activate a dedicated virtual environment with the following command:
python3.8 -m venv ./venv
source ./venv/bin/activateAlternatively, you can also use dockerizePip configuration from serverless-python-requirements. For details on that, please refer to corresponding GitHub repository.
This example is made to work with the Serverless Framework dashboard, which includes advanced features such as CI/CD, monitoring, metrics, etc.
In order to deploy with dashboard, you need to first login with:
serverless login
install dependencies with:
npm install
and then perform deployment with:
serverless deploy
After running deploy, you should see output similar to:
Serverless: Using Python specified in "runtime": python3.8
Serverless: Packaging Python WSGI handler...
Serverless: Generated requirements from /home/xxx/xxx/xxx/examples/aws-python-flask-dynamodb-api/requirements.txt in /home/xxx/xxx/xxx/examples/aws-python-flask-dynamodb-api/.serverless/requirements.txt...
Serverless: Using static cache of requirements found at /home/xxx/.cache/serverless-python-requirements/62f10436f9a1bb8040df30ef2db5736c8015b18256bf0b6f1b0cbb2640030244_slspyc ...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Injecting required Python packages to package...
Serverless: Creating Stack...
Serverless: Checking Stack create progress...
........
Serverless: Stack create finished...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service aws-python-flask-dynamodb-api.zip file to S3 (1.3 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
.................................
Serverless: Stack update finished...
Service Information
service: aws-python-flask-dynamodb-api
stage: dev
region: us-east-1
stack: aws-python-flask-dynamodb-api-dev
resources: 12
api keys:
None
endpoints:
ANY - https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/
ANY - https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/{proxy+}
functions:
api: aws-python-flask-dynamodb-api-dev-api
layers:
NoneNote: In current form, after deployment, your API is public and can be invoked by anyone. For production deployments, you might want to configure an authorizer. For details on how to do that, refer to http event docs.
After successful deployment, you can create a new user by calling the corresponding endpoint:
curl --request POST 'https://xxxxxx.execute-api.us-east-1.amazonaws.com/dev/users' --header 'Content-Type: application/json' --data-raw '{"name": "John", "userId": "someUserId"}'Which should result in the following response:
{"userId":"someUserId","name":"John"}You can later retrieve the user by userId by calling the following endpoint:
curl https://xxxxxxx.execute-api.us-east-1.amazonaws.com/dev/users/someUserIdWhich should result in the following response:
{"userId":"someUserId","name":"John"}If you try to retrieve user that does not exist, you should receive the following response:
{"error":"Could not find user with provided \"userId\""}Thanks to capabilities of serverless-wsgi, it is also possible to run your application locally, however, in order to do that, you will need to first install werkzeug, boto3 dependencies, as well as all other dependencies listed in requirements.txt. It is recommended to use a dedicated virtual environment for that purpose. You can install all needed dependencies with the following commands:
pip install werkzeug boto3
pip install -r requirements.txtAdditionally, you will need to emulate DynamoDB locally, which can be done by using serverless-dynamodb-local plugin. In order to do that, execute the following commands:
serverless plugin install -n serverless-dynamodb-local
serverless dynamodb installIt will add the plugin to devDependencies in package.json file as well as to plugins section in serverless.yml. Additionally, it will also install DynamoDB locally.
You should also add the following config to custom section in serverless.yml:
custom:
(...)
dynamodb:
start:
migrate: true
stages:
- devAdditionally, we need to reconfigure DynamoDB Client to connect to our local instance of DynamoDB. We can take advantage of IS_OFFLINE environment variable set by serverless-wsgi plugin and replace:
dynamodb_client = boto3.client('dynamodb')with
dynamodb_client = boto3.client('dynamodb')
if os.environ.get('IS_OFFLINE'):
dynamodb_client = boto3.client('dynamodb', region_name='localhost', endpoint_url='http://localhost:8000')Now you can start DynamoDB local with the following command:
serverless dynamodb startAt this point, you can run your application locally with the following command:
serverless wsgi serveFor additional local development capabilities of serverless-wsgi and serverless-dynamodb-local plugins, please refer to corresponding GitHub repositories: