|
1 |
| -# azure-pipelines-task-template |
| 1 | +# Azure DevOps Databricks REST API |
2 | 2 |
|
3 |
| - |
| 3 | + |
4 | 4 |
|
5 | 5 | ## Introduction
|
6 | 6 |
|
7 |
| -This repository contains a quick start template for creating and publishning a custom azure pipelines task written in TypeScript. It uses [webpack](https://webpack.js.org) to bundle scripts and is configured to use [jest](https://jestjs.io/) to run tests. |
| 7 | +This repository contains an Azure DevOps extension for interacting with Azure Databricks via [REST API](https://docs.databricks.com/dev-tools/api/latest/index.html). It supports Databricks management on clusters, jobs, and instance pools. |
8 | 8 |
|
9 |
| -## Getting started |
10 |
| - |
11 |
| -There are a few small steps that need to be carried out before you can start using this template for your purpose- |
| 9 | +You may find this extension **useful** when: |
| 10 | +- You are running Spark (structured) streaming jobs attached to automated clusters. |
| 11 | +- You are leveraging instance pools for [fast startup time](https://databricks.com/blog/2019/11/11/databricks-pools-speed-up-data-pipelines.html) and scalability. |
12 | 12 |
|
13 |
| -1. Create two guids and replace `{{guid_production}}` and `{{guid_development}}` occurences with these values in webpack [config](webpack.config.js). You can use [this](https://www.guidgenerator.com) tool to generate guids. *(It helps to have different guids for different environments as it allows you to install multiple extension flavors in the same account)* |
| 13 | +You may find this extension **useless** when: |
| 14 | +- You are running Spark jobs on an interactive clusters, or running on scheduled manner. |
| 15 | +- You are running jobs via Jupyter notebooks. |
14 | 16 |
|
15 |
| -1. Enter your publisher and author name in place of `{{publisher}}` and `{{author}}` strings. These files have references to these strings - [webpack config](webpack.config.js), [task.json](./src/custom-task/task.json) and [manifest.json](./src/manifest.json) |
| 17 | +## Getting started |
16 | 18 |
|
17 |
| -1. Create a personal access token for the Azure Devops account where you would test run your task. Refer [this](https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/use-personal-access-tokens-to-authenticate?view=azure-devops) to learn how to create a personal access token. |
| 19 | +- Install an extension via <https://marketplace.visualstudio.com/items?itemName=raksit31667.azure-devops-databricks-rest-api>. |
| 20 | +- Add Azure DevOps task similar as a code snippet below: |
18 | 21 |
|
19 |
| -1. Set following environment variables to enable inner loop commands- |
20 |
| - - `BUILD_ENV` = **development** |
21 |
| - - `ADO_PAT` - Your personal access token |
22 |
| - - `ADO_ACCOUNT_URI` - The default collection URI for your account *(eg. <https://dev.azure.com/{{name}})>* |
23 |
| - - `DEV_PUBLISHER` - Id of your publisher account |
| 22 | +```yaml |
| 23 | +- task: VerifyAllDatabricksJobRunning@1 |
| 24 | +displayName: Verify Databricks jobs |
| 25 | +inputs: |
| 26 | + jobs: '<your-job-name-here>' |
| 27 | + region: 'your-region-here>' |
| 28 | + accessToken: '<your-databricks-access-token>' |
| 29 | +``` |
24 | 30 |
|
25 |
| -1. Run following commands to verify setup- |
26 |
| - 1. `npm install` (Installs npm dependencies) |
27 |
| - 1. `npm run dev` (Builds, packages and publishes the extension) |
| 31 | +Note that every task requires **region** and **access token**. You can find a region from your |
| 32 | +Databricks workspace URL (e.g. <https://southcentralus.azuredatabricks.net/?o=0000000>). For access token, you can generate via **User Settings**, see an instruction <https://docs.databricks.com/dev-tools/api/latest/authentication.html>. |
28 | 33 |
|
29 |
| -1. Refer to [package.json](./package.json) to see the list of all commands. |
| 34 | +## Contributing |
| 35 | +Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. |
30 | 36 |
|
31 |
| -## Typical inner loop commands |
| 37 | +Please make sure to update tests as appropriate. |
32 | 38 |
|
33 |
| -| What changed? | Which NPM command to run? | What it does? | |
34 |
| -| ------------- |:-------------:|:----- | |
35 |
| -| Task implementation | `npm run dev:task` | Build, package and update policy task without updating extension | |
36 |
| -| Anything else | `npm run dev` | Build, package and publish full extension | |
| 39 | +## License |
| 40 | +[MIT](https://choosealicense.com/licenses/mit/) |
0 commit comments