tap-grafana

tap-grafana is a Singer tap for extracting data from Grafana. It leverages the Grafana API to pull data into your data warehouse or data lake.

Built with the Meltano Tap SDK for Singer Taps.

Installation

1. Adding Tap-Grafana to an Existing Meltano Project

To add the tap-grafana to an existing Meltano project, follow these steps:

1. Navigate to your Meltano project directory: Open your terminal and change to the directory of your Meltano project.

    cd your-meltano-project

2. Add the Tap-Grafana extractor: Use the meltano add command to add tap-grafana to your project.

   meltano add extractor tap-grafana

   or update your meltano.yml file with below configuration

   plugins:
      extractors:
        - name: tap-grafana
          namespace: tap_grafana
          pip_url: git+https://github.com/degreed-data-engineering/tap-grafana
          config:
            api_key: <API Key for Grafana OnCall module API service>
            api_base_url: <Base url for the Grafana OnCall module API service>

3. Configure the Tap-Grafana extractor: After adding the extractor, you need to configure it. You can do this interactively by running:

    meltano config tap-grafana set --interactive

   Or, you can set the config environment variable in your .env file. For example:

    TAP_GRAFANA_API_KEY="your_api_key_here"
    TAP_GRAFANA_API_BASE_URL="https://oncall-prod-us-central-0.grafana.net/oncall"

4. Test the Tap-Grafana extractor configuration: To ensure everything is configured correctly, test the configuration using:

    meltano config tap-grafana test

5. Run the Extractor: Finally, run the extractor to start pulling data from Grafana into your Meltano project. You can specify the target loader in the command. For example, if you're using target-jsonl as your loader:

    meltano run tap-grafana target-jsonl

By following these steps, you will have successfully added tap-grafana to your existing Meltano project, configured it with your Grafana API key, and started extracting data.

2. Install from GitHub:

pipx install git+https://github.com/degreed-data-engineering/tap-grafana.git

Configuration

tap-grafana requires an API Base URL and an API key to connect and authenticate with the Grafana OnCall module APIs. These are mandatory configurations.

• api_key: This is your Grafana API key.
• api_base_url: Base url for the Grafana OnCall module API service.

You can set this API key in your environment variables:

export TAP_GRAFANA_API_KEY=your_api_key_here
export TAP_GRAFANA_API_BASE_URL="https://oncall-prod-us-central-0.grafana.net/oncall"

Alternatively, you can create a .env file in your project directory and add the following line:

TAP_GRAFANA_API_KEY=your_api_key_here
TAP_GRAFANA_API_BASE_URL="https://oncall-prod-us-central-0.grafana.net/oncall"

Configure using environment variables

This Meltano tap will automatically import any environment variables within the working directory's .env if the --config=ENV is provided, such that config values will be considered if a matching environment variable is set either in the terminal context or in the .env file.

Accepted Config Options

A full list of supported settings and capabilities for this tap is available by running:

tap-grafana --about

Pre-Requisite tor run above command

1. Install the Tap Grafana: If you haven't already installed the tap-grafana, you need to do so. The installation method can vary depending on whether tap-grafana is a standalone tool or part of a larger framework. If it's a Python package, you might use pip to install it:

  pipx install git+https://github.com/degreed-data-engineering/tap-grafana.git@main

Usage

You can easily run tap-grafana by itself or in a pipeline using Meltano.

Executing the Tap Directly

tap-grafana --version
tap-grafana --help
tap-grafana --config CONFIG --discover > ./catalog.json

Executing the Tap Within A Meltano Project

Use the meltano config command to list the settings your extractor supports:

meltano config tap-grafana list

To set the appropriate values for each setting using the meltano config command:

meltano config tap-grafana set <setting> <value>

or

meltano config tap-grafana set --interactive

If you encounter issues or need to verify the configuration, you can use the meltano config command to test the extractor settings:

meltano config tap-grafana test

Developer Resources

Follow these instructions to contribute to this project.

Initialize your Development Environment

pipx install poetry
poetry install

Create and Run Tests

Create tests within the tests subfolder and then run:

poetry run pytest

You can also test the tap-grafana CLI interface directly using poetry run:

poetry run tap-grafana --help

Testing with Meltano

Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.

Next, install Meltano (if you haven't already) and any needed plugins:

# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-grafana
meltano install

Now you can test and orchestrate using Meltano:

# Test invocation:
meltano invoke tap-grafana --version
# OR run a test `elt` pipeline:
meltano elt tap-grafana target-jsonl

SDK Dev Guide

See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.