Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Provide support for scraping Azure Synapse workspace, Apache Spark & SQL pools #1513

Merged
merged 11 commits into from
Feb 18, 2021

Conversation

tomkerkhove
Copy link
Owner

@tomkerkhove tomkerkhove commented Feb 13, 2021

Signed-off-by: Tom Kerkhove kerkhove.tom@gmail.com

When implementing a new scraper; these tasks are completed:

  • Implement configuration
  • Implement validation
  • Implement scraping
  • Implement resource discovery
  • Provide unit tests
  • Test end-to-end
  • Document scraper
  • Add entry to changelog

Metrics output:

# HELP promitor_demo_synapse_apache_spark_apps_ended Amount of apps ended running on Apache Spark pool in Azure Synapse
# TYPE promitor_demo_synapse_apache_spark_apps_ended gauge
promitor_demo_synapse_apache_spark_apps_ended{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.Synapse/workspaces/promitor-synapse/bigDataPools/sparkpool",instance_name="sparkpool",workspace_name="promitor-synapse",pool_name="sparkpool"} 0 1613500949918
# HELP promitor_demo_synapse_sql_pool_dwu_limit Amount of DWUs defined as limit for SQL pool in Azure Synapse
# TYPE promitor_demo_synapse_sql_pool_dwu_limit gauge
promitor_demo_synapse_sql_pool_dwu_limit{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.Synapse/workspaces/promitor-synapse/sqlPools/sqlpool",instance_name="sqlpool",workspace_name="promitor-synapse",pool_name="sqlpool"} 100 1613500949918
# HELP promitor_demo_synapse_workspace_builtin_sql_processed_bytes Amount of bytes processed in Azure Synapse workspace
# TYPE promitor_demo_synapse_workspace_builtin_sql_processed_bytes gauge
promitor_demo_synapse_workspace_builtin_sql_processed_bytes{resource_group="promitor-sources",subscription_id="0f9d7fea-99e8-4768-8672-06a28514f77e",resource_uri="subscriptions/0f9d7fea-99e8-4768-8672-06a28514f77e/resourceGroups/promitor-sources/providers/Microsoft.Synapse/workspaces/promitor-synapse",instance_name="promitor-synapse",workspace_name="promitor-synapse"} 0 1613500949885

Discovery output:

[
  {
    "$type": "Promitor.Core.Contracts.ResourceTypes.SynapseApacheSparkPoolResourceDefinition, Promitor.Core.Contracts",
    "WorkspaceName": "promitor-synapse",
    "PoolName": "sparkpool",
    "ResourceType": "SynapseApacheSparkPool",
    "SubscriptionId": "0f9d7fea-99e8-4768-8672-06a28514f77e",
    "ResourceGroupName": "promitor-sources",
    "ResourceName": "sparkpool",
    "UniqueName": "promitor-synapse-sparkpool"
  }
]

Fixes #1477

Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove tomkerkhove changed the title feat: Provide support for scraping Azure Synapse feat: Provide support for scraping Azure Synapse workspace, Apache Spark & SQL pools Feb 13, 2021
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove tomkerkhove marked this pull request as ready for review February 18, 2021 06:26
Signed-off-by: Tom Kerkhove <kerkhove.tom@gmail.com>
@tomkerkhove tomkerkhove merged commit 3acf6c2 into master Feb 18, 2021
@tomkerkhove tomkerkhove deleted the synapse-scrapers branch February 18, 2021 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide support for scraping Azure Synapse
1 participant