Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentinel and Kusto new providers #656

Merged
merged 18 commits into from
May 10, 2023
Merged

Sentinel and Kusto new providers #656

merged 18 commits into from
May 10, 2023

Conversation

ianhelle
Copy link
Contributor

Sentinel and Kusto provider updates

This change adds replacement drivers for the MSSentinel
and Kusto data providers.
Instead of Kqlmagic these drivers use the azure-kusto-data
and azure-monitor-query SDKs, respectively.

Currently these drivers are enabled alongside the existing versions

  • in future versions we will make these
    the defaults for Sentinel and Kusto respectively and,
    ultimately, deprecate the older versions.

Some of the main changes with these new versions:

  • They the provider names MSSentinel_New and Kusto_New when creating a QueryProvider
    instance.
  • Both drivers support setting proxies for firewall-protected networks
  • Both drivers support custom configuration of the server timeout via
    a timeout parameter
  • Both drivers use integrated Azure authentication by default and
    support the auth_types and tenant_id parameters used elsewhere
    in MSTICPy
  • Some of the previous parameters have been deprecated:
    • mp_az_auth is replaced by auth_types (the former still works
      but will be removed in a future release).
    • mp_az_auth_tenant_id is replaced by tenant_id (the former
      is not supported in the new providers.

Note: in order to use these new versions you must have
the azure-kusto-data and/or azure-monitor-query Python packages
installed.

For more details on how to use these providers, see:

Changes specific to the Sentinel provider

  • It uses the provider name MSSentinel_New when creating a QueryProvider
    instance.
  • It supports simultaneous queries against multiple workspaces (see below).
  • Supports user-specified timeout for queries.

Changes specific to the Kusto provider

  • It uses the provider name Kusto_New when creating a QueryProvider
    instance.
  • The settings format has changed (although the existing format
    is still supported albeit with some limited functionality).
    See the Kusto provider documentation for details.
  • In the earlier implmentation of driver you can specify a new cluster to connect to in
    when executing a query. This is no longer supported. Once the
    provider is connected to a cluster it will only execute queries against
    that cluster. (You can however, call the connect() function to connect
    the provider to a new cluster before running the query.)
  • Filtering pre-defined queries by cluster. If you have MSTICPy query
    definitions for the Kusto provider, these will all be attached as methods
    the QueryProvider, when it is created. However, as soon as you connect
    to a specific cluster, the queries will be filtered down to show
    only the queries that are intended to run on that cluster.
  • New APIs (exposed via the query_provider):
    • get_database_names - return list of databases for the connected cluster
    • get_database_schema - return table schema for a database in the cluster
    • configured_clusters - return a list of clusters configured in
      msticpyconfig.yaml
    • set_cluster - switch connected to cluster to a different one (you
      can use the connect method to do this, which also lets you specify
      additional connection parameters).

Related changes

  • Proxy support added to MSTICPy - this is currently only used by
    the Kusto and Sentinel drivers but we will expand support to other
    components in the near future
  • Functions that were previously accessed from the pkg_config module
    have been added to a msticpy.common.settings module. All settings-
    related functionality will be added to this module.
  • Added msticpy global settings page to MpConfigEdit
  • Refactored the QueryProvider class in msticpy.data.core.data_providers
    to separate related functionality into mixin classes. Added ability
    for a driver to supply a filtering function to show/hide query
    functions based on criteria set by the driver.
  • Included azure-kusto-data and azure-monitor-query to the Azure
    setup extra.

Unrelated fixes in this Pull request

  • Fixed method used to determine whether Keyring is supported on
    the current platform
  • Fixed errors in ProcessTree handline processIDs of different formats
  • Fixed errors try to read file with no permissions in Settings file
    browser component.
  • Added Pulsedive configuration to the settings editor.

Documentation changes

Added:

Updated:

@ianhelle
Initial code for az_monitor_driver.py
b3ddae9 
Added provider_settings.py for global proxy config
Add settings.py as a facade module for common setttings functions
Added lazy_import function and generic functions for implementing dynamic __getattr__ and __dir__ functions for __init__ modules
Added ability to create (partial) WorkspaceConfig from connection string.
Moved MpConfigEdit and MpConfigFile to dynamic imports in config/__init__.py
Changed MSSentinel import in ce_azure_sentinel.py to be imported on demand
Extended _execute_query to extract and supply timespan parameters to driver query functions
Removed deprecated imports or nbtools and sectools from nbinit.py
Importing get_config to msticpy/__init__.py
@ianhelle
Removing use of multiple workspace IDs from az_monitor_driver
9a19cd5
@ianhelle
Adding workaround protection for process tree NA values for Bokeh 3.0
14f1328 
Moving bokeh 3.0 requirements back to 2.4.3 for compat with panel.
@ianhelle
Initial code for azure_monitor_driver
bf99abc
@ianhelle
Refactoring methods from QueryProvider into mixin classes
98ed367
@ianhelle
Azure kusto driver based on azure-data-kusto
6fb01ba
@ianhelle
Azure Kusto driver and Azure monitor updates:
cc22c6c 
- Adding to DataEnvironments and drivers/__init__.py
Adding raw Kusto response test data
@ianhelle
Merge branch 'ianhelle/data_providers-refactor-2023-04-04' into ianhe…
8625c75 
…lle/azure-kusto-driver-2023-04-04
@ianhelle
Finished unit tests for test_azure_monitor_driver.py / azure_monitor_…
bd9fa80 
…driver.py

Implemented driver properties dictionary and use of DriverProps class to normalize naming in multiple drivers.
Added code to allow driver to override environment name for reading queries.
@ianhelle
Added documentation and fixed unit tests.
dc734a3 
Updated config documentation
Fixed keyring_client.py test for valid backend
Merged several global settings (proxies, http timeout) into "msticpy" section of config
Added settings editor support for msticpy global settings in ce_msticpy.py
Some Mypy fixes in file_browser.py
Removed some commented-out code from query_source.py
Added filtering for queries to query_source.py and data_providers.py - this allows individual kusto providers
to show only queries that are relevant for the connected cluster. Additional test files and unit tests to support this.
Added consistent support for proxy settings and timeout in azure_kusto_driver.py and azure_monitor_driver.py
Updated setup.py, requirements-all.txt and conda-reqs.txt to add azure-kusto-data and azure-monitor-query packages.
Added alternative to custom_mp_config test utility. This patches get_config calls in specified modules - more complex to set up but does not rely on a lock file, so allows more unit tests to run in parallel.
@ianhelle ianhelle added this to the Release 2.5.0 milestone Apr 17, 2023
@ianhelle ianhelle requested a review from petebryan April 17, 2023 22:38
@ianhelle ianhelle self-assigned this Apr 17, 2023
@ianhelle
Fix for sentinel and azurekusto test failures
2541b28
@ianhelle
Fixing test and lint failures
eefd09a
@ianhelle
Merge branch 'main' into ianhelle/azure-kusto-driver-2023-04-04
ed2de79
@ianhelle
Fixing settings source for test_sentinel_core.py
7a80fe0 
Change folder name for test_azure_monitor_driver.py (not sure why but seems to have an effect on linux)
@ianhelle
Merge branch 'ianhelle/azure-kusto-driver-2023-04-04' of https://gith…
c54e3e0 
…ub.com/microsoft/msticpy into ianhelle/azure-kusto-driver-2023-04-04
@ianhelle
isort fixes in polling_detection.py
bd54f7b 
pylint check name change in azure_auth_core.py
adding validate_config to settings.py
added ability to get workspace using case-insensitive ws name, ID or key name in wsconfig.py
removed some debugging lines from sentinel_utils.py
suppressed pylint error in data_providers.py
fixed field initializer in query_template.py
adding more logging to azure_monitor_driver.py
adding more logging to azure_kusto_driver.py
removing matplotlib import in nbinit, simplifying config loading code if config is already loaded.
Moving matplotlib to extra in requirements.txt, setup.py
Black re-format of test_periodogram_polling_detector.py
Added tests for WorkspaceConfig to test_wsconfig.py
Fixed test_azure_monitor_driver.py to use consistent settings.
Fixed test_nbinit.py to avoid using default loaded settings.
@ianhelle ianhelle mentioned this pull request Apr 21, 2023
Merged
@petebryan
Copy link
Contributor

Love it - great user experience here as its a near seamless update.

@ianhelle ianhelle merged commit b248797 into main May 10, 2023
@ianhelle ianhelle linked an issue May 16, 2023 that may be closed by this pull request
Replace Kqlmagic with azure.data.kusto/azure.monitor SDK #510
Closed
@ianhelle ianhelle mentioned this pull request May 16, 2023
Closed
@ianhelle ianhelle linked an issue May 16, 2023 that may be closed by this pull request
Better Error Reporting and Timeout for KQL Query Provider #451
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@petebryan petebryan petebryan approved these changes

Assignees

@ianhelle ianhelle

Labels
None yet
Projects
MSTICPy post-2.0 planning
Status: ✅ Done
Milestone
Release 2.5.0
2 participants
@ianhelle @petebryan