From c0fad011264a7ec31632eaf43c75add008549041 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Fri, 21 Apr 2023 09:47:18 -0700 Subject: [PATCH 01/26] Bump httpx from 0.23.3 to 0.24.0 (#655) Bumps [httpx](https://github.com/encode/httpx) from 0.23.3 to 0.24.0. - [Release notes](https://github.com/encode/httpx/releases) - [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md) - [Commits](https://github.com/encode/httpx/compare/0.23.3...0.24.0) --- updated-dependencies: - dependency-name: httpx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen --- conda/conda-reqs.txt | 2 +- docs/requirements.txt | 2 +- requirements-all.txt | 2 +- requirements.txt | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/conda/conda-reqs.txt b/conda/conda-reqs.txt index b633e0ff3..31f70d2d3 100644 --- a/conda/conda-reqs.txt +++ b/conda/conda-reqs.txt @@ -17,7 +17,7 @@ dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 html5lib -httpx==0.23.3 +httpx==0.24.0 ipython>=7.23.1 ipywidgets>=7.4.2, <8.0.0 keyring>=13.2.1 diff --git a/docs/requirements.txt b/docs/requirements.txt index f800b5103..32c3c5033 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -2,7 +2,7 @@ attrs>=18.2.0 cryptography deprecated>=1.2.4 docutils<0.20.0 -httpx==0.23.3 +httpx==0.24.0 ipython >= 7.1.1 jinja2<3.2.0 numpy>=1.15.4 diff --git a/requirements-all.txt b/requirements-all.txt index d1bcd09fb..a82655fd5 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -19,7 +19,7 @@ deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 -httpx==0.23.3 +httpx==0.24.0 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" diff --git a/requirements.txt b/requirements.txt index 7e899fe0f..d11fd1d98 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,7 +10,7 @@ deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 -httpx==0.23.3 +httpx==0.24.0 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" From b24879708c3fb71ef9b3b94ba0b8820ec423ce54 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Wed, 10 May 2023 15:10:42 -0700 Subject: [PATCH 02/26] Sentinel and Kusto new providers (#656) * Initial code for az_monitor_driver.py Added provider_settings.py for global proxy config Add settings.py as a facade module for common setttings functions Added lazy_import function and generic functions for implementing dynamic __getattr__ and __dir__ functions for __init__ modules Added ability to create (partial) WorkspaceConfig from connection string. Moved MpConfigEdit and MpConfigFile to dynamic imports in config/__init__.py Changed MSSentinel import in ce_azure_sentinel.py to be imported on demand Extended _execute_query to extract and supply timespan parameters to driver query functions Removed deprecated imports or nbtools and sectools from nbinit.py Importing get_config to msticpy/__init__.py * Removing use of multiple workspace IDs from az_monitor_driver * Adding workaround protection for process tree NA values for Bokeh 3.0 Moving bokeh 3.0 requirements back to 2.4.3 for compat with panel. * Initial code for azure_monitor_driver * Refactoring methods from QueryProvider into mixin classes * Azure kusto driver based on azure-data-kusto * Azure Kusto driver and Azure monitor updates: - Adding to DataEnvironments and drivers/__init__.py Adding raw Kusto response test data * Finished unit tests for test_azure_monitor_driver.py / azure_monitor_driver.py Implemented driver properties dictionary and use of DriverProps class to normalize naming in multiple drivers. Added code to allow driver to override environment name for reading queries. * Added documentation and fixed unit tests. Updated config documentation Fixed keyring_client.py test for valid backend Merged several global settings (proxies, http timeout) into "msticpy" section of config Added settings editor support for msticpy global settings in ce_msticpy.py Some Mypy fixes in file_browser.py Removed some commented-out code from query_source.py Added filtering for queries to query_source.py and data_providers.py - this allows individual kusto providers to show only queries that are relevant for the connected cluster. Additional test files and unit tests to support this. Added consistent support for proxy settings and timeout in azure_kusto_driver.py and azure_monitor_driver.py Updated setup.py, requirements-all.txt and conda-reqs.txt to add azure-kusto-data and azure-monitor-query packages. Added alternative to custom_mp_config test utility. This patches get_config calls in specified modules - more complex to set up but does not rely on a lock file, so allows more unit tests to run in parallel. * Fix for sentinel and azurekusto test failures * Fixing test and lint failures * Fixing settings source for test_sentinel_core.py Change folder name for test_azure_monitor_driver.py (not sure why but seems to have an effect on linux) * isort fixes in polling_detection.py pylint check name change in azure_auth_core.py adding validate_config to settings.py added ability to get workspace using case-insensitive ws name, ID or key name in wsconfig.py removed some debugging lines from sentinel_utils.py suppressed pylint error in data_providers.py fixed field initializer in query_template.py adding more logging to azure_monitor_driver.py adding more logging to azure_kusto_driver.py removing matplotlib import in nbinit, simplifying config loading code if config is already loaded. Moving matplotlib to extra in requirements.txt, setup.py Black re-format of test_periodogram_polling_detector.py Added tests for WorkspaceConfig to test_wsconfig.py Fixed test_azure_monitor_driver.py to use consistent settings. Fixed test_nbinit.py to avoid using default loaded settings. * Fix for 0.14.0 of statsmodels --- conda/conda-reqs.txt | 2 + docs/source/DataAcquisition.rst | 2 + .../api/msticpy.common.proxy_settings.rst | 7 + docs/source/api/msticpy.common.rst | 2 + docs/source/api/msticpy.common.settings.rst | 7 + docs/source/api/msticpy.common.utility.rst | 1 + .../msticpy.common.utility.yaml_loader.rst | 7 + docs/source/api/msticpy.config.ce_msticpy.rst | 7 + docs/source/api/msticpy.config.rst | 1 + ....core.query_provider_connections_mixin.rst | 7 + ...y.data.core.query_provider_utils_mixin.rst | 7 + .../api/msticpy.data.core.query_template.rst | 7 + docs/source/api/msticpy.data.core.rst | 3 + ...sticpy.data.drivers.azure_kusto_driver.rst | 7 + ...icpy.data.drivers.azure_monitor_driver.rst | 7 + docs/source/api/msticpy.data.drivers.rst | 2 + docs/source/conf.py | 3 + .../data_acquisition/DataProv-Kusto-New.rst | 488 ++++ .../DataProv-MSSentinel-New.rst | 292 ++ .../_static/kusto_query_display.png | Bin 0 -> 48821 bytes .../source/getting_started/SettingsEditor.rst | 10 + docs/source/getting_started/msticpyconfig.rst | 216 +- docs/source/visualization/NetworkGraph.rst | 2 +- msticpy/__init__.py | 1 + msticpy/analysis/polling_detection.py | 3 +- msticpy/analysis/timeseries.py | 6 +- msticpy/auth/azure_auth.py | 6 + msticpy/auth/azure_auth_core.py | 2 +- msticpy/auth/keyring_client.py | 17 +- msticpy/common/pkg_config.py | 22 +- msticpy/common/provider_settings.py | 120 +- msticpy/common/proxy_settings.py | 64 + msticpy/common/settings.py | 30 + msticpy/common/utility/package.py | 31 + msticpy/common/wsconfig.py | 149 +- msticpy/config/__init__.py | 20 +- msticpy/config/ce_azure_sentinel.py | 12 +- msticpy/config/ce_msticpy.py | 38 + msticpy/config/file_browser.py | 33 +- msticpy/config/mp_config_edit.py | 9 +- msticpy/context/azure/sentinel_utils.py | 1 + msticpy/context/domain_utils.py | 2 +- msticpy/data/core/data_providers.py | 477 +--- msticpy/data/core/query_defns.py | 2 + .../core/query_provider_connections_mixin.py | 98 + .../data/core/query_provider_utils_mixin.py | 375 +++ msticpy/data/core/query_source.py | 14 +- msticpy/data/core/query_store.py | 21 +- msticpy/data/core/query_template.py | 73 + msticpy/data/drivers/__init__.py | 2 + msticpy/data/drivers/azure_kusto_driver.py | 880 ++++++ msticpy/data/drivers/azure_monitor_driver.py | 608 ++++ msticpy/data/drivers/cybereason_driver.py | 15 +- msticpy/data/drivers/driver_base.py | 103 +- msticpy/data/drivers/elastic_driver.py | 15 +- msticpy/data/drivers/kql_driver.py | 7 +- msticpy/data/drivers/kusto_driver.py | 6 +- msticpy/data/drivers/mdatp_driver.py | 4 +- msticpy/data/drivers/mordor_driver.py | 45 +- msticpy/data/drivers/splunk_driver.py | 32 +- msticpy/data/drivers/sumologic_driver.py | 10 +- msticpy/init/azure_ml_tools.py | 2 +- msticpy/init/nbinit.py | 19 +- msticpy/resources/mpconfig_defaults.yaml | 10 + requirements-all.txt | 2 + requirements.txt | 1 - setup.py | 13 +- .../test_periodogram_polling_detector.py | 4 +- tests/common/test_wsconfig.py | 253 +- tests/context/azure/sentinel_test_fixtures.py | 43 +- tests/context/azure/test_sentinel_core.py | 85 +- .../azure/test_sentinel_dynamic_summary.py | 32 +- tests/data/drivers/test_azure_kusto_driver.py | 1165 ++++++++ .../data/drivers/test_azure_monitor_driver.py | 433 +++ tests/data/test_dataqueries.py | 11 +- tests/init/test_nbinit.py | 1 + tests/msticpyconfig-test.yaml | 6 + .../testdata/azmondata/az_mon_raw_query.json | 255 ++ tests/testdata/azmondata/az_mon_schema.json | 2464 +++++++++++++++++ tests/testdata/azmondata/query_response.pkl | Bin 0 -> 15812 bytes tests/testdata/kusto/T1_MDE_Cluster.yaml | 132 + tests/testdata/kusto/T2_MDE_Clusters.yaml | 134 + .../testdata/kusto/T3_MDE_Cluster_Group1.yaml | 133 + tests/testdata/kusto/T4_NoCluster_Info.yaml | 131 + .../kusto/T5_Help_Cluster_Group1.yaml | 134 + tests/testdata/kusto/T6_Group2_Cluster.yaml | 132 + tests/testdata/kusto/kusto_raw_resp.json | 2003 ++++++++++++++ tests/unit_test_lib.py | 89 +- 88 files changed, 11227 insertions(+), 900 deletions(-) create mode 100644 docs/source/api/msticpy.common.proxy_settings.rst create mode 100644 docs/source/api/msticpy.common.settings.rst create mode 100644 docs/source/api/msticpy.common.utility.yaml_loader.rst create mode 100644 docs/source/api/msticpy.config.ce_msticpy.rst create mode 100644 docs/source/api/msticpy.data.core.query_provider_connections_mixin.rst create mode 100644 docs/source/api/msticpy.data.core.query_provider_utils_mixin.rst create mode 100644 docs/source/api/msticpy.data.core.query_template.rst create mode 100644 docs/source/api/msticpy.data.drivers.azure_kusto_driver.rst create mode 100644 docs/source/api/msticpy.data.drivers.azure_monitor_driver.rst create mode 100644 docs/source/data_acquisition/DataProv-Kusto-New.rst create mode 100644 docs/source/data_acquisition/DataProv-MSSentinel-New.rst create mode 100644 docs/source/data_acquisition/_static/kusto_query_display.png create mode 100644 msticpy/common/proxy_settings.py create mode 100644 msticpy/common/settings.py create mode 100644 msticpy/config/ce_msticpy.py create mode 100644 msticpy/data/core/query_provider_connections_mixin.py create mode 100644 msticpy/data/core/query_provider_utils_mixin.py create mode 100644 msticpy/data/core/query_template.py create mode 100644 msticpy/data/drivers/azure_kusto_driver.py create mode 100644 msticpy/data/drivers/azure_monitor_driver.py create mode 100644 tests/data/drivers/test_azure_kusto_driver.py create mode 100644 tests/data/drivers/test_azure_monitor_driver.py create mode 100644 tests/testdata/azmondata/az_mon_raw_query.json create mode 100644 tests/testdata/azmondata/az_mon_schema.json create mode 100644 tests/testdata/azmondata/query_response.pkl create mode 100644 tests/testdata/kusto/T1_MDE_Cluster.yaml create mode 100644 tests/testdata/kusto/T2_MDE_Clusters.yaml create mode 100644 tests/testdata/kusto/T3_MDE_Cluster_Group1.yaml create mode 100644 tests/testdata/kusto/T4_NoCluster_Info.yaml create mode 100644 tests/testdata/kusto/T5_Help_Cluster_Group1.yaml create mode 100644 tests/testdata/kusto/T6_Group2_Cluster.yaml create mode 100644 tests/testdata/kusto/kusto_raw_resp.json diff --git a/conda/conda-reqs.txt b/conda/conda-reqs.txt index 31f70d2d3..14f8aa860 100644 --- a/conda/conda-reqs.txt +++ b/conda/conda-reqs.txt @@ -4,10 +4,12 @@ azure-core>=1.24.0 azure-mgmt-core>=1.2.1 azure-identity>=1.10.0 azure-keyvault-secrets>=4.0.0 +azure-kusto-data>=4.0.0 azure-mgmt-compute>=4.6.2 azure-mgmt-keyvault>=2.0.0 azure-mgmt-network>=2.7.0 azure-mgmt-resource>=16.1.0 +azure-monitor-query>=1.0.0 azure-storage-blob>=12.5.0 beautifulsoup4>=4.0.0 bokeh>=1.4.0, <=2.4.3 diff --git a/docs/source/DataAcquisition.rst b/docs/source/DataAcquisition.rst index fc7c655c8..4013551e3 100644 --- a/docs/source/DataAcquisition.rst +++ b/docs/source/DataAcquisition.rst @@ -17,6 +17,7 @@ Individual Data Environments :maxdepth: 2 data_acquisition/DataProv-MSSentinel + data_acquisition/DataProv-MSSentinel-New data_acquisition/DataProv-MSDefender data_acquisition/DataProv-MSGraph data_acquisition/DataProv-LocalData @@ -25,6 +26,7 @@ Individual Data Environments data_acquisition/MordorData data_acquisition/DataProv-Sumologic data_acquisition/DataProv-Kusto + data_acquisition/DataProv-Kusto-New data_acquisition/DataProv-Cybereason data_acquisition/DataProv-OSQuery diff --git a/docs/source/api/msticpy.common.proxy_settings.rst b/docs/source/api/msticpy.common.proxy_settings.rst new file mode 100644 index 000000000..eb0736f64 --- /dev/null +++ b/docs/source/api/msticpy.common.proxy_settings.rst @@ -0,0 +1,7 @@ +msticpy.common.proxy\_settings module +===================================== + +.. automodule:: msticpy.common.proxy_settings + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.common.rst b/docs/source/api/msticpy.common.rst index 5cda2c32b..2146fa1dd 100644 --- a/docs/source/api/msticpy.common.rst +++ b/docs/source/api/msticpy.common.rst @@ -27,5 +27,7 @@ Submodules msticpy.common.exceptions msticpy.common.pkg_config msticpy.common.provider_settings + msticpy.common.proxy_settings + msticpy.common.settings msticpy.common.timespan msticpy.common.wsconfig diff --git a/docs/source/api/msticpy.common.settings.rst b/docs/source/api/msticpy.common.settings.rst new file mode 100644 index 000000000..74dab0dbe --- /dev/null +++ b/docs/source/api/msticpy.common.settings.rst @@ -0,0 +1,7 @@ +msticpy.common.settings module +============================== + +.. automodule:: msticpy.common.settings + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.common.utility.rst b/docs/source/api/msticpy.common.utility.rst index cd235f316..6f622c02d 100644 --- a/docs/source/api/msticpy.common.utility.rst +++ b/docs/source/api/msticpy.common.utility.rst @@ -16,3 +16,4 @@ Submodules msticpy.common.utility.ipython msticpy.common.utility.package msticpy.common.utility.types + msticpy.common.utility.yaml_loader diff --git a/docs/source/api/msticpy.common.utility.yaml_loader.rst b/docs/source/api/msticpy.common.utility.yaml_loader.rst new file mode 100644 index 000000000..21b164f63 --- /dev/null +++ b/docs/source/api/msticpy.common.utility.yaml_loader.rst @@ -0,0 +1,7 @@ +msticpy.common.utility.yaml\_loader module +========================================== + +.. automodule:: msticpy.common.utility.yaml_loader + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.config.ce_msticpy.rst b/docs/source/api/msticpy.config.ce_msticpy.rst new file mode 100644 index 000000000..433815734 --- /dev/null +++ b/docs/source/api/msticpy.config.ce_msticpy.rst @@ -0,0 +1,7 @@ +msticpy.config.ce\_msticpy module +================================= + +.. automodule:: msticpy.config.ce_msticpy + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.config.rst b/docs/source/api/msticpy.config.rst index 8fbd79930..d5e267190 100644 --- a/docs/source/api/msticpy.config.rst +++ b/docs/source/api/msticpy.config.rst @@ -17,6 +17,7 @@ Submodules msticpy.config.ce_common msticpy.config.ce_data_providers msticpy.config.ce_keyvault + msticpy.config.ce_msticpy msticpy.config.ce_other_providers msticpy.config.ce_provider_base msticpy.config.ce_simple_settings diff --git a/docs/source/api/msticpy.data.core.query_provider_connections_mixin.rst b/docs/source/api/msticpy.data.core.query_provider_connections_mixin.rst new file mode 100644 index 000000000..e2c876e28 --- /dev/null +++ b/docs/source/api/msticpy.data.core.query_provider_connections_mixin.rst @@ -0,0 +1,7 @@ +msticpy.data.core.query\_provider\_connections\_mixin module +============================================================ + +.. automodule:: msticpy.data.core.query_provider_connections_mixin + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.core.query_provider_utils_mixin.rst b/docs/source/api/msticpy.data.core.query_provider_utils_mixin.rst new file mode 100644 index 000000000..d778e6f80 --- /dev/null +++ b/docs/source/api/msticpy.data.core.query_provider_utils_mixin.rst @@ -0,0 +1,7 @@ +msticpy.data.core.query\_provider\_utils\_mixin module +====================================================== + +.. automodule:: msticpy.data.core.query_provider_utils_mixin + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.core.query_template.rst b/docs/source/api/msticpy.data.core.query_template.rst new file mode 100644 index 000000000..a5982ecb0 --- /dev/null +++ b/docs/source/api/msticpy.data.core.query_template.rst @@ -0,0 +1,7 @@ +msticpy.data.core.query\_template module +======================================== + +.. automodule:: msticpy.data.core.query_template + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.core.rst b/docs/source/api/msticpy.data.core.rst index 19413a893..d7497b48b 100644 --- a/docs/source/api/msticpy.data.core.rst +++ b/docs/source/api/msticpy.data.core.rst @@ -17,5 +17,8 @@ Submodules msticpy.data.core.param_extractor msticpy.data.core.query_container msticpy.data.core.query_defns + msticpy.data.core.query_provider_connections_mixin + msticpy.data.core.query_provider_utils_mixin msticpy.data.core.query_source msticpy.data.core.query_store + msticpy.data.core.query_template diff --git a/docs/source/api/msticpy.data.drivers.azure_kusto_driver.rst b/docs/source/api/msticpy.data.drivers.azure_kusto_driver.rst new file mode 100644 index 000000000..3c44ffc6a --- /dev/null +++ b/docs/source/api/msticpy.data.drivers.azure_kusto_driver.rst @@ -0,0 +1,7 @@ +msticpy.data.drivers.azure\_kusto\_driver module +================================================ + +.. automodule:: msticpy.data.drivers.azure_kusto_driver + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.drivers.azure_monitor_driver.rst b/docs/source/api/msticpy.data.drivers.azure_monitor_driver.rst new file mode 100644 index 000000000..1ff8db125 --- /dev/null +++ b/docs/source/api/msticpy.data.drivers.azure_monitor_driver.rst @@ -0,0 +1,7 @@ +msticpy.data.drivers.azure\_monitor\_driver module +================================================== + +.. automodule:: msticpy.data.drivers.azure_monitor_driver + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.drivers.rst b/docs/source/api/msticpy.data.drivers.rst index bb2e3fcf5..4bf42fe41 100644 --- a/docs/source/api/msticpy.data.drivers.rst +++ b/docs/source/api/msticpy.data.drivers.rst @@ -12,6 +12,8 @@ Submodules .. toctree:: :maxdepth: 4 + msticpy.data.drivers.azure_kusto_driver + msticpy.data.drivers.azure_monitor_driver msticpy.data.drivers.cybereason_driver msticpy.data.drivers.driver_base msticpy.data.drivers.elastic_driver diff --git a/docs/source/conf.py b/docs/source/conf.py index 8b69b6c56..34f5e924e 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -226,6 +226,7 @@ "azure.identity", "azure.keyvault.secrets", "azure.keyvault", + "azure.kusto.data", "azure.mgmt.compute.models", "azure.mgmt.compute", "azure.mgmt.keyvault.models", @@ -235,6 +236,7 @@ "azure.mgmt.resource", "azure.mgmt.resourcegraph", "azure.mgmt.subscription", + "azure.monitor.query", "azure.storage.blob", "azure.storage", "bokeh", @@ -259,6 +261,7 @@ "nest_asyncio", "networkx", "openpyxl", + "panel", "passivetotal", "pygeohash", "pygments", diff --git a/docs/source/data_acquisition/DataProv-Kusto-New.rst b/docs/source/data_acquisition/DataProv-Kusto-New.rst new file mode 100644 index 000000000..7712368e2 --- /dev/null +++ b/docs/source/data_acquisition/DataProv-Kusto-New.rst @@ -0,0 +1,488 @@ +Azure Data Explorer/Kusto Provider - New Implementation +======================================================= + +This is a new implementation of the Azure Data Explorer/Kusto +QueryProvider using the +`azure-data-kusto SDK `__ +(the earlier implementation used +`Kqlmagic `__). + + +.. warning:: This provider currently in beta and is available for testing. + It is available alongside the existing Kusto provider for you + to compare old and new. + If you are using the existing implementation, see :doc:`./DataProv-Kusto` + +Changes from the previous implementation +---------------------------------------- + +* Use the provider name ``Kusto_New`` when creating a QueryProvider + instance. This will be changed to ``Kusto`` in a future release. +* The settings format has changed (although the existing format + is still supported albeit with some limited functionality). +* You could previously specify a new cluster to connect to in + when executing a query. This is no longer supported. Once the + provider is connected to a cluster it will only execute queries against + that cluster. (You can however, call the connect() function to connect + the provider to a new cluster before running the query.) +* Some of the previous parameters have been deprecated: + + * ``mp_az_auth`` is replaced by ``auth_types`` (the former still works + but will be removed in a future release). + * ``mp_az_auth_tenant_id`` is replaced by ``tenant_id`` (the former + is no longer supported + +Kusto Configuration +------------------- + +Kusto Configuration in MSTICPy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can store your connection details in *msticpyconfig.yaml*. + +For more information on using and configuring *msticpyconfig.yaml* see +:doc:`msticpy Package Configuration <../getting_started/msticpyconfig>` +and :doc:`MSTICPy Settings Editor<../getting_started/SettingsEditor>` + +.. note:: The settings for the new Kusto provider are stored in the + ``KustoClusters`` section of the configuration file. This cannot + currently be edited from the MSTICPy Settings Editor - please + edit the *msticpyconfig.yaml* directly to edit these. + +To accommodate the use of multiple clusters the new provider supports +a different configuration format. + +The basic settings in the file should look like the following: + +.. code:: yaml + + KustoClusters: + ... + Cluster1: + Args: + Cluster: https://uscluster.kusto.windows.net + Cluster2: + Args: + Cluster: https://eucluster.kusto.windows.net + IntegratedAuth: True # This is default and is optional + +You can have any number of cluster entries in this section. + +Specifying additional parameters for a cluster +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can add authentication and other parameters to the ``Args`` +sub-key of a cluster definition. In the following example, +the TenantId is specified along with Client app ID and client secret +for *clientsecret* authentication. + +.. code:: yaml + + KustoClusters: + DataClusterX: + Args: + Cluster: https://xxx.kusto.windows.net + ClientId: 69d28fd7-42a5-48bc-a619-af56397b1111 + TenantId: 69d28fd7-42a5-48bc-a619-af56397b9f28 + ClientSecret: + KeyVault: + +The ClusterDefaults section +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you have parameters that you want to apply to all clusters +you can add these to a ``ClusterDefaults`` section. + +.. code:: yaml + + KustoClusters: + ClusterDefaults: + Args: + TenantId: 69d28fd7-42a5-48bc-a619-af56397b9f28 + Cluster1: + Args: + Cluster: https://uscluster.kusto.windows.net + Cluster2: + Args: + Cluster: https://eucluster.kusto.windows.net + + +Creating ClusterGroups +~~~~~~~~~~~~~~~~~~~~~~ + +You can create a group of clusters that you can reference by +cluster group name. This is useful if you have clusters in different regions +that share the same schema and you want to run the same queries +against all of them. + +This is used primarily to support query templates, to match +queries to the correct cluster. See `Writing query templates for Kusto clusters`_ +later in this document. + +Loading a QueryProvider for Kusto +--------------------------------- + +.. code:: ipython3 + + import msticpy as mp + kql_prov = mp.QueryProvider("Kusto_New") + + + +Connecting to a Kusto cluster +----------------------------- + +Before running queries you need to connect to a cluster using +the ``connect()`` method. + +See +:py:meth:`connect() ` + +The parameters required for connection to a Kusto cluster can be passed +to ``connect()`` in +several of ways. You can provide a full connection string or parameters +for ``cluster`` (and optionally, ``database``). +In the latter case, you must have configured +settings for the cluster defined in your msticpyconfig.yaml. + +If you have the cluster details configured in msticpy, the ``cluster`` +parameter can be one of the following: + +* The section name ("Cluster1" or "Cluster2" in the configuration example above) +* The full URL of the cluster either the actual cluster name +* The host name of the cluster (e.g. "uscluster", "eucluster" in the example) + +In all cases these are case-insensitive. + +These are all equivalent: + +.. code:: ipython3 + + kql_prov.connect(cluster="Cluster2") + kql_prov.connect(cluster="eucluster") + kql_prov.connect(cluster="https://eucluster.kusto.windows.net") + + +If the cluster is not in your configuration you must use the full +URL of the cluster. + +You can optionally specify a default database to connect to. The database +can be changed with each query (either by specifying a ``database`` parameter +or by using the ``database`` metadata property in a query definition file +(see `Writing query templates for Kusto clusters`_) below) + +You can also pass authentication parameters in the ``connect`` call: + +* auth_types - to override the configured Azure credential types +* tenant_id - to override your default tenant_id + +.. code:: python3 + + kql_prov.connect( + cluster="Cluster2", + auth_types=["device_code"], + tenant_id="69d28fd7-42a5-48bc-a619-af56397b9f28" + ) + +For more details on Azure Authentication in *MSTICPy* see +:doc:`Azure Authentication <../getting_started/AzureAuthentication>` + +Kusto QueryProvider methods and properties +------------------------------------------ + +The Kusto QueryProvider has the following methods and properties +in addition to those inherited from the base QueryProvider class. + +* :py:meth:`get_database_names() ` + Returns the names of the databases for a connected cluster. +* :py:meth:`get_database_schema([database]) ` + Returns a schema dictionary for the tables in a database a connected cluster. +* :py:meth:`configured_clusters (property) ` +* Returns a list of the configured cluster read from msticpyconfig.yaml. +* :py:meth:`cluster_uri (property) ` + The URI of the connected cluster. +* :py:meth:`cluster_name (property) ` + The host name of the connected cluster. +* :py:meth:`cluster_config_name (property) ` + The configuration entry name for the connected cluster. +* :py:meth:`set_cluster(cluster) ` + Switch the provider to a different cluster - this is a more restricted version of the ``connect()`` method. +* :py:meth:`set_database(database) ` + Switches the default database for the provider. + +Running Ad Hoc queries +---------------------- + +You can run ad hoc queries using the ``exec_query()`` method of the QueryProvider. + +.. note:: You usually need to specify a ``database`` parameter when running + ad hoc queries. + +Writing query templates for Kusto clusters +------------------------------------------ + +The details for configuring and connecting to Kusto clusters +are enough to allow you to run ad hoc queries. However, if you want to +create and use parameterized queries there are some additional steps +that you need to take. + +Please read the general section on +:ref:`Creating new queries ` +if you are not familiar with the general process of creating query +templates for *MSTICPy*. + +The queries for Kusto work in the same way as for many other data providers +except that you can (and should) specify the cluster(s) and database for +the query to use. + +Controlling which queries are displayed and runnable for a provider +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Since Kusto clusters have widely varying schemas, it only makes sense +to run a query on a cluster for which it was designed. +MSTICPy enforces this by allowing you to specify parameters in +both the query template definitions and the cluster configuration +in ``msticpyconfig.yaml`` that correctly match queries to +providers connected to appropriate clusters. + +When you first instantiate a Kusto QueryProvider, it will read +all queries files available for the Kusto DataEnvironment. However, +when you connect to a cluster, these queries and filtered so that +only ones compatible with this cluster are available. + +If you have query definition files (query templates) you can +try this by creating a Kusto QueryProvider and running the +``list_queries()`` method. Then connect to a cluster and run +``list_queries()`` again. In the first case, you should see all +queries that you have defined but in the second case, you +should only see queries that have been built to run on that +cluster. + +.. code:: python3 + + from msticpy.data import QueryProvider + kql_prov = QueryProvider("Kusto") + kql_prov.list_queries() + +.. code:: python3 + + # new cell + kql_prov.connect(cluster="Cluster2") + kql_prov.list_queries() + +This is explained more in the later sections on `Kusto cluster specifier`_ +and + +Basic Kusto query format +~~~~~~~~~~~~~~~~~~~~~~~~ + +The query template format for Kusto queries should look like +the following. The ``data_environments`` item must include +"Kusto" in the list of applicable environments. + +This example show the metadata section for a query file, highlighting +the items that are specific Kusto queries. (``data_families`` is common +to other query types but has some Kusto-specific usage that is different +as explained later.) + +.. code-block:: + :emphasize-lines: 4-12 + + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [DeviceEvents.hostdata] + cluster: https://uscluster.kusto.windows.net + clusters: + - https://uscluster.kusto.windows.net + - https://eucluster.kusto.windows.net + cluster_groups: + - Group1 + database: hostdata + tags: ["user"] + defaults: + parameters: + table: + # .... + sources: + list_host_processes: + description: Lists all process creations for a host + # .... + + +Most of the query file is identical to queries for other drivers. +However, the metadata section has some additional items. These +are explained in the following sections. + +Kusto database specifier +~~~~~~~~~~~~~~~~~~~~~~~~ + +You can use the ``database`` item to specify the cluster database to +use. For backward compatibility you can also specify this in the +``data_families`` entry using a dotted notation. ``data_families`` +is also used to group queries in the query provider, so using this +to specify the database name is not recommended. + +The following examples show the different ways of configuring +this. + +For the following two configurations, the database used is ``DeviceEvents`` +and the queries are grouped under the ``hostdata`` family (the +queries are attached as methods to the QueryProvider). + +.. code-block:: yaml + :emphasize-lines: 5,6 + + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [hostdata] + database: DeviceEvents + cluster: https://uscluster.kusto.windows.net + +.. code-block:: yaml + :emphasize-lines: 5,6 + + # Deprecated format + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [hostdata.DeviceEvents] + cluster: https://uscluster.kusto.windows.net + +For this configuration the database used is ``DeviceEvents`` and the +queries will also be grouped under the DeviceEvents container. + +.. code-block:: yaml + :emphasize-lines: 5 + + # Deprecated format + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [DeviceEvents] + cluster: https://uscluster.kusto.windows.net + +.. note:: The when using the ``data_families`` entry to specify + the database name, only the first entry in the list is used + for this. Subsequent items still work for creating + data query groupings. + +Kusto cluster specifier +~~~~~~~~~~~~~~~~~~~~~~~ + +Adding a cluster specifier matches queries to the right cluster +and prevents a query from being used with +a cluster and database for which it was not intended. + +You can specify the cluster to use in three ways: + +* Including a ``cluster_groups`` item in the metadata section. + This is a list of cluster group names that are defined in the + ``msticpyconfig.yaml`` file. Queries with one or more ``cluster_groups`` + entries can be used against any of the cluster definitions in + ``msticpyconfig.yaml`` that have matching cluster group names. +* Including a ``clusters`` item in the metadata section. + This is a list of cluster identifiers (URIs, names or configuration section names + that are defined in the ``msticpyconfig.yaml`` file). These queries + can be used with any cluster configuration entry that matches one + of the IDs in the ``clusters`` item. +* Including a ``cluster`` item in the metadata section. + This is a single cluster identifier (URI, name or configuration section name + that is defined in the ``msticpyconfig.yaml`` file). These queries + can only be used with the cluster configuration entry that matches + the ID in the ``cluster`` item. + +The cluster specifiers are used in the order above until a match is found. +You can include more than one cluster specifier in a query definition file. +If no match is found, the query will not be added to the query provider. + +.. note:: For queries that have no cluster specifier, they will + be added to the query provider but but may not work. + +.. tip:: If you want to avoid these queries being added use + the parameter ``strict_query_match=True`` when + creating the Kusto QueryProvider as shown in the following + example + +.. code:: python3 + + import msticpy as mp + kql_prov = mp.QueryProvider("Kusto_New", strict_query_match=True) + + +The following examples show the different ways of configuring +clusters to match queries: + + +.. code-block:: yaml + :emphasize-lines: 6,7 + + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [hostdata] + cluster_groups: + - Group1 + database: DeviceEvents + +.. code-block:: yaml + :emphasize-lines: 6,7 + + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [hostdata] + clusters: + - https://uscluster.kusto.windows.net + - https://eucluster.kusto.windows.net + database: DeviceEvents + +.. code-block:: yaml + :emphasize-lines: 6 + + metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [hostdata] + cluster: https://uscluster.kusto.windows.net + database: DeviceEvents + +.. note:: you can also use cluster specifiers (using the same syntax + as show above) for individual query metadata. Each query has + it's own optional ``metadata`` sub-key. Setting cluster + specifiers at the query level, with different queries assigned + to different clusters in the same file may make organizing + your queries more difficult, so we recommend only using + cluster specifiers at the file level. However, it is possible + to do this if you need to. + + +Logical flow used to determine if a query is shown +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This flowchart shows the logic applied using the query definition +and configuration parameters to determine whether a query is +shown or not (i.e. whether it appears in ``list_queries()`` and +as attached to the QueryProvider as a query function.) + +.. figure:: _static/kusto_query_display.png + :alt: Flow chart showing how queries are filtered based on query metadata + and configuration settings. + :height: 5in + +Other Kusto Documentation +----------------------------------- + +For examples of using the Kusto provider, see the samples +`Kusto Analysis Notebook `__ +and `Kusto Ingest Notebook `__ + +:py:mod:`Kusto driver API documentation` \ No newline at end of file diff --git a/docs/source/data_acquisition/DataProv-MSSentinel-New.rst b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst new file mode 100644 index 000000000..c7db7a53e --- /dev/null +++ b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst @@ -0,0 +1,292 @@ +Microsoft Sentinel Provider - New Implementation +================================================ + +This is a new implementation of the MS Sentinel QueryProvider using +the +`azure-monitor-query SDK `__ +(the earlier implementation used +`Kqlmagic `__) + +.. note:: This provider currently in beta and is available for testing. + It is available alongside the existing Sentinel provider for you + to compare old and new. + If you are using the existing implementation, see :doc:`./DataProv-MSSentinel` + +Changes from the previous implementation +---------------------------------------- + +* Use the provider name ``MSSentinel_New`` when creating a QueryProvider + instance. +* By default, it uses the *MSTICPy* built-in Azure authentication by + default - you do not have to specify parameters to enable this. +* Supports simultaneous queries against multiple workspaces (see below). +* Supports user-specified timeout for queries. +* Supports proxies (via MSTICPy config or the ``proxies`` parameter to + the ``connect`` method) +* Some of the previous parameters have been deprecated: + + * ``mp_az_auth`` is replaced by ``auth_types`` (the former still works + but will be removed in a future release). + * ``mp_az_auth_tenant_id`` is replaced by ``tenant_id`` (the former + is no longer supported + + +Sentinel Configuration +---------------------- + +You store configuration for your workspace (or workspaces) in +your ``msticpyconfig.yaml``. + +For more information on using and configuring *msticpyconfig.yaml* see +:doc:`msticpy Package Configuration <../getting_started/msticpyconfig>` +and :doc:`MSTICPy Settings Editor<../getting_started/SettingsEditor>` + +The MS Sentinel connection settings are stored in the +``AzureSentinel\\Workspaces`` section of the file. +Here is an example. + +.. code:: yaml + + AzureSentinel: + Workspaces: + # Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig + # Specifying values here overrides config.json settings unless you explicitly load + # WorkspaceConfig with config_file parameter (WorkspaceConfig(config_file="../config.json") + Default: + WorkspaceId: 271f17d3-5457-4237-9131-ae98a6f55c37 + TenantId: 335b56ab-67a2-4118-ac14-6eb454f350af + ResourceGroup: soc + SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20 + WorkspaceName: Workspace1 + # To use these launch with an explicit name - WorkspaceConfig(workspace_name="Workspace2") + Workspace1: + WorkspaceId: "c88dd3c2-d657-4eb3-b913-58d58d811a41" + TenantId: "335b56ab-67a2-4118-ac14-6eb454f350af" + ResourceGroup: soc + SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20 + WorkspaceName: Workspace1 + TestWorkspace: + WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c" + TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a" + ResourceGroup: soc + SubscriptionId: a5b24e23-a96a-4472-b729-9e5310c83e20 + WorkspaceName: Workspace2 + +If you only use a single workspace, you only need to create a ``Default`` entry and +add the values for your *WorkspaceID* and *TenantID*. You can add other entries here, +for example, SubscriptionID, ResourceGroup. These are not required for the data +queries but are recommended since they are used by other *MSTICPy* components. + +If you use multiple workspaces, you can add further entries here. The key for +each entry (e.g. ``Workspace1`` or ``TestWorkspace`` in the example above) +is normally the name of the Azure Sentinel workspace but +you can use any name you prefer. You use this entry name when connecting +to a workspace. + + +Loading a QueryProvider for Microsoft Sentinel +---------------------------------------------- + +.. code:: ipython3 + + qry_prov = QueryProvider( + data_environment="MSSentinel_New", + ) + + # or just + qry_prov = QueryProvider("MSSentinel_New") + +Optional parameters for the Sentinel QueryProvider +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +``timeout`` : int (seconds) + +Specify a timeout for queries. Default is 300 seconds. +This parameter can be set here or in the ``connect`` method +and overridden for individual queries. + +``proxies`` : Dict[str, str] + +Proxy settings for log analytics queries. +If proxies are configured in *msticpyconfig.yaml* this is used by default. +If specified as a parameter, specify proxies as a dictionary of the form +``{protocol: proxy_url}`` + +The only protocol used by the driver is "https" (other protocols +can be set in *msticpyconfig.yaml* but only https is used here). +The proxy_url can contain +optional authentication information in the format +"https://username:password@proxy_host:port" + +If you have a proxy configuration set in *msticpyconfig.yaml* and +you do not want to use it, set ``proxies`` to None or an empty dictionary. +This parameter can be overridden in connect method. + +Connecting to a MS Sentinel Workspace +------------------------------------- + +Once you've created a QueryProvider you need to authenticate to Sentinel +Workspace. This is done by calling the connect() function of the Query +Provider. See :py:meth:`connect() ` + +This function takes an initial parameter (called ``connection_str`` for +historical reasons) that can be one of the following: + +* A WorkspaceConfig instance +* A connection string (this is option is being deprecated) +* None - in this case it will connect with the ``Default`` entry from + your *msticpyconfig.yaml* file. + +If you omit this parameter you use the ``workspace`` parameter +to specify the workspace entry from ``msticpyconfig.yaml`` to use. + + +Connecting to a Sentinel workspace +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When connecting you can just pass the name of your workspace or +an instance of WorkspaceConfig to the query provider's ``connect`` method. + +.. code:: IPython + + qry_prov.connect(workspace="Default") + qry_prov.connect(workspace="MyOtherWorkspace") + + # or, passing WorkspaceConfig + qry_prov.connect(WorkspaceConfig()) + # or + qry_prov.connect(WorkspaceConfig(workspace="MyOtherWorkspace")) + + + +MS Sentinel Authentication options +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +By default, the data provider will use Azure authentication +following the parameters defined in your ``msticpyconfig.yaml`` file +(or the default values if you have not configured them in this file). + +To read more about Azure authentication see +:doc:`Azure Authentication <../getting_started/AzureAuthentication>` + +You can override several authentication parameters including: + +* auth_types - a list of authentication types to try in order +* tenant_id - the Azure tenant ID to use for authentication + +If you are using a Sovereign cloud rather than the Azure global cloud, +you should follow the guidance in :doc:`Azure Authentication <../getting_started/AzureAuthentication>` +to configure the correct cloud. + + +Connecting to multiple Sentinel workspaces +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The Sentinel data provider supports connecting to multiple workspaces. +You can pass a list of workspace names or workspace IDs to the ``connect`` method. +using the ``workspaces`` or ``workspace_ids`` parameters respectively. + +``workspace_ids`` should be a list or tuple of workspace IDs. + +``workspaces`` should be a list or tuple of workspace names. In order +to use this parameter you must have these workspaces configured in +your *msticpyconfig.yaml*. + +These parameters override the ``workspace`` parameter. + +Connecting to multiple workspaces allows you to run queries across these +workspaces and return the combined results as a single Pandas DataFrame. +The workspaces must use common authentication credentials and are +expected to have the same data schema. + +.. code:: ipython3 + + qry_prov.connect(workspaces=["Default", "MyOtherWorkspace"]) + + qry_prov.SecurityAlert.list_alerts() + +This will return a DataFrame containing the results of the query, +the results from each workspace will be indicated by the +``TenantId`` column, which will contain the workspace ID of +each workspace. + +.. note:: This is a mechanism implemented by the underlying + **azure-monitor-query** + client library. It is independent of the MSTICPy capability to + add multiple connections to a query provider (and run parallel + queries against each workspace). You can use either of these + but we recommended using + one or the other and not both simultaneously. + +.. warning:: Connecting to multiple workspaces like this means + that the ``schema`` property will not return anything. This + only works if you connect to a single workspace. In this case, + it will return the schema of this workspace. + + +Other parameters for Sentinel ``connect()`` method +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +For ``timeout`` and ``proxies`` see the section above. + +After connecting to +The WorkspaceConfig class +------------------------- + +You do not need to know the details of this class but it is used +behind the scenes to provide workspace configuration information +to the Sentinel data provider. + +``WorkspaceConfig`` handles loading your workspace configuration +and generating a connection string from your configuration. +See :py:mod:`WorkspaceConfig API documentation` + +``WorkspaceConfig`` works with workspace configuration stored in *msticpyconfig.yaml*. + +To use ``WorkspaceConfig``, simple create an instance of it. It will automatically build +your connection string for use with the query provider library. + +.. code:: python3 + + ws_config = WorkspaceConfig() + +When called without parameters, *WorkspaceConfig* loads the "Default" +entry in your *msticpyconfig.yaml*. To specify a different workspace pass the ``workspace`` parameter +with the name of your workspace entry. This value is the name of +the section in the ``msticpyconfig.yaml`` ``Workspaces`` section. + +.. note:: the ``workspace`` parameter value is the entry heading in + your ``msticpyconfig.yaml``. As mentioned above, this may + not necessarily be the same as your workspace name. + +.. code:: python3 + + ws_config = WorkspaceConfig(workspace="TestWorkspace") + + +To see which workspaces are configured in your *msticpyconfig.yaml* use +the ``list_workspaces()`` function. + +.. tip:: ``list_workspaces`` is a class function, so you do not need to + instantiate a WorkspaceConfig to call this function. + +.. code:: python3 + + WorkspaceConfig.list_workspaces() + +.. parsed-literal:: + + {'Default': {'WorkspaceId': '271f17d3-5457-4237-9131-ae98a6f55c37', + 'TenantId': '335b56ab-67a2-4118-ac14-6eb454f350af'}, + 'Workspace1': {'WorkspaceId': 'c88dd3c2-d657-4eb3-b913-58d58d811a41', + 'TenantId': '335b56ab-67a2-4118-ac14-6eb454f350af'}, + 'TestWorkspace': {'WorkspaceId': '17e64332-19c9-472e-afd7-3629f299300c', + 'TenantId': '4ea41beb-4546-4fba-890b-55553ce6003a'}} + + +Other MS Sentinel Documentation +------------------------------- + +Built-in :ref:`data_acquisition/DataQueries:Queries for Microsoft Sentinel`. + +See also: :py:mod:`Sentinel KQL driver API documentation ` diff --git a/docs/source/data_acquisition/_static/kusto_query_display.png b/docs/source/data_acquisition/_static/kusto_query_display.png new file mode 100644 index 0000000000000000000000000000000000000000..2266aa5e0338ec2f2ff417ce2d9382c2998545c1 GIT binary patch literal 48821 zcmeEuXH=8R*DoFmHlztiR}fGTkS-;lNKxt1lUqU>E8iAXB?j^%90dxGA)9Kb7qfJ9+8kBBd#3%eI7htdh_I^BMAvD z2mbHO)91I>Nl0p+J(YW;U98z~3opgG5dUBUFV9R$aVJkY$M3Bo=P~e);#dkLY zHO^fpm22ow5IuA2ugjMeqLfBWC}_@~{S^4^QU8q#uW!&an!x-EH)Q+W;(ohJ?d5A* zIinHi5|7;sJ}d3IdxxHHsZ7kz#t*XF7bqREx^sXlHHeP*#oYv zjuiQHt#(-HYUr$ZL9^T(8Z>Vw-`Q|l5ZpuQdiwl%{`;QYfTwy}8%4_Rv$b4ro=>|{iJ5wM(>Hex+0VKtAk3g+mmL-HgbZ#0m}N zW%R#DAN{U&`_!me*CvTXa=Fljt2emuzTicpN>!OSIrep(Li?hy5l z0N3;V$ci!W?QH?52bh;TjyQ(9EXim0k8`;Mcia+f9W?gJp~LsP@SYjouj|oJcOZzs zbu@-p>R561;dxUT84Q`9kjc_R0CsBDTx5Aw1>Ta>JZVf0%Z&}0-1+GR-+t`|8^rNK z`>xszedWj3GS{bq_bXp$TPu&IZggAN>XOzjsuU<=0iU{aZsW2aL)xR4c}2?37+2xJ zOJu;h=2N`5{riDvOsPkdP1!U=*@6!b6`htf<^zVlPf9yedOz2!O!8cz@uJ6nVRTxi ze5V+iWi08fW%xej8Z(AVl$x3md?I1~h@4Ldp+gVO#<31LvU+}DczU2M3#7cmFV)v^BGn#K)KK`~H2)k_eUZP#MF zx5r*`hk~s|j+#-KF%Ukkml6B%9jb6@S}N!*!3s2@hJiZPUIlet-4n#+(Hk(mTK?D> zX03l~1hcKdELjtZm=J9Zgy;zAKSU~a{H?N#m#1MaScJr`&o@W5Hvaj%8y{sddXq+1n(+lt^FA#-JiKNl!@MYDmO&7N`bJ171Ys9k7;^5`w;>cB3wblfadJAM;fcq_=u2P( zvtxd#8T;&HV4kNnkb6)P=xWBWXKU_OdmThoUX;n1c4#*u1Mi*Wnpk$^_kqRC4q5M8 zk5PdNF?&tA|3|dNPfx@Jl8wBZ^091E+dv_KVo)XAZJ=($>kCaUQt?HF$-5iRw+!9w z@H&BfH*l42fWV+)t{_OjwPosb><-U`?bmxVKow#q>~k>>A0=wjg4*lPsNgqV94aU2 z2dU$yE~i^TkY9LaigKHN$F%}QL{Qz;T)`7)_@t5LQHP1w!VhULdKNd!c^9O{gOq#V zOEWsk7HByc&*B)!SB%J>yJ167dabj8x0Lg$E>QwoKVs=cCHJMd`Z+rjuSJjc+aYxLUat3du5cN{XP4v3o;>0W0I85}S}^WW$?>i0MqpPiEba3(8WGvN%}8n|1YXOwfoj z(E`KcDZ6V6SHK&R0~#dWO+RHC%JYV(FOjv5@vwExl*X#ZIA=msw*uV@=-C^t%x@~Q|Ja30pdX;50o?*SV zgJ;iijxDmu`A5KG2%#~A{xYQ?`QDpsJPQ}}MM5DRq@EA@@vPo1VnDI{l=xl$1wUz~ zrl5szpa1i%!UM%}OZwk$U1(8U_E~8rEWSha5e)m0WfP1Px7tGoo=LWIC7SR}kWhTW zY-Ksl-#J~XCqK;-d-3*U-shz0!10dEQc;&QdIXsGRN7CNBJYlxsMC_e30Eu`?M;E? z(i5iEE|1v^m`4I=zyB{J*EcqtI>&YVIQL)u|4;wVazZ%*Ip5Q?D_!NYmL4(EmXj8Y zc^s|Y0AxvBXsju@$@pZZH$zR@eZIXarI~fyYhPpOve)y&i3+F1B(r?KG=nzyJ*e6x zjVa4W*Ix(j9X}JVkAvU_R=-v?@jzUQ&WAk zFg>M%z3gGb_ARJR9#x9^Epo`R@@0Ko2%9E=6|vh}&dpgeD=6A8XyOC(%mzbDe+7BR z948@BeDkBQD$J-h!3Op#>mn7M%U@n`mQR;R0-Bx7q+(RLy=uh#MC+Ob(%=+ z!k=Yl8p`W_H>-q3IO>OBO&xW$i39HLRi;R`m~2DOxZGo6`TKK8wS^6ehC2|FLu$<* zC8iAFY8ewhxeTTwbaC#&GhF+2=Z9s7P8(i=P^iZhK-Ewr1(f7@a)1kAoB&)qX4;b8 zQqSh-)pJ<4(@D$w)5~z-3mq|J|gHgRopRSR5B3>IU63--w%NAL1|BFSWATvf$lc73M zqMgldX|EjO@c}wIB$e?_{D#fW-cIy{_*T$TVPKZ<2l9H4;vpY-2(*jN_-JQp;v|#5 z94(0Z>MA~1I#5UKGHnEtt_V*Fyt-b%G?|d{M&;JD;l|F4K$N?Amj^il2B%CO!m+Mn}g`f zG83L#VdV@+t|&ZxZ{u&_&grn>q)tbRl<3A*wRy{Dvvh`$mL1*Ckr9;ty-f~imocBg zgJaZ^5!z8r=JhO7k!|uxdSu;c5~k@tJ>jRL%egHo=aT7s93L77VW?G!U@72#<*Gv? zXxy$t)$YjScPJv$e=ob^#xjM%;-jVGEG}`dsCiyH!M2@ZBI(_(uR*&8%#5S6s^c# z{HDgv4ae0Pz!-4ceV`=uoh;eZiv72!)O{*COts-+r;-W;CI`rp3h^FYh=G3i_}(mRXEzW_^xZ%XWUbpSa zi>phIC@ZFcGeuG~^E50rp71GpuH{TiRnA1$e8#-RM5?u5`l=s|1VU`qL|ekKJWV5WyM@ba+)I2hyiYffG}Ga?RfvDJ!_jbmFDDBlGE3L_ z!EhhWHgY+;Afeo~zcqw7RfHoI4QK7rF%4d#Rs`x{kzS^EFn@V3DcJPl^${a~hKk&;SsyxwOZp3khi!T#5!0NV z%q3HQ27w2+w?z~06hxWN9}$d+ghOi4T{Q)*{}J)#uIA%?Xjwsx+0mjj8}7RL=EE>u zvn%=4#0jH11hV-y!k9~FV?jOQ4pi+{$i*2`8;KWgC~F|)@%~G;RICkf*r;E5x_qYFFP~gyuaV9N zjo;1u=75rSM(1qot*_CJ?Rxj;6}-Uct1FsyRcH9fss}3qZD$kb^bY&4 zurUWM2_Z|m0MK2&lNGx`W7k<^J^w=6!;202g2q4O-{3hGz?wKn_m-`ZTASbK!Rfn+ zqk!l)1{?)<1wWP3Q0v3NRO0kBl)22#`*+OZAaD!40)7;ayT8l8frO^re8OPCR2cmi zH$bIG!~ou5%a(d$P7Y08yB!{Fqc@0Rw-Y2n5R>bqFVvkv{mfb>RB8&`8?h zNf@&>Z{4)cwZk8dz^7PlRdv!@jsVW~NQ}mZxgdljFJ~!^9*oTACl9c%HL4H2_kQlX z#mBBBu%@Qdc~ikN4{QsvXO~2+r8YA?V6@I}CRn^Ueh5vS*I`bqhINwVvQ8AOR!qBg zrtt6upP2=Y*hQW*;`S<9V-^C)k}|*?QY~15$II&$1TXp~a3^KMri811KDLJ?h_Lo_ ziP~vO3X$2ZtB+z_fV^z%^00U=CS~l7rsMP(#y_MBa3NNUKVya;17S1-b)Gz6n_tKv zqmw>)(s95xo@L?YD>H=>6>u3{pSrsg<~~-W*zfuV|F#`f3Z%qpRYY&Uc^X?LT>Zs$ zMGfF;q}G=K&e&QZEg%oYjn>xGmlXL`sQHrsDpnoXNao;)NexG?$9jgVIcO#iC2Hjw z7UC~9BU`PCh`2YBm|fW?$W78La8sZFwxSAV1F^EtXTXgI6|#dWwdWW$$$5&5}M1F zOCUBAZC*sIp;?0m512|0nCh>}IAUP#_Bc+$#@dmF%b$@|bX$|MB#)bhJFz4Q_5iYIhUiX*X!C>aN z8OGJKSwJ81^^67$aQjEp+OB%a)`o!K)LArvl;975@LYg_(!qj&!JKn^N_$@Z-=deG zb~_5(_k4h(k4KMaH(9W>Ml<$cwGtS8Y@ z>&J#-d~0=_2%K4v?;(;}bElZVnH}*B+gw~_@1ORR8(6jr9Y8|>nrkZ&+JdYFp-#Sx zwg{2pj9a)=wdf%NJc;KHM0kqNYzgp=IlbZemc-Fhdbs>(A_sx&@becdPDcFH2Joa5inec)FqPS?U@V&a@cJ~ z<>P3>1lxtP%GOEjP;zWn7lEm%dw@tVoHyM7DDZ3fxVj>jPT7vAR+SeUSo#yY2=cKb z-bI9&vQv9A#uIL@z{re74UI;+$0u2JAd!X`y2)OXvAOXog?6?}YG;GS6_xNzCGdE{x=M^NW z$aN@zDPC;d1TqJ>{vt~F8Op*qEbQk)B5nyffcx13z-?bYvMFhsF`4fzOO&l0Q6(8A~ z6ci=|uBl!Cs*2E#IDU5QEL`p0oS5x_POi@-Py8lo90t@GF1P$hrP$J%5h5sM)2eKE zwn<@U%7IB=)OPAS`+=-oyMGxk<7)aOuahZk1aR;1A6Rc}o1%iKGusVBznN^Jskx4h zwRKfiGmtB*?t{vS_g7ml4YosWdb5bxj|&aZwC1&w-jsQnHQ6E4B8PS?sC4*M9;9pR zIE$X`)iWqv8}_h|4(g#r38QO93VlT?vvV2|=; zGM{A%RVq0@X0XOTjYym1wPrs{&DGo$+W)gfJhBdA|Epn0vdYGKr1Z_ydp}qGD_3&S zxdm}nws##@J3W zKuTx@Yd^+8TT~eiL72O3y%*5fdF7o>m`+jI=)b(O{uzI?O}oD)vVCG;h&4{AwVN zVy*-gSAiq+%L}z)&#B@W|BYkS#QPlLs?iGd`l$zB2_k>q!fB-VlI}$L4-weNB{Y%~ z+v=XvlY9dc#Njw^=$NrAS2@?}-jzrDRH3e1o`&qOx4wb8Wv7rujFkFn2hS26F?O_WE0L(RSPn^VWCHP*pnYAlRhQl;#4t;tU}90!sq zjQc818+0iSQN=b6GNA>u`CC5};(j#Bvv=isL7luCfjP}Nue4#}u9y7^D^}t%ru+hVH z?)6zTNWZzg+W8`ZD&OptpB_59dMQpajWo)#jG9tu!nyxpVclCNTa(ZjDTeW1y>8P3=9?&gl8Y(byf)d2`+!{aa=M){n%re(Q${W}N5x%d1d*2JiZ znSEShJN?LWUzx)+k(0>j($IRzxki&2?%aKzhXZ3ROA%hKi?c4GeS_DPMtLF%Zn9n? zv*Vz)dqYpHZRR6(j~V_Q_kQ3|Eq#WCs_C{>iZBlC6co_^BDvD_(5FEphs5-VW;4z0 zgU#~$Fk_C_s@5t?idJAWGmA7ox_A{Uj;nhcnCLiB)5EtX8CuHku&EE|`X{>H3boaz z6srh&E*7it%qUUZ=2c_3%}TqZVtvB=Hb!9hp%-yJmx@hP#WtNju>ZB)zM<1cA}pfN z&t8GDo4%DU;v&q)kdmoiuc*Db8amjXjjFEN_Q-pbZ{I=76YHztzR{XXcjDI49VRAF zuTS3|glKG&|K2haV;Rn~h>0xWDjxS?Kr_KZ52>sIcQeYjRJ36g6?H9Dolnfa=aJ^{ zmzGhh>PKT^sli&q23V^b&}Ga<&Xe?uA5RWcxUZCiLgE4p@-UGGnxg}?>495(X-%Sz ziqpB$P>6!1gqVYgTin51S5!M%G<~u|O}Mm#k74}iIvA8N8%dv-lXH4e3=9YjJz(v~ zm1pU0&D1fjY@~U)<~WQciC<;r!9CFBv)m~)o9omxp8xFB!DQ;j6XT|+3IE+o!?O|* zoJR7hiKb$I+Xy>Mt+yr%`zCz)? z2=nUu0SC@f7HTBvnT0%L2U`RxREcx^qY#3_oikqM;d~rS@`9;qLT}(5jdSst9_^p@ zF9asJa=fz;WJZspW^%PgNZ@@Vrt!(0gayZxgcOG+9gI2rQB+CX{CaPyJ}$_;6+ zGT1uX)gq_w@Cq^b3n|Dgh-EwunsxKL-LN-@W;V}Md;qdAlSNDci-Q4Z!?|C(MSiBhO zyZblIt+RA9maucz`!~i~5}KRxE@Td*mM+E*@uL&`m%A*i0W832xIz8WVpm8axn>kJ~u#5Cpms80jxC)10n0WWnnf$ zoh>ctlAmwopJl_DY-eeQ`j;VQ`Z1j2R$*O&E1dzq+B}r_@&RNy9aut*MVf^DZmHR- zt+Q}u*7vCo25bSP`L)>JPkYt}B$3pJ-Fq9CQ@Ief={l?rZnf$?34%hGz^d5|k%cO? z-h8NXCM!-e7fn|!Td&{ZU~O%s%G|;JOEwpQVF$~+PvVRt&k946q;_sq4$9Z-^EXsg z4`0%x*4C^68(J}T+z-hq8M8HjdFoPnSV0j>bIa9+Vbi&y(zNb%op1v)Yt26{Vt>es`1C_2ydJJCKuRAhGNH?olVF@SdosQB#o)~wGM35Df% zbeFBIec$<)dCm@FOtl6T;WK26D`sKAwrsEn0kMzv8ge(6My_j!ugPl7RkXa0`a~zx zb#olnvmV3W#V~?>DO??P_vCf~S`Mw{&G9|M?%@``7>at*e8~xtfeyhLfKkn z`B9^}H4iJ99fbQUl2=02wr;EKI?smq|*-0pfGzI+y2z@`E7R$`N5UoBe4gNxWSA zDqP~xT=%uP>QuIO%GCvkW{lbKS{_~6W zARQiaw@AAmk@9XhScTFAj3O*rHq-%!)5a=xQU7@Kz9(Cjvw^D$am24cA>-^$Dbs>m7YQ#jiTU3R&!SIetn#cnS8r&fA6W z19Y@DD|sP(Dg>hBB~Mh>`e*AL8< z?A#*Pe#WYeAx;t2Dkb_P{=jmZo@4C)7-*AtTxs00D%HPYBe|kjNGL9%j ztlao4ZqwTSdL&##xdd6CzRf#pc)rffuI*f@#!fjrt)=Va=L0i%Qu?)fBw(J~i|I$V zc5-HrKU-oW>XiFKf6jPG7`g55wLA`+7pL2R<36vAa)^|^j+IjHI~%LDDZRv1i&8vtvU@ zM^`r51sTR$hvi3nF40t_9YiX{WlK1cHcLem-5XS}s}XuIFyJ{DZrJaNy#;{tN-2wObeao>>pRGk2aU?mG)eYlu9jNHN1Z5 z_9YrNDG-7w$DkP^qIiZ=Z)AazUg{G`j5|N}XYs*Ody3y)nw4KZYlTuAnO)O`#>1r_ z-M?Q8IW0%c{Q6lRCZp&hdsha*#46J@pzl_gw`;B4u{QczwDh&E)VcO~UiCV%kTP_p zv7c_VaJ6>TS$%NQO5?q0Z7q}@e$pCu5`80W(av4ulW!=kf_l&=g@LDbnjEl*K!Fwn zXG1z_LDy1IWyhp-?Nj3^v*4=M`?RxMX`($}!}@k!-WZb=(=+t>gri;5;>V?8;uJR^b^4V9Q*CBgRIQgp3~_w z^IDXcZowaHT8Z}z8|?O?jLlB&ErfIi@H)LoY%&q4rV(#a+OE;x(l5njeVV$w3GM%) z(Yzb~z8x?Kn{=4VUz`0E8HUXUyZi5RCWQ{9ogjWwYj1zucdJ6QwB_%#UuT{0~!#J!fFz<#rL6=b8ma1v{cKN;P{4 zXR${8^Wj*tr`}|`)lah!JzsD6BnvpdGH)8#b2q9quPvElvs?{j%{|7MC6v9iql&WQ z_|&yhlxcWeW2et3FcEB~v1CtbOKIp)$Whi(Z=5y&fj4->ROTwkQcM^5xi%S#Qa?UL zeZHr9ZQu#gzJ56*%rYE;O6?95gBpCcPafS)YE_)Nc)Ng7EHx<_i`$Cjh|Wm%#B0depN17>1g)BN(fh< z-7x%pO#;z>ZMxjO&$pQMVGqan9PQCI*P!<)Ykt^*e$Y4Om2N{eTZ@QG>0mq4y#D=NTwg;-+jW|&@ldrnGecNFz_N%WEMmHFK^kWwZT;mCL(LjR zeWs+FUkP}Wt4^3Uq*z&4m_7?=SBt5|to4aoY-r4a5WgyLmLv1vbi#g`b)Is!l*MC~ zd^hi9mZ+u$tF})0*|I=oEIY2qt}de%!R6n-f15QLa&1BUeuKGkXm5WV_Hxcgs8qET zLz%e0{YqoF4eGq0FVvbUHn1ApKMBvy~?=btBt4(jVf^-YIU6 z++WCF;-G96D=^IGck;vdZZ=FL7+07)BArMm8~L&QcAOcWMq8+bQ~z}TLolS%7Ky9( zg>}eQI}FwoZw$T$VL$b>l|hg>Yij873zL8ZlR6GbhWouUSLID}Jt71riqKU%AUva6 zSCHxuY|pMF9mQhPSoS)6MoY}UU?iZ+d&3*)9tkvsznC8g;a(VL?;RDXO*bSrA)nF9 zXNttT%&gmv%R2t^v;T?FC9f`I zNTyvf@KOz&y;JZP_6K22{=5Xz4CyW14edFC3+^wD=^ zeHM)ksKkF>q-ps=1(r40oOZo|%DlgP9XmGXHs|N=lIe}LVSEf7m+cGG%{Q8h;)+zF zgpxvzWA3py3NIo~9JlRe$%VFE-!yUzg=!cqTU?VumZ_!mvZi#UQ~@_}-tJWdLhpEw zW#>}H4apc!Y|v%bS3EdjA73e2F5J6}vyfy?`S2f8NIhZcHYUne*3btg1m@tvaW_)@ z80TBU_%c_8u?m{TJjT%nd!swAK22KpIfoj{|KyLjL?Z@H-S!w~*vOsw54%575$uyv zBKoEyX&`O9CPTSvn_GI7>jh^zj{&l2%T6zTD>|E6cc+Mrs0*8%Ii%|9ej+Jnkff6W zbz&1P^$FL6UTR8Nh2<$c&v1aZ`%*An`axeDhrOewDHFnr48_?xpbTvE!UP&}A* zBysDR*=Wdpl<|BZr=0+CaYAi;=`t-lwmq=7D7TL9nXi& zHO(8=IWpQMMrn{H$4TywsCe*W&IVpRuZSjYD7g)tFPyk+Fqg z(^1tFe9+lh>~6EQH5{B&)_e@s_(fPL1v$PjQ+qY``Iez7G)J_Idr-l3?P{8VRonRP z%uZib_SW2nEvM1d#xB&^<{H_=8eqrOorl)vYPL1#CZb0%bBr6wW1UqcIZgg$+l*#r z=r)@gzOmsi3LkpRQeek%+9IXyvJJ^Df2rMD^Dpzxn>+RDx!TxbpYMWrk1GA-d_Xna zWp-~Tzu{0faGsk3ck`8@NcBEnDn`>AmSERey<@yWn*4&F%#NignXgM?0v=Sy<8|SS z*>j=uV%mOTHpvfM_S6&JHATVd^;vaLr?vf)nF5GneA63L+?&ZB$Jb%DxO#+TeMlB9 ze9(Z@!Y@CeN{7u0CwbFHelkhDWWGKppPr;3(%XEy`vq0m zS&kKo@AHa<^K#pbOW9wb%UAn62lm?&%*HjAn)q2~EH~z}?9kh;izAszCrE`h&F9f# zrG9*6fYy8NdMzj)TJ0-+hRU7XMM&O@_#?q2-DjcIx#DM>d}5{k)^h3xCz_;U3&4h7 z!w#>{Wl7svw`nZ-i_xErR~;NvUH&*2m#V1MLw94_CY@$DhrRL+sY#e|%c!7@k>rF# z;CGejGr}w{wmNpu3_NNqzDpaqDYND3W7r=;9%SKlZG8L8pF-0iUgMY;aKQ1b;6S8w zR<@cf=UAO{{W)i8rdAxacEH}M8aOcM|32CASh#eyZD`%J%fzz%Xhe0H(me+$#pmw# z09T&*P=zZpCVG{l&-jI1WredMN`7#9P$uD6riaOZP3858p@RIrjjUw}!rZ!z4!9My z3>il@*-)byz)g5wM56>8o$P4T`HM|}RIm5#{$gGx?8hH z0XBUX$?W`S!)$E0?V(HFh2J<4F5r~Q*4&NfuP__=B8G7fuTm=^LbAnHc)w6$$@3$Ut5SPw0)5AQaFIi}lqST#?4Mv%X?WBFq;rZ+AhmWC1zi2H zaP@2ms?^~t!#E4;GqH-rdz0ai92ma>hq#=POgMx$VRSeIG8I3^Fy*%hrka@;kE5Ib z?pE*Ow0KstBd&!sWIhvU=k!sARHwsX7kdahWCb$;RF9&Zuo8wP~74q}I$^v^r@-;Pq6oO_C^IF*STxZ)>46jvAGJLxZcA zB|6u66lgF)SXw$o3Vxt1^I5n>V4;+a%3N*33*HWUD-i&<#$zP=oh2MbMCbx z5s;SR{>qt`^RMMBDl3-WPZcy3UON|*NIlW&{bR9?2cbX#SVXmB=N97fO`$VF`fjuH zS0$MK8X_K38H?$gwYdT?o`(>GNo4--di@flqJf#IB}7rD`TAG>26OA&7)O6GJH-B2 zWk^wH6?ZU1;xA_{;Zp6~J^!*-oxvmCJ(w8wjpZCyPxs%*yrBHH2uR~fn0>w%)(<$G z@S|~}CsrM&@klNtuC0`eKY>c*ZHrs?^;-%7SuKw_rX+e4J2m|kXE!2g`qI==H9x|( zKi2}E5v-#9hUFK7+Q%#o;$njLHOCD4i{HVp!JjJp+3OXSTPmL_T8XwWB9@xWt+yV{ zUA4=I^%M)+XQwDB%moyptE&kezV<5sTA=5AKfJhsK%(gL;)d&s7uDTO-0aGv3$ zM8u3w7%n`dl$&A1f$LiBR(qk|BN#)NpI>U{d=Zx zD&+g@y;jwp^UgfZj|_SBimu+@bXGsM%u5KZn2R#+*yv$l81aO4tee(|YwK;^77zQG zjd1CbkTHGvI6xo-C0aN$yiQBD!yxG^P#bmj(R4%kfh9OENUx8n?+z`R3ZC6Gl6xEB zVmcpWpKfs#e&m^iDxJD6j4T=Tk&+5=Vu8a9WZ-W?Jg}DH>>#Gr>vqv(M>s@5`eWVq z^ncn#6=-lfU-JfVaYYOey%{trmv%@gl9vXF+zobMe#~EFpbS<;t*x*dsf) z+)}`CO;s|x#nmJe>20mjpjzy!_tevARyPnnq7iDX(QW$j=@{B3-t>)Zw^?DG5ewi2 z`sawNRCUeN6>d+v$b>_cV}%Jqcbri{z*2nQ2i*;h6Wpj3Ll!84we9_p^-o9Hv$$VA zM)30PUeCpP7a1T;(`1SuL2^E(rnbV-ySfYX2XTw z=uSi5_)^2bB1CRk#R5Ad8n(X6<28PVNFk-o+<&%>2oqW$AW{Z zpwsml(lr0FQK*`e$VO2-_9m=k19@{gyR(9l-gyeHMX#V{anr8%MbIIqO@TPzL*PO6 zWEiUSq4UT(C#lOzh}6JC^$7m-Ntk6Gy_alZQ@BkcjqcI2JS6(G|>t^65DHv$`{wr_~EOTJW4A%to5R&e2`jI=?>D>Mtve@Z*jWO?eH)d;M0; zqhZX*ta)XlX#%RZ5m~~#*TZafy6wp9giFzn70K)d-ahBs_?=h2LKkLJd)FMgM3FmA zF#IG>MXGaR-_B{q4_pefaqi_JR8c-8v+F=+4Hr$h0CT#3_cSQ~ajy60>|x;1zHN{t zoKO03_A~Gq1MdS%7Wd{iV2LwI8OLWVo|>p>JdaMIDNEPe%AE(Qs8dU=AE50eNYJMz zIac+I0GO$0_0GDV;To88J(m9V7xPiNnjTv?l6u0&Fj5^o-7s)Q`&1b}*k`PFtlUBv zN959A%P(}n3Yv%2ebl4uN`blc5;Yr=A4HNNqa_n3JNky#r*igw-v&#>stv-y1QgkZsACVu)xbJS{ih(34_Fa@-Kt0UfN|>29C+#8}Dq4s-CTs87BAkN8lk( z`9=zto_|y=k5CIS!4~w`Z_WjCA;v*2zcFFAl+nEFh51(CYz#WOI&Df3YH@%8Xde_2XOJ_#y|RLqc|b82o^f^}9ir zK+Flq&0MT21z7=ak$!0KfidbE!??M%FNFb~UnP`k)-|4E7p!8%_`l&_xz29e)OqWZSK*dAuw-V5OG=*@`*2atPua&0|mTrN6|@<9}{N5`NIy5YAFl z!mQCvFT)7Bdu#<30X-Oh+G-oObY%WG-wkU0kik1vONYkFTKKcAh0g?u`odf*f-mRd zhR3R(LZPoy(?%+W6aoHe1rVe-sfGt-oOd(LuujUyZ^OAFhFr1Om0Gp^AUgtSe4ng<=MQ zlFz_dZHJF2w4gal9-abDRL{<2F5m}C{e2pNHI$d}JR$37(64r*q^^3;1z6Mc0|ZZj zNHurv0$^7}uVS5w@4n;`1gfMLvhmGr zXD8y3NbmzBS6A?Syx>!eH%yGxy`}`N`MG8BIYF&B5uKDs(mQHk=fW`ZJ8J;n$Ir(w zg77c}mD%lF(pV)xt_6^D3<#gi3J8UZz+M1w_1wp&4#1oyXM2FV6%K5>83Cr!IYJ4U zTs)R@KP~Zu`UOvn!U<9k@^JYKf>NwH=LdT*N_3Ck)~#xq*8M{O_`k~|E z9ej9Qz{&y+a(g*$Oy%9xJ)K;ZJP@S#3)vKW9OD~#v8_n$1Un929RT68&SJ=y`L!hH(sn7yyK7Ym<3a_3o`Wgj6iiTZSrDw%GMs`L zPRx$yY@hm{ANFf4V0h;8ks(R5Zy-s3-oo9KYhcmJ08t9cT85l}M*>jte;zMC3Iy?P z>RW`j(*N^qj5u-!ABC&oybpNa2qI0cAxUq64X49K!vNxQcBKgzTqua*2*CJKP{M!v zUSahq-dy#Qsl~!UAOEwp^@hWApYln?NKg+S7Wv=mT{4o*1?t4Du>9zaTbBX=w0@5N zi#CvR8sL64DHr0=e~YQNXo#TyB^0FaP1oB!m%=^n<$ky$+a=v_LTC@Ac#`JURcnU}?xS zC4)mMW9gIc^pXlYvw;Q=++G+z4Ek>%D)!3ilH8Z-z^GGrvA|80XS>F2}j5- z=lp@Xz~i}U6zv@Wk-SE(R?|M(fm!+=UmZwV=U25C4lzO=7EVi{8?j_zx34h5#}<%} z;!v+|vW|N2>AeSH0xIEoedgeqFv+ar?qv#i19{U0{whzzHPNsdTdkEaa7HwbqEV-S zs{LW_dK}Y@o{!Ny5PXGRN_ZM8Il0J*;!=(3YH(!0^y$=q{PnjO%|C4RgTQVD+3w=+lky(VqOS zq~DM~9+4Ny^qJpW6l+N_J2L-N!i@-tmt5isn=g&PlCRYyfRMk)6+!o#x(zI@kt3Dx zfIMvtcQn6fK}51`6bR4;(snkbF`c7`eW`i)DL|tr@x8KdY<~42|1u{1L@`2F(LUjX zoiV`Wodi)Eewd-m@9uLJheKA$gL?jfkl3s_x~9@~fH=4a+uMmflO}9r(LQe(z)S*k z=Z9HN&-3XQ@53QX@e7wzs?I*-AgasyOOMb3(y4}IqxA|&60Lv!KpBE`skb$4*84uCCS_Se-tyZ>|{x%@OXX= zf9Jvbw}12rpK1c^4^f*z>c+!2|0*rWH^Eynf^Ir)?$%Wuz$XSk{>nw*OeI(?_kGOA zr@Ed>#^8>LbN}i!NKwOEqCaMhgalYIqVWKkP;Y<#Ya>8X-OUI8`>VV=y+Awe(Xo4a zH^=9s^D9qnlHTUAkB7%#;daRxl68<9ipO6+e)A!q-dk`+3CpPeR5HaP1g5nRy(}@Wn|9N)&9hX(OKkst{4*tMw z^$)liNy6#X1pe+L{<9g>{0ZO^4EMwByI%COMESaT?{Ih7$gOXC#CC6Es-g&g0}cOq zDcrtUiY-_QFPEi(%T}bp4K2r4Qsuz49IqFThAd@9!95ywd`A6s4`2SF#P5bANY62d zV2CIv3Yox$lg*Eh4+bA@G=$7(N|xmgw(m7+ zRwo*P%Xz%WqW+M56Mu!wL88NAU5D4-Gp?sQO=`*B9&1UCwjb7}d_zF?L<+d7E8YJy zG;jZ~&qr!!HN|J+x36UjxLgJ#5}ccwwLcC%gKP_;Yagphq2*5wFg_sPkmZgEq(g%Y zkAXjHs;AMl458q1!6zJ) zBUh5;%R-8&*WMS_9cS(2?wmZW*S8>YU_EXKBYL#gxd)Nt+nB3aJUK+ufcy+DkmfPL z7rA_K%baMh_6d`}GpBv|1AoK!}V^6`my`C@!2KG&IoFN{p4Sxk-e`1kN|`Niv~1 zzla3_5JVab|IlC#jsx%oiq+i-L@u#P6O$46{&3-r53%QX_df|6?_`LK24T{88m>1H zvnl*P2N2(*Jgo>AnTfVI_2M{j2><++9c)w(m{Af$G~A>669gI$;^v7Bs2&k%SRvAI z)A&z%g!FY{PD}1PeBOk7{m5q`o12y15G($h7V(P)WIzCEvU*4C!QeeHapT~T=Nmzg zN&ymgXw%*hBrgGY{qbh6=1L6#6#BFK8$n{;1nq}zVlu~{RjkL~;mkfe|BJo146AaB z`t@xrL{z#|L=Z)!8!SLc5fGJ9KsuxsjT=E~fl5i2(&3^Tm6Gm;g>-i)ea5pEy7zwL zAMd%&xATc>F`t;vm_6n_eluCMLqMGnP%K*tAs(}luJ03rm0zooD=oLmLz4rs3_CZ$ zm9`3zQ#)h&Q^efy+&%@`1QM;c~70D12pTzkZL=PL<>H1y*` z8jQxj88FI_^St<$J?-NaStYIVE6mcB{p#}mFx#Jv%CtMKlol-uW1tQeDMik+D6qTG z??ldH`fazsboG%zuZ;h*|Fy25DGJRb+j;l#e9cA<&(&;%BqJj3~hn~YrvxX59 z*|3FO_i_!RQ7eO=F&<}0XR}C2AF*PcX3M9!i@UOl!h3Reb0v+7pKMig7OiVo^!i@* zE*xX*^g>@$d+0N(%Jt$~o@?POq0;^A{CLZdWj&M^tVujmmAO`eN>!T3oVnXdaJD^c zV1eq%qf6)0<;)Q(o+~hwpK~hfDK~o5%Quqm+KHa#$CS5A&Jc^1g^HBq|C$?`_YM8k zpIkoXPStDMSHkz`7DgPh53Blpw_EU)kVP=IH!b32h5w}4_ki}UHMgn9e$oaf;)UxY4%%orU_i%kvr+CpYh@8i7`&m;K=?5h~Js71<}q)Tq>@73{^+0&1^H zJ(PPqE<0wdm0b1r=#4&=P3N5e%|<={o>ydv zhe(W8croO5aqlull49DExV}fG(mZ?}Gi>txgMZ&$q4%OE+Fkgr)7fXl7Fdgrgq@lz z+Eu@7uk3K7`3<27Ou~-d9Kf~y-18GIer74_vQ!*8hLVK)XT3U@9{6C{PtAWI^IJ?# z8B5>y#F{Z(WUc>GP;a-Cy8JcMbisnZ>@RT^1;`fhmd9W0&svy*+R5o~6&D8Zs>4nu z7Em4t12ExSYq9Vs&cX;<&adsd8p!Xj>%xqRE}rl0+NLa>0_O!Zy#Hctv@ZkS>DmW| zd9DP~Nrk<%YkkbYED-z=g7yVFe;EnB%o2NBDLTwyX#g?&*zhghffTm+>&3JNx&|tl z6*Z%871YAa`eP1Hb6;1jKWW0iCM(KJ)jPty2lW$WVa@-B`eXK>{`l%btGeFjLiZdn z#r-CO+Xd=rB04DOjz@`)=~ODbvp?c9dH4*5AQjw21rW>g)`9uqrn#=1zH>sQK3^rR zkYIrNo432h8RBTLR*j|g*fWciu3?q@1Tt%pT;u4ChgI1;shV!N-zwrS25{sQQ*z~q z?*HO*)eH#t)@*#H)BsU;>00*_%E4F%+l3TO`)JA*7Dn10DW{SstzUq3TMfNtqfc3x zJS&Ge%!02&!~pQIHVneN6!l$kl)WLbx9PEuJ(wJtxsFI5iJcs6rg^f?tZKDSl2A(3 zTIZCQaz#YoRUV0S;BkjOQuaQCyvUa12Z#pucH;B!VwS2KQ&s#k?q^oq?uo^aRETo0 z%G<1v+(;TkcOCR+^c0E>63cBMcj|7sm+_g{PRy6E<~ncpQJ%a2+1(A5Q)r{A$g_8D z3emmWQZsBN@w_&T&VL;-h?V^f}WepXP7z1shAAl3k_b{iH`c1NuYSEezJ8#Q9XpH#Bpcy&a-f zBU+gZ20$3E`CwD6dLN6mTcxVjj2^J{n|`708FR&^nBjibv>iQMnKom_TRv&J1gQ!u z5`>D)P$-JOig9gjL|=^}8f^yr+!UQ_wdwNZTP@o)95Mg4zpEq5GUh^mwmj)|&yx0M zbLhp_rDF;jJpXV zPJ19kkXqL2nd;qv?=Xe4RX_Ivea=Hwpn&tu6lg5^4L7qy@Z|E;fziARtCy;bZBno-6u7j%44fB?-Dld%KNjWCN1(g zObdU)z+N~z9;r8s++gaThH;fllQT!oNU0d;Y@PaS&%zt)=InUY6k0hOXmQI1D=sILD1GtYIkhUz_o-bA0d5XmqmTy>9{L|Be&bs9nnT&mPkC)=Vr%_go(@SYvk(fT-QC1 zYGsp%juxGe3yz=igHR?;*@PfTD5yg)htr4Vo9$AW6O-(S^vqF$u$t$}F=klNl`eMKcE+Uj*S7iR2u=8w z(lMzyV1g5{3n$NIfK>LIhnGwYB^BHJv_GDbV@Giw5aahFSCX+z_mJ2N>ap038>>F4 zkgwspS{&s)A5{xnyBd_A3NROD5L@A0nX{D4jTGDgWI3~T}L^GaAA5{HM zOnTx*rEs*`btIYsc6L7MqBvlyPO&`}vERN=%t@Hk#1NTk9amB=`lMj(#Ac zeL6dVM3|NZh8}|p>6P4k#$MJW-Q;@Ic74V7EM>$r3k_b#YR9&j3E9feh9ss*W?FnU z;k++~4q6m8IZ5%NH~uM8Q1Y8-Yi&g|x6HO{?v)erXA=u)z4>mCbQ0&D@x2>)jce6; zl)kOD;^15~IPe<1?6}l)uaGg6LsNEoLEczXc`VL} zzC9>Zr8_@gdH#q`HrGJ5=D0<_;FF!cf*}#sqghB%gR4v4Bmmm}ED%1yc;-4n@6N8o z$+nddQuN6ASXW|&n2#9UXiBactNJ>R>sm9!Y?q(3zedfFZ%d$eAF7M`$?KL6fAba~ ziJtgH)to#0-XuMr)|^w?O!bwBVZ5sI<%SG-0N~PhX)b>FbR^{o z5ol*u_E)P{IHfEm#Qf5sUY$xBa?9;C30v#BL*#zGSrZ(@(!~S``Gj@BCuitIa3J?# z9a+=@h`wZZ^rK+d2^f&bnrC&|Qu$Z^TlX0su+L;SjA2cvBOLxDXgPo=p z(AgNP#0E5Ui7NFiSp7`pp%F3r-qSSN>69zD15z^JOiD!X%ZFuE>Vm(b$YUSRllqG= z9QnDZGal)*mWXxS)(0JucE`}pmznQQmophPH=uk7*;Y)v>xM?2p~Pos2#=105 z5~5M`Vs8p?L6bJ|OYe9OcuapSCn{OzuS|bPFSkr!o_oxxDV!f0`G&P>39*L|rlG(KC zX&$*OQN-wpNz7{i#rX=S`-=pDH(a94Vz`WRiRvlcBczQ>4i$wuK?8B z#8-N(FC0VdcO&|3T}EqsgnTaNa`K%BTeplNW!IFYMrmApWzzjJazoOK_qjq6JCB3y z^JPnRo?LUD&mE-m$*Dj2s~Un!Y|d2PeL@r!g#TlzDX$CzCn%jbE=Vcw-=WK6`?OvXSIvVpMs7i2Y ztnaHyr0^#mMeIyhD@3&EYf>3)J=(c(r_nqrMlJ{w&$sXabk*Z`B1r|kY{%k!@zC&h zlzE}`L4bx!5NNpgy1}^NKcL~acEM*8O$N4Z?mwGo?b~$_h3;w<+0sLCVBG}L)Qk;+ zcHTZjAC9$NEb0(^1S^iXK1%GvZ9cOVZAV@s-xE~9<&?rt$*34i*sW&X@%XyAoQ(-~ zda-oh_p;ylHnRiMab9D)N2mG621o`9)20jV-*V;72@cWlMVR7e{qFpgx4Q>B)W(Pr zUBZR(8huo4#Fqj#o`Nu{r$)MpY4x*M!n^q$fyJHG)A80aO3PQ*X zXs@Ib+iiHl#{6XNU_(++?^A|H4A^`$gT|K-N@TkB5+kR>xynCFZp}ZPJI^6Jq>ksi|^uS|M+3FRcS^Y6Hp@SxHYcm3_dq znh9>T-}ZdcOlYz}E;+V@(SS`NgKv3PI$55goLbHYc?vZzQY6Qeae?$@Pdl}NJVR%2 ztDsUnKvHw?km#w-u0R1wtY;ed;2`4l3oroZIu|-X5wEi^_40>Dn2)Be3NzJ$Zms9+ zU}sxdVX5ZuYEJQuB>~D_&u0)Nv`dn`1At%;hy$3ePEF6hZu$LXH<^+K>y6D?<;o%c zVT_Z0?L%$5kvt~;g`YC$Ir^{m{(8jMxiBoy-UZm{Umplf+8e6XLIKE>=PV+TBREBS z47{wc{ZRy=pbw?Wl+S5~H2kH5BT^ff^&4`T*1l&>TpD+Ef7Uo zg;vI2Xmg{(?ens`;PnnnX7sDfxeY-?;8uTTbTlazUSR-Otm06rPH$jUn;(UQW+*r> zp(!1d-%n_wDfNsci7k62`@}riB$IHXs+pSL>Q&1ZBzhyai{0v4$Dd#rEwfj8mSk_S zdT+^~X`tr33GPC+0~k1qo(R0WZ>f^4TXVmwW7d)WryWGLiUShTZX#m8x@z6VbD>QC zcey>g0?YahC|JcSnS-f8gW5s4=x;Wf9q5TK+K$7jl)Xz2&B~PP*UG@TxmP%xSK|g# z$Ds5^B@9qko${_2b1|%j^VsP3tBegsVOXpICu!FfISKC1J<~~=aul_v0|<2&`8EqZ z$^oWKR_+3x*b<5L{rS}N+=1l|0Vd^2XF=c6tpqbyV9-g^sjgZ}8Cs*biVeQ6seB6% z(d1dj$1cCy9RW+zLWc}shM~}eVz?u8WqkU*hV%~hQW)1vOP%KNPAj#Ox9`<;tv>ot zRiuDCoI`0>;uH_rb&E(zW`0dY*2s+3u18t+RM@Ekx{lc2S-l0NlFftwSh8iITbEsE zFMIanFFalSYE@V!%_d`X&G#x-pH#ZY#*B)p;o`b2Cv*7Qf>s$oq+3QHrm<@Kvkck@ zyt_ZRG>Cd`l_{G;4)d{8*cHan;Myum<0Mw6+Q8^(QkEhSJt+x7o>-#VR8L`nptAZp z>;)V1{v@s4qV-JprmZ@q=*%W5!&)>6G-`X>TcwwNnO! z`uuAP=dQr5%tG8n?S>*L>VIxsgOveLWGWN1nCCxaL(m2QP zdJUy0Pu7qw@~Pmm`UyXj zkeGtGk!YE;jfCIGR=!-U9J_+F@DrtJrBw0eCl6L0v~%^%WFQ5f3y77*Wy#^p| zt5KP9*XHx$KULBqJ{`f0lF|0ss1qL%;!ZD~Gqq?V`cJvCZbSo7lM^zTGsyBe22*cd z+_*62v$6TL&_>Ta;GMNTs6`C7 z!Q*z=4cV(&W(vLtCbUs+Q041}MT%zv=)R7~KNl>KSRkyD11P~ayy-M+?t*rs}*T%dn+F~b3OR6Qd&7|x+K z8L`Au*w}%QEJ{ikVdC6;P~drj=Pu(4oo(-_8TvTd%$ga8N=alRNLDsbZpc%Y6p=fq zY!&V!X8EjuyFb)j$hi|JBMGP*fDDo`;IKNW7r0yz<6Aki!X8Tpx-vi)W}6`uaT*wI zm>dwaGs7AZptz4XD>ud6MlvuB+YLyfFjEXJF$qV^HNCrtyAu+S#E9>Epd21<3x|#h z?t~3-Bys2aC*)55YeC4jH^r*p?gWYmNi3PfCGKHyZ@tuz#M-?(X{T_Mo+ZV6V7^H&2fLCwQzq(UgORF0LEO6)A!+49jIMXN`(Mmt^*ynI0T)( zDc<}?@AWNkn^gLimqHmFO&XkCql&MODh)D%CHe}}TTC~V!9yJn8 zr@ktLU~WVryiScVy<-hw5}H)Z79cqZ`~8-!+Ys=wmxUj2vdv8H#(~rBSaHFd^Qcx< zT;ZiV??Gc<2#O{_84hZR(#_w%*g@>d3E=CBDzBd3BheQ2n{2ouJ?1dU={;oamov{c zuyp&<7mvFG+;JMfFZ(Z{jv#yM+=)7T7g!Pm{1W~lT?!k_Ukr*4Tm~=nBKp7C`X5@WAxPbRu`hyg6$mxh!~eTXj_zYPzxL0<-Nyu|z(0!Pz|CVD zd~0F7Z0k#lLG@j1Nvz^XwEo6{ns0wf@pU6P3(1@D94$Qjw%PlIt@x&fPV%9@YlE<6 zmI8G|@B3A|nQl`QYU;%pwxZ$DvA=7OeWh+9uV602^nU>01bP90vzd{jb)?1-aDV+> zEe1KnBr68UXlt(S%i8MX0K{fU@NbkYnBi?^e(%6`$mYt=U$??BHob7gi7*_GkJ z=t`cNm4?eMKS!RgN`5VHFIxOl*9JK{btgCFwONZTmn;X?os0`CcUnZ{o{(L!PR+Ax zPOHilGOH?;${%)IEK92js$bvSmE68Ys>Wu}okn01y7Y6`?57LRZCuxdi=KxAQ?^~O zS~+=QtS=$5#gTcDRC7zb#6nN&rbDb$RRj=q>msT&j`qflen7fmrx}%Yp=^c#xq{W% zkIcH}mUr2HmsJ;MKk;lY5Dcsj8dZN@+L#yM*14G@n0Fy9`c=hH zYK6+hDG_xiRduR;&dhsNp_J>6%TV!0OEA{H_A@48zr886hXzBOc5*1pQbPFFnGp?T z=8IV071`fx*6nFLRi+?QL^m0ku}9Y(@5GB8+n>k2|NC0kX=kE0w)^%$bjkg{pZ{?k z|KEty^Ai6r|9xG7XWX1c_6o_%27d;S6aFhsQ}3M#l>XoR4oolIFhkYGO{cF10h&D_ z>iiQQwO z;Y2SMN90s8M2TPbA^v@Ey{eLx!lDW52O!xK3(&9W}cEteYK%6@QUVrF4BJpD-z zTw(x-Z9NB$NmwF&kkFN#s#hjbFghsCgG`YUyM#CEt4R1a=+(L~;{V>c6@vR+ONO@^ zt^RNIU|3Fo4(&*?B~B#OX0HR`nZ--@N@U>$ZOxkTBEtE5(;_+=ct1${zPD_{3EtYa zLHIXl78k=-3eo|cMTCJN-OK{kkf2gr(5i}0-@BjSpVGjP&fg-5OGhv)WGehLAOHTp zMu4({^qw|5Tsjhlfu!GF2aE!qZ$AoBgpmHUHW&GhT;xCv=z<>vAmkay^A*BWfOS*sZM>O2<>7w!00oLv%4$UA7jEmn0*f#(57Q_vI~}L+ zg<7Majmyo%y=uJ9Z9d-0RC|l0QBZclsrx3r#|zayTE|_N4SQn9z`#6`1WZb{?8zXUlf3H7Ii{QU$ z=s7NvmZLhjzk91M&mXX&K_1$$jK8~b9QPN{HDn4pa@8I8bK?=jtAWIGyW1}A@0Gn| zrr*-LH*x=Vy^jBlZ#3T}-Ese2^|*=4!U+ZBw*##4@ZUX;kY7*=YjrMuUvU+LGz{U+ zHu90pDx~A4tS_%mRJtu&|<)vEWAGZL_!GMyE8y(Y>F*m(sAOt*boq6bhp2 z;p(q1OdXNP)*$i;D;JFol}k0wui(G$vM?{-cv5L7@1Q^@QD3%352Be1a(+qa%>kOV z3Nohg`j-yx_z# z0XS_lg@H!b?w+kyMV4tXky~?Lcq@Xu#e<=&Hm)v=v^|4<{O)^?w8X2F z3!lW_#8$7yqigj)P4CRbe9aae?@1`W%#GVK)L}zn+7`ct%#t4Y!deV+ea0i!F zhz{2v2bKCHbqJb1wLjUqNN(kR)l0}++1^5O_ksq)RESx80ECD3n*5p@nB*cav~3-r zx`A&ZJ)NLc4b}57qh@8!BO$%!w{pJiGlQ|zE!7XA#kJW3%Oi(E4_1xDs3~XhZkCyG zNhv4^2bTObJvXv6qCa#!4g{~f$!6_{on|uVfvGAlHlTrnI32@h=t^?gk4NekP7iQd z9b9zjvq#hBo73NSmoTw#*pBP3#8RBp*P*@FGdq&&tDwTR)}ml|wmNC@)>GeGBQec# z!Txns`G(J2Z4G1DuW_-re4>}Ek`60T_y~BlbcxQIh|Ja^{SbdzDGzSlR8E_fh;ObZ zT${VJLBdmX1|ys*tDfcN0J>sCim9U;8XH}b%iBE!1m&w$C50hcsq`J5Gh2cJjd#RcP|& z7)rdaM(dnk=b-8|;Ve?M?0@Y+%twU;no}-8)S+A4bHg+@K5>Zpu{>tfAt1CL_mj;8RDd>-v>MzbB&C%!f< zzB)A9KMUD+u{ZJEUK1tS2>hmFsos^`e%sIUqmH;P2^`A@0IbJvzzG1b%r_nYfaT3K zP&;V*T2LjEbb8a-d%NWx_+*{Rq$LxvL^SMp!#oF-f5ZBu!EjHnEeI~LuhN8G(LWIf z8cw;Bd~0{3CAY-y%`tViKJ>k{!$MOPF|yHd(qw*XEo$}W5!^bu%Y_se2LeKql4@f^ zNj+K`?jdx5dkfhtMo;rxXboSY*ASw)s!i~@?O7YZXcHXMCd4r!I#oF&K2pGAJYN~R z5iyu-YA+%#XZFC8tBe zMfWr-byWCtqa&&4;bP)Y9@`FMzhaKNDe5n(A|}30k2W7VsQsnE{$#WN5rOy0`9BBi zssuOs!iRTWw4lw~u1Lqa<#q_9g0K$K2k8ca7V5;i=UmfiDO&bVPEDug9s(gaV_;h@ z>-SNFOQ`2`kVoFB^4p?3*yNtAUJ={%i=ZNp$<5kUu^^5sKFVia276ano9mLLx5X?` zXC~-V3J7WTzAoMT+OBj;hB)*}_wf$LmM$LX)i1nviAev}N7t4f;b0{r>)`0cg!PkO7%sckObCbd|nvA)mHsLw@??P32o;EI{!cQ}8xN;bP1vDPk`%lnU z-Lbz$`O#I+`ZaG0w&`Ze`PY=Lx`T?#2LeV11HbR{hKgx4xhi;i4jT#fI%Chbx>|EHF>iJLSB8`@_*U zq9-8K7B{_U%{7Bf$^S;Olz3eAu5Z2NsiiTF*m0Oj76F4r*^{e+(ylr8XzzpKMMeky zA61%1gpU>DB${hJ=t=_2MGYH0^vg(8`=`MVZmz@6Mo)m75}z3pxzE6b{C{3HS*9@^9UO^5)LJpN3<@WFB(x%zIdO_}}&mEr}Gg@!S`$noS zW+USVP!3!&DIkfL093)J7Wp|nKFf6@%pdQLbaQ5!Z4@#l}y>>3kYmH`Z(BbIv-#Bod^50QBO!sIxk;RVuq!#V^q# zB057X5S_qXCJUtXI;q{EMg)39I->0_)B>bkU4hcH36qO6kx$z6S6xplb}VogSXTa; zIBT(fXUcC?)5VXK&M7#7!ln8#2VM)*#!#HZP9gi02&_q}O_c=?_=z7ypB8410vcXV(sARB{xRX1%|`Y&kw&_ac$vuqsg{ z^#{*scV!Rp7+G|P?UyCusnMbR)^>uK3;Q0_>`;KNXlfZ8K5i5oD!hEM;k|)*E|!qm zz@WsSs$sIgCS&XS%de4Szxoy~TQP3S@={j0?|-}Fuj<$*EG+Dm%yn}1J%ydZ3%AHV%HFz!*iy5&>+a3Z zhn3a-uwm~kTYAq?*xmKM|A^}Xt$Op@9olm|2DuTr|?Ji=7*@_($f-?)7{ zZNU8w``hyB8j2g`O*e$J+B)w$X-Xf;Y;)!U1wTwxp1^wYFJCTN#dEv1{VAff2d~)g z)`2eP(1N@b!raN?gfqvi=8k}HrWzug*~w?s0K%EYSHm$UMxU33cDerX%y3=CCbj2j zL45oucTO~iMjI2h$Rv_$meFs=rn(Oe)E9IidN6FadW1`)D+zw1U6aN5DH|B49nRfq>sd^B%B#}>v3#cmqJQ~5X?WfR z1QdY+#+qS&S$2MS;VYF;41pL!N1#LJySB8UJ@w1za^+Bzi~PKW^?-0yPt)P{VP(2z z>w&v7g1`My5tM}si{gvV4>{THYq}=PlxsMcY7+Kq8$ys-6Nai|Gu;BZhiM1ZWiU zIj!BqeuJylavJe1$)ryAh&JQ9s#CdmI+2q-XHnYyH(s=WQ#OsNKzYlu*gidlzB?3) z7r2aP+f1W3Q%qE4n$|bVw>kISai6OvQBA+Yv~-m(bq2QuvE|Id1-6$hh#5rwn8EgJ zY~?SzW$=)-N)8fcMkm*~n5}(oX};@!S*M%5)VP^O$uIP%pc8L0kyr)ViEovxnkgfu zxqzO)ruInPu_N-s6{xqJoVMmC$9g=4s+%}8)TH_o`;Glmx>ujgC{lE?IWo{{=|AhC z&&*G7S9q|pbLB{S$6UO`VOq;abjxFE2u=6PN6Irh2c>SbSGE0aTz*Ux z#j71Z>gp@fB6;Gn8=w^D70M$VPdk}7*1ByTcUl(ftqZQf#i6pe3Z;0vc?ZcV8r_aX z57f|UPt3pblFW!saPQJQ67eCp%WjuDpZZM&eU7ewAm4R=5pvV-cw&!cmQ9$Lr@@a( zRjOaF4INjaQbG@s@E|zq(h~>Tte$_V4-3o*d6r73_38P0VnsqVeJM~<3|kmN=qjJ; z%IN*TUKmCQD$sRV=k`wK`&p0@p1D5OEL15^`91U_?M>oxswaZI1bUe1K`p{Q&^heAC ztJAAtHitOYw2siqhjrN*q5Kl*xokN^!g8uy(NY?Obg5p1OPAp|Vw+`MUfv5K>pXn2 zX>SYUO);RvPq-+$Eb`8hW_jLoUbQpJ_~8RRf}8L$^;MUA5 zBQQgK#OoglUAVpV{lH{JdxEzrgsHCiL;Hv4YMdM93OYV@%xDHvC%Kjv7ed`*L^YnT zNY5q6(hJa6D^k!%R3znKSS<3nidN}u%&HSG9BHpCSpCit@Jm_pkB|iF<6w#N&&nfq z6dcQx+hj-`UUcxDKncL{raFJo#)uBMH|3%qeYiwnxjhj>J;>$ApxNm8VlJKSd%C^9 zX;wU(L$!n)K0rRoLr)NxzJ%i$j*?^IBLg}Yv}NDwOEL5z@`?9ZxISw58fYu7I;hao zP{depUa*Wm9F1WShk3MHL>ei|tY#3;T^#c6Y8|i9L$!(MwVjidOttBdkxuf#=bN)p^tE`v&?Bf$?iUc?$<3fv^eZ>4o(T^#m=GzxQ!5UE#oGhckRJqHp1K)E7 zD1;}QM)$=OBjdyDY*AXA19m!X*uo`q&F!^2CclfBmh)=PU}_1F7!P2_z4#-cBiMDYBsx8ETn%0l36b^(Z?hR)bixn9blT{=QL_a`48 zSn#&qn+E+%x{ZN9Isf$CE%zz-cdy#;6rr2L^z7eBv9ofAaL0~G{RX>)0%I53%T+nu zqQU(ZchYytB;^HKdNj#)rUiEGc@K|N_sqSy^{4IVqG90+0M7#-urHM?CrQ=!n}W*E z1ON#NKQ`oMHkkiTEAH#xSeFP6>r%RA9;qbJ(1mS{pMl_!^Hq%bft%04zX~2(Xxv1V zcgsQnrxC7=9%K<>+rYe#ZzJfPJ;hVOs|dB#Ep&2KJTg4li$z%7SM1Z-ht@sBl)W|n zjTf@{=ib0J!Ct%81(yXWW$#~2nq{FoA63*PvN-w_)OP||?zA3xG;@#A?;$WzWoF4S zjSZtOIWn#9VO4UOEg?W=X*GglI+yN9=WoMBIdB$T0g0`TK~QPkLf@Iw>*uI*iTO*0 zxP?+oTNCk&E~VLlzdiW@D>&5F%D@+lj2O?#X)RtO_{HruIGCVc<>MRYxxqN_cn5-8v;$Tz=U7=CPj zKJxdoIz$kPru}{pxNf0d>1>-HO{4vjY%2#bUl4bc2>t6@h_ozq{QfB$$6_IExvB8i z@G@+GwNDP4!$m1@(G7P8E@58iY{nrX0nR*aoBUu?kE2WxdyITDLuB1na+^BH_%YSpF&9@ysj(71+5Y>EimhK;KT3pb^_+pmxq9(a)t#&;K(r?K zkhds52B--(=Z^BHcVytmGZ-9imE1U_3TtJMp*v$-ri@42kj$RDkcWKx`9p}bmGbs$I1gjY7xQBhQf>IeZ6=Utt_VjoAAEB{<1}*Vz=*L*5#>-ujzBLz-NTf(>%d#O+b!`Cs3- z<(3#Bd>QV}k|f-nop=%p1HU%lbVHy;5rZ+*;N-oEufmoFWcw;E_pmJlV-11gl~m-P zBMDiEF@8M6DRSXQaH1qd9P{>F6-Qz{ml8#2t-!thx`qTN3FayP0LhiEx+#*YeApC* z5BcHJhXcconc-d>&Wun@Ii&c(d_i0dXCO{KoU&IMPL&L&696Oqg2X&7(H@vfL~v~t z$Kna>-6`FPOT4X$!^VnMelumr9AQB|9+zmQiUZdAECDOmR2JX6^ASi7g3;)VaE~3z z8bA``_Y%8-nfhC5EGD}ZNhI4#jKm3tp^{;P6K%wVMtNh=BVLK zaQb-WAj{4x;JX@da9qF*Q20}lqwFmJjTWR(9 zo4VYR65MiwJGWncm30E2D!2)d4?Nm%RoqvqbWH*%tUwm3PL}mtSo}xd9(o~Pgz^k} z;6M=$dq6PLInSFqV7X6ZRd{H>ZB?kFFLL|8@hEYm>-!fTH2`O6p1$!fJZc_Ts-xup zqPBhl%5cd0U(^VE?AJ{tsuXO>$-uSh|U0|C0|eE8UNH zv4Uyr#Ex926L~46{vAlB|M;wgcB-cD&hA2Gu^TO)Bx4~An~JiT%nZ>L{Pr)5O-O(J z2q&{fBRGvBysiWP#J9tTsEaWEy9i=COm1FWt0SkQ8Ki)tV79cDzc>DU49uky6(?Bf zL~L{~<>!F=;Xh_Zx30_F@PXEcSJ7WM++04~@IDvi z;7B7!+^LhZT}h`lvtUE@Ki)9Ky~gW16-LD;D9C$5GwHjCUsc!|xcF9H|GN^kNza&a z=6|a0PODEph$z$u4dST3eK)O9t@Cm`7)DP{Wcg;ew|uiXrN9YS9mdzPy{nWk;^Z_k zmp>cdrO}L@c}aD38IFDZZRC9ur%X14^x@2)H-BD#uu(LA`17XX$nDL8&$Nb*T|+MU zyYhZ9*Kiir)VH{J8wj*EtZ>Aap@q1AHWWcd@mYaWwXAGZWfo$yv;V~dniCupr0dT` zx|fk>`_f95^c7_k}79KZcYNOdkEBzul-mImyhy7*l!XeT( zJlnLZXY6c6Y=D#aUF;EF;*$5xe=R}=dAO`{!X@7xo7RWT@C`j57aY9P;6M?$$v zLPRo*TS|A9MLgt)+Y79ON`#h=wc(??FSZ}ewP?0CP*gBS0^gkSUtX}CY9ho>^MJ*C zHU2PmO-#PSuQqLKTrW45Jexjn{g4lZoiidp2%SA5+a(wT`4>y6jZ-^YL2H}U^LG7E z&6QL=vpem3eHS2B5t)!>rRCkprA0VL%!yw8v zFx1;rjA z`rsgbp0`JuI#UAe@%wh@&f50j$3Yxo+Dq4+^%7gZalrY;Kc1PTw;iIaEz?T2@Acl; z_dj?v)Wjoy+X8e3_i^nF1bf(O!u~%dM}!z}68>%7|IM%Z_p5I`ww`Zl5nUb*SslJF zvN1?9c1BKpXElAJXW#9A5npzhU3Gk9Y|)H7Qmk)bIP}5+U^nxAr+V+HA=dFh z#i-!_Q5H>mUIkzFzpi=yw>lN8Dp^Mt9@ryr+0CZk&+-upx=uq?G_#b*-^HoXn81Jh z93BH*kp9P&Q|IhT@==Q*Pj;8aVQ?KNvj<6lJza?K14ueOGA^nd5tb;sgU%0QzFRg>Jh5UbVF$0x2PLa^+p3AX=AvSE92YjeE(vs%&RYvCfj zJ)qQ|XG;)ZO^rk^fmxjz*gKe9aeRM5I(t)8gJYHI=!go}K#`%y!-uN`Vf55&WI zyxOc`I8i60($>SkLHE?E%!i9-XM>DK-gH)FQyw0Y5pn$;$(vk5{`H=#x9l$;Kr{yr zT}$brKhPLR_>;k;C3kye@y(+M-lAzI;jQgmyFtsO@Qv2V0nck$a}iq3OqKkwjKDU& zlHp%VkjbeX@$cZwzi{zDV`Y2oaQJ$}I!F13oe@9tlAY~Yh0xW8^l_o$#kh7a?Mm5U zI78cE_LtoDxr2yKp|k55JK^1#)_5ypC6nrlvx51Sv!UsNnuHUHK2x!o^sKW3m_Z%u zBpu7q_lM(UBbsD$c|eVEjGH`tEYmoo<>vLfrFq~i?&T4QT2p&xRsKcwp z>zV8BWY{xz7NAxocD|11!LBqn_aawGdR$r`%IQA1oJQ8T3?lPk)8EF|R2qh)XKoESzXs6_+bwYRAPhpI_$(g8!vGd`I>I-M@EXevUOGji(cb4WoVW9Il7~6<6G8=NF zDa(cX>9kHU)Jdr-7O&TE==%>33g*BLWiE%wc2J44@B+vgCTW%mboA*GRwcIRYrPRZ zcx_<&jtK5+ZmcET*HgM4QRQva-tn&3*Mks)noDW^*)U}^{H9}`KOq=`k`#2tY@SUL z)^$BHf@*mgb0M%Ha<<*x_a@FIV(%pc%8Kvq%<}PU&2zj-k>gc#6xwN$!0;i_=HrnL zNx5$c+=U!o!2XQ}W0B6z$;(E`ASY?B8*J#L%8}RAc$WI0C@we5Yhg~iVW>MKmQxdc z9lr$L@Mjz8)+nB3^!A!ciryv|Qm4r`=qoSGsUSV4I_G3sWez;$L}cP)jC_a5?N)ZL{I`&=IzSEQ~~ zcyE%NaN-kv&9j*PxKVNGSG!}jO298FwD@~G)_e5Zb>m;4AkppznO`Qv*TbC=4hPB4*?_HAat zy7_5xCIaY^J}_c*P_>sAPcc_^bm$g3>r;ycvokg@vTKr8-G}!Mqlq=&d$Au#OrTE_ zv``A;$sr8pyBo0aRL<9Z44@KopDrB=1=m9603 z00B{m*?-g3O4RnTy|lL?npSo?Ha6FTFbmoSAleU2IHHJOI6MoG4IdJe7T77ZlOHzZ6rzL;SP zneg_ycz}yBO=Wt3?g@QPGZOc&oqP%tzjobz>mBt&sMG^fi}`I@9kdb6DP4G4`$*Ttp_3V?imjFyd1+k*%`emYpe!3^Gf=ZQz;W8_hWu1 z?)dAXyU3LHD*xJ6)x4ASqKQ*08Fibzv=geIx_(?$ZtX^#u2;!m>(Sc{U9IjHo7=q~ zpHXKy#n`-2KM=x@o*2)0`lo-S1f`6RxnX6nQ@T6kRXTOz@cFg2USx_;sOF2+MrWQ#x?>LDp+LPF`{a~0btyWz zB7NiIBi8rv*BwAiws_IG0O`6Qr`J?1HVNe~&-N~Vyt?In!3Wi;Qu2e;;B~lDc+Xlb zigdCqr3nDQgoBHzEoYULQ+3(j=JnPM1*kBjX{MF3Nvoi?-lM)N`Z-=YCurk14 zM$f_j$~bPWgU+^@WiDR4_~?h6=*+E7WFK=zTh^qgn({fBQmSU7z*R;{;)%FB=8Q>Y zpF11Fb670zEGwmb_gB#w`grw}kX80$kH^MB&MOsl?bs}%mU5#~(%qygzS)Rl!*e!L z+Y$kf%_^ctG~m4MYCa9kS(r?k2u95zy|DyN5GU4+7lvX3^k)MVzKRH4{nrVH(>hZ*6bPv1Knx+#f1b=pEuCguVRlk^=rPkar%N{J!ZU9Nj;K3 z;S7zT+~n;Y{h=JD#=Bgz5p5L-eLU8cwB!22!-8n_Pia@~ymOeZ@DHgog4yoxQZ^7m z)yJ0fXI)Y}p;PPOthQbMnX4#nDVca6a!6OUUcxY?!dz#2aHcRhi0Cm@Tgq+UwJ2;7Ur(#?OmzcGy$4h5%ygD% zR#kq_SeKv5lfFI~6;~2*yCxHmp$r2pl}M*t#=eY_-qwp-r|p`!oJxB&trIDDEUaQF zUJR%-+<8l4_nvn|&FImL$0Jl@O>%8;-A$5#*;}Q0GGZYSpjXFAdB`wvrWq zAbgQ_1+BK(i2`;Vv)Yp)ut8Ek9^BWqG{$$#;!$&HL86Uol6R=rw2G|k;x0~RwjqGFzIExyJ5x}V;Rf*&aLNnz3=tD*YE%LkN5s(uG#MUyPtDD=bX<`R$Mqb zic7n$=Zxo9V3ln9Y5ADXg1I82^(W3UT#Ps)ZmIoTDVigz6q^#mMK~^|QB!fVt-9NE z!Bdd1c!T86l?!X>OTw5^L{Z;#Plty`);wqLNACyi=IAJ?O}p;m9VZv!56#6lAL+LU z|GQZv>hvWAN(-$rptWvKQhKb$bX4gX_X^U4&t2L9*sG}4SS9GX3dOwaiGvuSdpO3I zgj?Z%lMP#3bAPZF30<_k(@D^h2@UsLf{@4tdQwCW{f8-xKTW}K2z#bR)&JzOc4mP6 z{&&qe*O)2CIWlG>oZNIVi8S^!;ozuOHBg;t{kqT^rq!lkEJn+?1y}ipZ?R zC&haCE4>1BXIjOZQE4M`)SV0gGPqe^8q*es=Z>_MpLfWHiVkl}xpl(8V8ANne{JeN zmAzr1eNN%e&Rsv2a(7(T8Wofvf!RSi!`S(XNiK$OEmC$FDqvST*dC`AiNK-f1t!$N zxg_$e(XDV?bGE?!I3x82UKUIz3tOCwpsCbo!3)i?`boo_#_7&WqX}}1BXh&A*3`PU z6JYlqC~uR1@BAhPM)|!<_57(fX3z|zl>~+xS={TV*gXe&W~4Ot1Ft_PaQp626;iCe=-HS@gm$A z{5Z7+Z@j2nno53s=hyrknpoAs&`Fr=o}$&zMm2wIUf4KmxiL1ZTYY0=q-SWXxzSwE zuHjbn~a4Hi;ETaMYA(hWqQdAXO_7^-|kWID0_V_r!Ajv!hPTS zWP(@B_9uhBt3(_BCGCKeZHU?3OSppF^W(VUFv=vphcaBie$#}0_F-;9PfY8vmLjYE z_S(~C!*xkEi8YtZ@QS?v#&n!U&J1|qdJe*~w@|xd5%JTgOHN4ZcyS#trp^{n zMa$iqb}UR(IMzOsI+%GW7gOc0;+KwMGUWaA`eUCO`LM4`dbm@hMVpRFW@j`Z zmdjAguy7A3q}W>JU0r}j(f$n@;hx6xyO8!HOQqdKHs9F>fo{%wL55NKpGw;For9n+ z>bp0O$jKHCvAzvx9;Q6SfbDC>Yb?A!c`r5N1vrvBuBDIBz=r zNK85NT!_ems>TzAjF26OJpN;@63uQUJV95ZIgRuYleCpD<(nygDvDQ|n~>hBamo(R zJMOXbDqsNW`UYeK+WRa@74i?v15yD)1U9uw%kunJI&5{q<(xL9DBH|DvEBUR4jI{q*}Y1qSKerby-abh9P;dI!=NN zxMzrs;H*`s{bHX1|2$gko*K97lLvJSYxc7=tDJAN0;V+h<-k$9xr~H>&3&f~BJ(SB z6(#vk#J9c5f=~%G^H_N&ohdAo9M`Q)cQn1%#wD{)gyjfeu0DX7d_O=!?G9AKX+bU) z8=a07Irw^E%)ap1y+D-M#q0drqvDe+&ICZBUXRE_4@d5@sJEW4;qaTw>seI)T)L&V zd(%%uQrSF zDdvft6rhidM3XU>fIJ0$@_}Ar>^A-fj3akPoHb2A6RZ=553VH&>$X+G?{}3;UIKI#u((7Gi=mqU~jV&&hm!(fO#` z3{M!u5{d+{;%>u3!@@IRy!PC?qJtkbFms0%cskXETm%O(`nUi!cdl2G96g%;b!=nx zS^EkG_AtYuf=lsD#LVQrhlwN=&vz%fZ2yYEw7rr)``$8FetHz_2l&NM6p)=O=D0hJ z&Cd^g2+QWDU2zhh0Y3wL7pOViGivL0IR^J3&YTfg1khW0)kP?UWAoFbOTIO1*MC=J zJjMBYuy(#Vx$Onq*&Uu`De|1hX%>|f=%4<_sQ!ny2uh$UqfLv#w_E)_jo^@jYd1Ll z;N{F&y4+F{5DVw4zzEm#|16T17~V-U%4L~8~Uc;k2soqSQjMCiRAP}um|;H)9(AQOJJ7tHUEF%@{Z}cL($Dn#yDi1gZ-J(H~uGQ)W}WC+-r-xNyH; z!ri6!)9WsJU`}w?(TCbDDjo|N$V2u@N8LQbQ@&U_GF9!rpm$qmilzu@F zVHET!4``bOF`a#^k{wTT*lgd_R{S-YyAGa}*P9A+ezufo^~I45R7*#pd(1b46J1@Y zii4(AUSLrHZ9;y-6*XS3BdvW#44_?KPp$SNhtrU|986We%(dS z_KjW1cU14Qdbt$_Gh?dlYA;ckLLIz8BvZ2)sZg&H*gMWMee61)TzRJXJO-8<{-SA0_~IX5}~r8 zH#?4TF$^Wf4!>~Modp}5p?ipzU`1{2;EU3$wspK7W!=*v)8%$LWgTb zsQ~EfK-5=X48#LijUeiB5HRPN82nZ5wxj&&9O+dX3R(WvHsB#6pp#_FF--!@KN~Q@ z@&cm`yb3bR?KbG`j|RYKm07zKvD+|vvav-J)%E+vmQFdpe8wgv z?Ud~&q*ZFr{T8U0t*QjGX}!Pfq&A5dnSX@K7DRpZ$G|N>MG*u@u&sjF4V?Q|%B1=L zq(0NQG9WjnQ524%wghVxzU?;jhvDyE56e8#R@2032k2@$iLNMa+Y!Xo4sH<`5E&3! zETebj`xOg|BQeLNJ0^;Am#pcj+nasNdULNg`*2VuFRr2Yxgq8AH=E8Y-R1$5bb_18 zz;;iRp(lFc(T7K;pan*w(~0(TeN!=bjO(YeEIh`qIbzlDk$ZzyB2F5Aq1+A@PEpae zN6X7-YmxH9<|_V@IkE!?8xUko$9x)E0+^-yjM+x*p3#iLc3qo+mvJK_%v?vXN{-#t z%rsL6ksm)E=nt{bf9$qAe-tRUU_v6Hw$#%ok74uaUJ3V+zvc!@WQ^W9)vpL$HLen+ z#sO(+cHzo2%m%@0zx5CR6hIFGUFZ8vRJ@}6zK`=Z(ab(mbH)*q_};61Hzvnp8he1Q ziTu?l7yKRy48+OqGQQSt;ukeCWCfc>FGw;{UfjWlF!OV~$g>)G8~4cMaA@Vh=KQi3 z_*bqq7Q>D)YF>ZPLMXoGyfuAqUgoj}q%B<7o5h&ybHyPKh>w>Vn)fQ^EENWQ_&tc7 z=bSJB4%JehT5FbP*-c+RY*}%Z#PhKq?oU#)FFUr=q}5@DZpTdAiz29z$!2m zZBux&^v;4u6dz~YM-I$b6gSTC3h(Zaj;kI?S%p`4kA@Vy^sY*uvR7ZAnh=~jS#0z@ zYM@AfmPdm+OHlkDhJ@nrRb@as`_FBXG7>Gjqy}k1Avv08kWNRz+Ij0ukG|r|M+OLQ zB6yVFnopt79g?cNtbA-eqg#3F8Q|IYc*(FW19^X|M&7m4ERvkJRr}RM3~AB@{lX7U zIC9SvIkUzBIB0J@=t(GEdpuxanQtyW--t=BUZe6A@5cC$cT9qbjs#L=JydvA%2tL` zOa!KVFH4v7GOG3g%Nx#!^rozQBR+iZv2Vv2C2=-~o|Q^SVyg!8dSdF1t^22~hlonU zEE?wBFkJ!=OIi_U=8$)WYgX@Pf%>YhA<_<2V-bY10Q#}T@l6~BaBu};$h%;1b(5w2 zDH;de8j>{*VGFG5qF3!)WwG7PTiqhwT1x3bWrJ}YT;#|MnRaiDtEFd8CLa~8C>#^a zZj6agBHixUw=c)D#^;7Z9%#*z;YXU7Fx2~q4|>w1Dz_$kUk-a4yMN$r_}`0#)%MPz zHc?kw_73nbKkZ`~Jf&1r+!Z27kF9JGx!JT*@A;J?W1YRVwZx`yZPCJJ&{<>puKYRu zTlxNDFga-XIGN;+doeyI19?$Nlobh~A7UEcFCT8fY4k{mp1wmd=3hR9i3`d}$>(F` zB99BSI^S&^)--2mr~lpemikbzBo05C(qg0ya9aH+b_-X^kCYfU4J8g9FX;r5Z%PAz zgqpOSsf_6PV5go`VM7sEF?uvzas6tMP(0@`yC8e2!Lef3!BGoIFtAmerIB5z9Md@s zrS{arGGh<)J8gS~2*KAwTb3a51CTLci2g+ajo1DG9iMbE^ZEC|vAWd>lDSUW2ldGG zB8Q+bC89Fm(0ep<_xhS8Isdiv_(i4n=E(;4poAZYpR7`jz=!Cz&q_egEnUjUnvA0y zzKzpGPq}mG-D3;-nL(V1^I4>RyG5!fLKWmmyW((AXMk2GV9_B0<+cw{5H#u!(LoJY zHcPUCQ(TjbuWxQprPCh?D7r`1+Ld|;?LF!4`UGR$Jk^FAoQI+z1EqeCdW7v6j<{zr z3Ui=6@{xx!gHq`U>~5n zmeB-FG>+ZM8Ojzil`+m%%tWQN-KcfVNi=pHlqlK*{(ZmU(MXW!I-=;4rB)z_02rfG zm@4OtU7AsODd3jV&Z?`&Lo|ZO8kACAYe%Zbc4y~M=`Xp~FE`h)p5j%<8LzjiE8`KZ z0N;l5*m96y);WXT(r*}`ZGAoC&8*8Fm@A3=nc-CeP{B4DB8g-{3;~xN3*T8|mr(Xt69Latv3+H8u3bHXm(;XaWGOi2RC)B-)aIX=+SyNJbm{L8l%U#D zRuZWK)t1)2oFJ8(0ihM@?o07N4A6OPk3>CGVa=oSIM5KKc{c9bAJi^3f+a=8&v1== zz=5|H;-g%iG0sB`TWACYmOaY8B_B}2nHO3C0}q1Br&jW(y7*5|^0l_;3kv+_8njch zAUKD%fzzhwF@50LZdd$n46JH20>XL=OK$a+IUs6sp}yGangp2A|)e8Q+_3&Tj^Q(dY literal 0 HcmV?d00001 diff --git a/docs/source/getting_started/SettingsEditor.rst b/docs/source/getting_started/SettingsEditor.rst index 1cec86003..101784af2 100644 --- a/docs/source/getting_started/SettingsEditor.rst +++ b/docs/source/getting_started/SettingsEditor.rst @@ -865,6 +865,11 @@ show here. Autoload Query Providers ~~~~~~~~~~~~~~~~~~~~~~~~ +.. warning:: This feature is being deprecated. Please avoid using + this. If you have a use case that requires this, please contact + the MSTICPy team - msticpy@microsoft.com. We'd love to hear + how you are using it and how we might be able to improve it. + This section controls which, if any, query providers you want to load automatically when you run ``nbinit.init_notebook``. @@ -919,6 +924,11 @@ There are two options for each of these: Autoload Component ~~~~~~~~~~~~~~~~~~ +.. warning:: This feature is being deprecated. Please avoid using + this. If you have a use case that requires this, please contact + the MSTICPy team - msticpy@microsoft.com. We'd love to hear + how you are using it and how we might be able to improve it. + This section controls which, if other components you want to load automatically when you run ``nbinit.init_notebook()``. diff --git a/docs/source/getting_started/msticpyconfig.rst b/docs/source/getting_started/msticpyconfig.rst index 5481f04dd..4244628d3 100644 --- a/docs/source/getting_started/msticpyconfig.rst +++ b/docs/source/getting_started/msticpyconfig.rst @@ -4,21 +4,43 @@ Some elements of *MSTICPy* require configuration parameters. An example is the Threat Intelligence providers. Values for these -and other parameters can be set in the `msticpyconfig.yaml` file. +and other parameters can be set in the *msticpyconfig.yaml* file. +This file uses a YAML format. -The package has a default configuration file, which is stored in the -package directory. You should not need to edit this file directly. -Instead you can create a custom file with your own parameters - these -settings will combine with or override the settings in the default file. - -By default, the custom `msticpyconfig.yaml` is read from the current -directory. You can specify an explicit location using an environment -variable ``MSTICPYCONFIG``. +.. warning:: MSTICPy has a *msticpyconfig.yaml* included as part of + the package itself. This is used to supply some default settings + and *should not be modified*. Any settings that you include in your + own *msticpyconfig.yaml* will override the default settings. You should also read the :doc:`MSTICPy Settings Editor ` document to see how to configure settings using and interactive User Interface from a Jupyter notebook. +.. important:: The settings in *msticpyconfig.yaml* are *case sensitive*. If you + specify a setting in the config file with a different case to the + setting name in the code, the setting will not be found. + + +How MSTICPy finds the config file +--------------------------------- + +MSTICPy uses the following logic to discover a configuration file. + +1. If the environment variable ``MSTICPYCONFIG`` is set, MSTICPy will + use the value of this variable as the path to the config file. +2. If the environment variable ``MSTICPYCONFIG`` is not set, MSTICPy + will look for a file named ``msticpyconfig.yaml`` in the current + directory. +3. If neither of these is successful, MSTICPy + will look for a file named ``msticpyconfig.yaml`` in a folder named + ".mstipcy" in the user's home + directory. This is typically ``~/.msticpy`` on Linux and Mac and + ``C:\Users\username\.msticpy`` on Windows. + +If you are using the :py:func:`init_notebook ` +function to initialize MSTICPy, you can specify the path to a config file +as the ``config`` parameter to this function. This will override the logic above. + Configuration sections ---------------------- @@ -28,6 +50,10 @@ Here you can specify your default workspace IDs and tenant IDs and add additiona workspaces if needed. If you wish to use the Microsoft Sentinel API features you can also specify Subscription Ids, Subscription names and Workspace names here. +.. note:: Although Microsoft Sentinel is no longer called "AzureSentinel" + the configuration section is still called AzureSentinel to ensure + compatibility with existing code. + Sample entry for MS Sentinel workspaces .. code:: yaml @@ -214,6 +240,12 @@ For more details on Azure authentication/credential types see User Defaults ~~~~~~~~~~~~~ + +.. warning:: This feature is being deprecated. Please avoid using + this. If you have a use case that requires this, please contact + the MSTICPy team - msticpy@microsoft.com. We'd love to hear + how you are using it and how we might be able to improve it. + This section controls loading of default providers when using the package in a notebook. The settings here are loaded by the :py:func:`init_notebook ` @@ -221,6 +253,72 @@ function. See the `User Defaults Section`_ below. +MSTICPy Global Settings +~~~~~~~~~~~~~~~~~~~~~~~ + +There are miscellaneous settings that control the behavior of +MSTICPy and some of the underlying libraries used by MSTICPy. + +.. code:: yaml + + msticpy: + FriendlyExceptions: True + QueryDefinitions: + - ./queries + - ~/.msticpy/queries + http_timeout: 30 + Proxies: + https: + Url: https://proxy:8080 + UserName: user + Password: + EnvironmentVar: "PROXY_PASSWORD" + + +**FriendlyExceptions** controls whether MSTICPy will catch and re-raise +exceptions with a more user-friendly message. This is mainly +applicable to use in Jupyter notebooks. Most of the MSTICPy exceptions +are displayable as an HTML message with details such as links +to relevant documentation and drop-down details of the exception. + +If this is set to ``False`` then the exception will be raised +as a standard Python exception with details printed as simple text. + +**QueryDefinitions** is a list of paths to folders containing query +definition files. You can add any number of these and MSTICPy will +search these for queries to add to QueryProviders. + +**http_timeout** controls the default timeout for the httpx library +used by MSTICPy. This can be simple integer or float that sets the global +timeout values for connections. You can also specify this as +a dictionary or tuple of individual timeout components: + + - ConnectTimeout: Float + - ReadTimeout + - WriteTimeout + - PoolTimeout + +You can also specify as a Tuple or of (default_timeout, connect_timeout). +For more details on httpx timeouts see the +`HTTPX documentation `__. + +**Proxies** is a dictionary of proxy settings. You can specify +different proxies for different protocols (although only the https +one is currently used in MSTICPy). We are gradually rolling out +support for this setting to components in MSTICPy - not all yet +support this setting. +If your proxy requires authentication you can specify a username +and password. These are combined into a single URL of the form +``https://username:password@proxy:port``. + +.. note:: If you are using a proxy that requires authentication + you should consider using the ``KeyVault`` setting to store + the password (and optionally the username) rather than specifying + it in the config file. You can also use the ``EnvironmentVar`` + to reference an environment variable that contains the password + (and username). + + Specifying secrets as Environment Variables ------------------------------------------- @@ -361,7 +459,7 @@ You can find it in the ``tools`` folder. Running ``config2kv.py --help`` shows the usage of this utility. The simplest way to use this tool is to populate your existing -secrets as strings in your ``msticpyconfig.yaml``. (as shown in +secrets as strings in your *msticpyconfig.yaml*. (as shown in some of the provider settings in the example at the end of this page). @@ -387,7 +485,7 @@ path of the secret (as described above). ``ApiID`` and ``AuthKey`` values will be used. The tool will then write the -secret values to the vault. Finally a replacement ``msticpyconfig.yaml`` +secret values to the vault. Finally a replacement *msticpyconfig.yaml* is written to the location specified in the ``--path`` argument. You can then delete or securely store your old configuration file and replace it with the one output by ``config2kv``. @@ -414,7 +512,7 @@ information. The advantage of using *keyring* is that you do not need to re-authenticate to Key Vault for each notebook that you use in each session. If you -have ``UseKeyring: true`` in your ``msticpyconfig.yaml`` file, the +have ``UseKeyring: true`` in your *msticpyconfig.yaml* file, the first time that you access a Key Vault secret the secret value is stored as a keyring password with the same name as the Key Vault secret. @@ -451,6 +549,11 @@ The documentation for these is available here: User Defaults Section --------------------- +.. warning:: This feature is being deprecated. Please avoid using + this. If you have a use case that requires this, please contact + the MSTICPy team - msticpy@microsoft.com. We'd love to hear + how you are using it and how we might be able to improve it. + This section specifies the query and other providers that you want to load by default. It is triggered from the :py:func:`init_notebook` @@ -668,20 +771,22 @@ back into the notebook namespace execute the following: any of the items in the ``current_providers`` dictionary. -Extending msticpyconfig.yaml ----------------------------- +Using msticpyconfig.yaml in code +-------------------------------- + +Settings are read using ``get_config`` method of the +:py:mod:`refresh_config` module. +You can specify a setting path in dotted notation. For example: + +.. code:: python3 -You can also extend msticpyconfig to include additional sections to -support other authentication and configuration options such as MDATP -API connections. Refer to documentation on these features for required -structures. + from msticpy.common.settings import get_config + qry_prov = get_config("QueryProviders.MicrosoftGraph") -Settings are read by the -:py:mod:`refresh_config` module. -Combined settings are available as the ``settings`` attribute of this -module. Default settings and custom settings (the settings that you -specify in your own msticpyconfig.yaml) also available separately in -the ``default_settings`` and ``custom_settngs`` attributes, respectively. +You can also set configuration settings using the ``set_config`` method. +This also supports dotted notation for the setting path. Any +changes made here are not persisted to the configuration file and +only available for the current Python session. To force settings to be re-read after the package has been imported, call :py:func:`refresh_config`. @@ -696,6 +801,9 @@ reflect the underlying YAML data in the configuration file. the actual configuration value with the secret stored in the environment variables. +If you are building a component for MSTICPy or work alongside MSTICPy, +you can extend msticpyconfig to include additional sections to support +your component. Commented configuration file sample ----------------------------------- @@ -703,6 +811,19 @@ Commented configuration file sample .. code:: yaml + # msticpy global settings + msticpy: + FriendlyExceptions: True + QueryDefinitions: + - ./queries + - ~/.msticpy/queries + http_timeout: 30 + Proxies: + https: + Url: https://proxy:8080 + UserName: user + Password: + EnvironmentVar: "PROXY_PASSWORD" AzureSentinel: Workspaces: # Workspace used if you don't explicitly name a workspace when creating WorkspaceConfig @@ -721,28 +842,8 @@ Commented configuration file sample Workspace3: WorkspaceId: "17e64332-19c9-472e-afd7-3629f299300c" TenantId: "4ea41beb-4546-4fba-890b-55553ce6003a" - UserDefaults: - # List of query providers to load - QueryProviders: - - AzureSentinel: - - Default: asi - - CyberSoc: - alias: soc - connect: false - - Splunk: - connect: false - - LocalData: local - # List of other providers/components to load - LoadComponents: - - TILookup - - GeoIpLookup: GeoLiteLookup - - Notebooklets: - query_provider: - AzureSentinel: CyberSoc - - Pivot - - AzureData: - auth_methods=['cli','interactive'] - - AzureSentinelAPI + KustoClusters: + QueryDefinitions: # Add paths to folders containing custom query definitions here Custom: @@ -818,6 +919,29 @@ Commented configuration file sample clientId: "69d28fd7-42a5-48bc-a619-af56397b1111" tenantId: "69d28fd7-42a5-48bc-a619-af56397b2222" clientSecret: "69d28fd7-42a5-48bc-a619-af56397b3333" + # DEPRECATED + UserDefaults: + # DEPRECATED - List of query providers to load + QueryProviders: + - AzureSentinel: + - Default: asi + - CyberSoc: + alias: soc + connect: false + - Splunk: + connect: false + - LocalData: local + # DEPRECATED - List of other providers/components to load + LoadComponents: + - TILookup + - GeoIpLookup: GeoLiteLookup + - Notebooklets: + query_provider: + AzureSentinel: CyberSoc + - Pivot + - AzureData: + auth_methods=['cli','interactive'] + - AzureSentinelAPI See also diff --git a/docs/source/visualization/NetworkGraph.rst b/docs/source/visualization/NetworkGraph.rst index 3f44a5a93..77e6d6275 100644 --- a/docs/source/visualization/NetworkGraph.rst +++ b/docs/source/visualization/NetworkGraph.rst @@ -306,4 +306,4 @@ References - `Networkx from_pandas_edgelist `__ - `Bokeh graph - visualization `__ + visualization `__ diff --git a/msticpy/__init__.py b/msticpy/__init__.py index 030dd1829..03affeb6c 100644 --- a/msticpy/__init__.py +++ b/msticpy/__init__.py @@ -131,6 +131,7 @@ __author__ = "Ian Hellen, Pete Bryan, Ashwin Patil" refresh_config = settings.refresh_config +get_config = settings.get_config setup_logging() if not os.environ.get("KQLMAGIC_EXTRAS_REQUIRES"): diff --git a/msticpy/analysis/polling_detection.py b/msticpy/analysis/polling_detection.py index b94a6ce37..f1803a2ac 100644 --- a/msticpy/analysis/polling_detection.py +++ b/msticpy/analysis/polling_detection.py @@ -14,12 +14,11 @@ the class PeriodogramPollingDetector. """ from collections import Counter -from typing import Optional, Tuple, Union, List +from typing import List, Optional, Tuple, Union import numpy as np import numpy.typing as npt import pandas as pd - from scipy import signal, special from ..common.utility import export diff --git a/msticpy/analysis/timeseries.py b/msticpy/analysis/timeseries.py index e12dff321..4c3eacd51 100644 --- a/msticpy/analysis/timeseries.py +++ b/msticpy/analysis/timeseries.py @@ -268,12 +268,12 @@ def ts_anomalies_stl(data: pd.DataFrame, **kwargs) -> pd.DataFrame: if time_column: data = data.set_index(time_column) - if data_column: - data = data[[data_column]] + data_column = data_column or data.columns[0] + data = data[[data_column]] # STL method does Season-Trend decomposition using LOESS. # Accepts timeseries dataframe - stl = STL(data, seasonal=seasonal, period=period) + stl = STL(data[data_column].values, seasonal=seasonal, period=period) # Fitting the data - Estimate season, trend and residuals components. res = stl.fit() result = data.copy() diff --git a/msticpy/auth/azure_auth.py b/msticpy/auth/azure_auth.py index a3d0b4d2a..3d7ed0268 100644 --- a/msticpy/auth/azure_auth.py +++ b/msticpy/auth/azure_auth.py @@ -175,3 +175,9 @@ def fallback_devicecode_creds( raise CloudError("Could not obtain credentials.") return AzCredentials(legacy_creds, creds) + + +def get_default_resource_name(resource_uri: str) -> str: + """Get a default resource name for a resource URI.""" + separator = "" if resource_uri.strip().endswith("/") else "/" + return f"{resource_uri}{separator}.default" diff --git a/msticpy/auth/azure_auth_core.py b/msticpy/auth/azure_auth_core.py index 7428a97ed..f969605a5 100644 --- a/msticpy/auth/azure_auth_core.py +++ b/msticpy/auth/azure_auth_core.py @@ -437,7 +437,7 @@ def check_cli_credentials() -> Tuple[AzureCliStatus, Optional[str]]: except ImportError: # Azure CLI not installed return AzureCliStatus.CLI_NOT_INSTALLED, None - except Exception as ex: # pylint: disable=broad-except, broad-exception-caught + except Exception as ex: # pylint: disable=broad-except if "AADSTS70043: The refresh token has expired" in str(ex): message = ( "Azure CLI was detected but the token has expired. " diff --git a/msticpy/auth/keyring_client.py b/msticpy/auth/keyring_client.py index a7a9f7b91..6f280d2f2 100644 --- a/msticpy/auth/keyring_client.py +++ b/msticpy/auth/keyring_client.py @@ -4,11 +4,10 @@ # license information. # -------------------------------------------------------------------------- """Settings provider for secrets.""" -import random from typing import Any, Set import keyring -from keyring.errors import KeyringError, KeyringLocked, NoKeyringError +from keyring.errors import KeyringError, KeyringLocked from .._version import VERSION from ..common.utility import export @@ -117,16 +116,4 @@ def is_keyring_available() -> bool: True if Keyring has a usable backend, False if not. """ - char_list = list("abcdefghijklm1234567890") - random.shuffle(char_list) - test_value = "".join(char_list) - try: - keyring.set_password("test", test_value, test_value) - # If no exception clear the test key - try: - keyring.delete_password("test", test_value) - except keyring.errors.PasswordDeleteError: - pass - return True - except NoKeyringError: - return False + return keyring.get_keyring() != keyring.backends.fail.Keyring diff --git a/msticpy/common/pkg_config.py b/msticpy/common/pkg_config.py index cebea72f2..b4f2d9a93 100644 --- a/msticpy/common/pkg_config.py +++ b/msticpy/common/pkg_config.py @@ -285,9 +285,7 @@ def _get_default_config(): title=f"Package {_CONFIG_FILE} missing.", ) from mod_err conf_file = next(iter(pkg_root.glob(f"**/{_CONFIG_FILE}"))) - if conf_file: - return _read_config_file(conf_file) - return {} + return _read_config_file(conf_file) if conf_file else {} def _get_custom_config(): @@ -345,12 +343,19 @@ def _translate_legacy_settings( return mp_config +############################# +# Specialized settings + + def get_http_timeout( **kwargs, ) -> httpx.Timeout: """Return timeout from settings or overridden in `kwargs`.""" + config_timeout = get_config( + "msticpy.http_timeout", get_config("http_timeout", None) + ) timeout_params = kwargs.get( - "timeout", kwargs.get("def_timeout", get_config("http_timeout", None)) # type: ignore + "timeout", kwargs.get("def_timeout", config_timeout) # type: ignore ) # type: ignore if isinstance(timeout_params, dict): timeout_params = { @@ -377,6 +382,9 @@ def _valid_timeout(timeout_val) -> Union[float, None]: return None +# End specialized settings +############################ + # read initial config when first imported. refresh_config() @@ -396,7 +404,7 @@ def validate_config(mp_config: Dict[str, Any] = None, config_file: str = None): """ if config_file: mp_config = _read_config_file(config_file) - if not (mp_config or config_file): + if not mp_config and not config_file: mp_config = _settings if not isinstance(mp_config, dict): @@ -426,9 +434,7 @@ def validate_config(mp_config: Dict[str, Any] = None, config_file: str = None): mp_warn.extend(prov_warn) _print_validation_report(mp_errors, mp_warn) - if mp_errors or mp_warn: - return mp_errors, mp_warn - return [], [] + return (mp_errors, mp_warn) if mp_errors or mp_warn else ([], []) def _print_validation_report(mp_errors, mp_warn): diff --git a/msticpy/common/provider_settings.py b/msticpy/common/provider_settings.py index 36090cf0d..4b50675cf 100644 --- a/msticpy/common/provider_settings.py +++ b/msticpy/common/provider_settings.py @@ -150,7 +150,7 @@ def get_provider_settings(config_section="TIProviders") -> Dict[str, ProviderSet name=provider, description=item_settings.get("Description"), args=_get_setting_args( - config_section=config_section, + config_path=config_section, provider_name=provider, prov_args=prov_args, ), @@ -230,8 +230,15 @@ def auth_secrets_client( _SET_SECRETS_CLIENT(secrets_client=secrets_client) +def get_protected_setting(config_path, setting_name) -> Any: + """Return a potentially protected setting value.""" + config_settings = get_config(config_path) + prov_args = _get_protected_settings(config_path, config_settings) + return prov_args.get(setting_name) + + def _get_setting_args( - config_section: str, provider_name: str, prov_args: Optional[Dict[str, Any]] + config_path: str, provider_name: str, prov_args: Optional[Dict[str, Any]] ) -> ProviderArgs: """Extract the provider args from the settings.""" if not prov_args: @@ -241,18 +248,16 @@ def _get_setting_args( "tenantid": "tenant_id", "subscriptionid": "subscription_id", } - return _get_settings( - config_section=config_section, - provider_name=provider_name, - conf_group=prov_args, + return _get_protected_settings( + setting_path=f"{config_path}.{provider_name}.Args", + section_settings=prov_args, name_map=name_map, ) -def _get_settings( - config_section: str, - provider_name: str, - conf_group: Optional[Dict[str, Any]], +def _get_protected_settings( + setting_path: str, + section_settings: Optional[Dict[str, Any]], name_map: Optional[Dict[str, str]] = None, ) -> ProviderArgs: """ @@ -260,12 +265,10 @@ def _get_settings( Parameters ---------- - config_section : str - Configuration section - provider_name: str - The name of the provider section - conf_group : Optional[Dict[str, Any]] - The configuration dictionary + setting_path : str + Dotted path to the setting + section_settings : Optional[Dict[str, Any]] + The configuration settings for this path. name_map : Optional[Dict[str, str]], optional Optional mapping to re-write setting names, by default None @@ -275,48 +278,66 @@ def _get_settings( ProviderArgs Dictionary of resolved settings - Raises - ------ - NotImplementedError - Keyvault storage is not yet implemented - """ - if not conf_group: + if not section_settings: return ProviderArgs() - setting_dict: ProviderArgs = ProviderArgs(conf_group.copy()) + setting_dict: ProviderArgs = ProviderArgs(section_settings.copy()) - for arg_name, arg_value in conf_group.items(): + for arg_name, arg_value in section_settings.items(): target_name = arg_name if name_map: target_name = name_map.get(target_name.casefold(), target_name) - if isinstance(arg_value, str): - setting_dict[target_name] = arg_value - elif isinstance(arg_value, dict): - try: - setting_dict[target_name] = _fetch_setting( - config_section, provider_name, arg_name, arg_value - ) # type: ignore - except NotImplementedError: - warnings.warn( - f"Setting type for setting {arg_value} not yet implemented. " - ) + try: + setting_dict[target_name] = _fetch_secret_setting( + f"{setting_path}.{arg_name}", arg_value + ) + except NotImplementedError: + warnings.warn(f"Setting type for setting {arg_value} not yet implemented. ") return setting_dict -def _fetch_setting( - config_section: str, - provider_name: str, - arg_name: str, - config_setting: Dict[str, Any], +def _fetch_secret_setting( + setting_path: str, + config_setting: Union[str, Dict[str, Any]], ) -> Union[Optional[str], Callable[[], Any]]: - """Return required value for indirect settings (e.g. getting env var).""" + """ + Return required value for potential secret setting. + + Parameters + ---------- + setting_path : str + Dotted path to the setting + config_setting : Union[str, Dict[str, Any]] + Setting value (str or Dict) + + Returns + ------- + Union[Optional[str], Callable[[], Any]] + Either a string or accessor function. + + Raises + ------ + MsticpyImportExtraError + _description_ + NotImplementedError + _description_ + + """ + if isinstance(config_setting, str): + return config_setting + if not isinstance(config_setting, dict): + return NotImplementedError( + "Configuration setting format not recognized.", + f"'{setting_path}' should be a string or dictionary", + "with either 'EnvironmentVar' or 'KeyVault' entry.", + ) if "EnvironmentVar" in config_setting: env_value = os.environ.get(config_setting["EnvironmentVar"]) if not env_value: warnings.warn( f"Environment variable {config_setting['EnvironmentVar']}" - + f" (provider {provider_name})" + + f" ({setting_path})" + " was not set" ) return env_value @@ -324,18 +345,19 @@ def _fetch_setting( if not _SECRETS_ENABLED: raise MsticpyImportExtraError( "Cannot use this feature without Key Vault support installed", - title="Error importing Loading Key Vault and/or keyring libaries", + title="Error importing Loading Key Vault and/or keyring libraries.", extra="keyvault", ) if not _SECRETS_CLIENT: warnings.warn( "Cannot use a KeyVault configuration setting without" + "a KeyVault configuration section in msticpyconfig.yaml" - + f" (provider {provider_name})" + + f" ({setting_path})" ) return None - config_path = (config_section, provider_name, "Args", arg_name) - return _SECRETS_CLIENT.get_secret_accessor( # type:ignore - ".".join(config_path) - ) - return None + return _SECRETS_CLIENT.get_secret_accessor(setting_path) + raise NotImplementedError( + "Configuration setting format not recognized.", + f"'{setting_path}' should be a string or dictionary", + "with either 'EnvironmentVar' or 'KeyVault' entry.", + ) diff --git a/msticpy/common/proxy_settings.py b/msticpy/common/proxy_settings.py new file mode 100644 index 000000000..f46be434e --- /dev/null +++ b/msticpy/common/proxy_settings.py @@ -0,0 +1,64 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +""" +Get proxy settings from config. + +Settings format + +.. code-block:: yaml + + msticpy: + Proxies: + http: + Url: proxy_url + UserName: user + Password: [PLACEHOLDER] + https: + Url: proxy_url + UserName: user + Password: [PLACEHOLDER] + +The entries for the username and password can be specified +as a string, an dictionary of EnvironmentVar: ENV_VAR_NAME +or a dictionary of one of the following: +- KeyVault: None +- KeyVault: secret_name +- KeyVault: vault_name/secret_name + +""" +from typing import Dict, Optional + +from .pkg_config import get_config +from .provider_settings import get_protected_setting + + +def get_http_proxies() -> Optional[Dict[str, str]]: + """Return proxy settings from config.""" + proxy_config = get_config("msticpy.Proxies", None) + if not proxy_config: + return None + proxy_dict = {} + for protocol, settings in proxy_config.items(): + if "Url" not in settings: + continue + if "//" in settings["Url"]: + prot_str, host = settings["Url"].split("//", maxsplit=1) + else: + prot_str, host = protocol, settings["Url"] + user = ( + get_protected_setting("http.proxies.{protocol}", "UserName") + if "UserName" in settings + else None + ) + password = ( + get_protected_setting("http.proxies.{protocol}", "Password") + if "Password" in settings + else None + ) + auth_str = f"{user}:{password}@" if user else "" + + proxy_dict[protocol.casefold()] = f"{prot_str}//{auth_str}{host}" + return proxy_dict diff --git a/msticpy/common/settings.py b/msticpy/common/settings.py new file mode 100644 index 000000000..38d13ec9e --- /dev/null +++ b/msticpy/common/settings.py @@ -0,0 +1,30 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Facade module for settings functions.""" + +from .._version import VERSION + +# pylint: disable=unused-import +from .pkg_config import ( # noqa: F401 + current_config_path, + get_config, + get_http_timeout, + get_settings, + refresh_config, + set_config, + validate_config, +) +from .provider_settings import ( # noqa: F401 + clear_keyring, + get_protected_setting, + get_provider_settings, + refresh_keyring, + reload_settings, +) +from .proxy_settings import get_http_proxies # noqa: F401 + +__version__ = VERSION +__author__ = "Ian Hellen" diff --git a/msticpy/common/utility/package.py b/msticpy/common/utility/package.py index 5b0c56f70..d317bb18c 100644 --- a/msticpy/common/utility/package.py +++ b/msticpy/common/utility/package.py @@ -4,6 +4,7 @@ # license information. # -------------------------------------------------------------------------- """Packaging utility functions.""" +import importlib import os import re import subprocess # nosec @@ -307,3 +308,33 @@ def set_unit_testing(on: bool = True): os.environ[_U_TEST_ENV] = "True" else: os.environ.pop(_U_TEST_ENV, None) + + +def init_getattr(module_name: str, dynamic_imports: Dict[str, str], attrib: str): + """Import and return dynamic attribute.""" + if attrib in dynamic_imports: + module = importlib.import_module(dynamic_imports[attrib]) + return getattr(module, attrib) + raise AttributeError(f"{module_name} has no attribute {attrib}") + + +def init_dir(static_attribs: List[str], dynamic_imports: Dict[str, str]): + """Return list of available attributes.""" + return sorted(set(static_attribs + list(dynamic_imports))) + + +def lazy_import(module: str, attrib: str, call: bool = False): + """Import attribute from module on demand.""" + attribute = None + + def import_item(*args, **kwargs): + """Return the attribute, importing module if needed.""" + nonlocal attribute + if attribute is None: + imp_module = importlib.import_module(module) + attribute = getattr(imp_module, attrib) + return ( + attribute(*args, **kwargs) if (call and callable(attribute)) else attribute + ) + + return import_item diff --git a/msticpy/common/wsconfig.py b/msticpy/common/wsconfig.py index 4b8ab5460..3bf3b28a8 100644 --- a/msticpy/common/wsconfig.py +++ b/msticpy/common/wsconfig.py @@ -4,11 +4,10 @@ # license information. # -------------------------------------------------------------------------- """Module for Log Analytics-related configuration.""" - - import contextlib import json import os +import re from pathlib import Path from typing import Any, Dict, Optional @@ -95,11 +94,23 @@ class WorkspaceConfig: CONF_RES_GROUP_KEY = "resource_group" CONF_WS_NAME_KEY = "workspace_name" + _SETTINGS_TO_CONFIG_NAME_MAP = { + PKG_CONF_TENANT_KEY: CONF_TENANT_ID_KEY, + PKG_CONF_WS_KEY: CONF_WS_ID_KEY, + PKG_CONF_SUB_KEY: CONF_SUB_ID_KEY, + PKG_CONF_RES_GROUP_KEY: CONF_RES_GROUP_KEY, + PKG_CONF_NAME_KEY: CONF_WS_NAME_KEY, + } + _CONFIG_TO_SETTINGS_NAME_MAP = { + val: key for key, val in _SETTINGS_TO_CONFIG_NAME_MAP.items() + } + def __init__( self, workspace: Optional[str] = None, config_file: Optional[str] = None, interactive: bool = True, + config: Optional[Dict[str, str]] = None, ): """ Load current Azure Notebooks configuration for Log Analytics. @@ -112,12 +123,14 @@ def __init__( If this isn't configured, it will search for (first) a config.json and (second) a msticpyconfig.yaml in (first) the current directory and (second) the parent directory and subfolders. - workspace : str, Optional[str] + workspace : Optional[str] Workspace name (where multiple workspaces are configured), by default the Default workspace will be used. interactive : bool, optional If this is False, initializing the class will not raise an exception if no configuration is found. By default, True. + config : Optional[Dict[str, str]], + Workspace configuration as dictionary. """ self._config: Dict[str, str] = {} @@ -125,16 +138,28 @@ def __init__( self._config_file = config_file self.workspace_key = workspace or "Default" # If config file specified, use that - if config_file: + if config: + self._config.update(config) + elif config_file: self._config.update(self._read_config_values(config_file)) else: self._determine_config_source(workspace) + def __getattr__(self, attribute: str): + """Return attribute from configuration.""" + with contextlib.suppress(KeyError): + return self[attribute] + raise AttributeError( + f"{self.__class__.__name__} has no attribute '{attribute}'" + ) + def __getitem__(self, key: str): """Allow property get using dictionary key syntax.""" - if key in self._config: - return self._config[key] - raise KeyError + if key in self._SETTINGS_TO_CONFIG_NAME_MAP: + return self._config.get(self._SETTINGS_TO_CONFIG_NAME_MAP[key]) + if key in self._CONFIG_TO_SETTINGS_NAME_MAP: + return self._config.get(key) + raise KeyError(f"{self.__class__.__name__} has no attribute '{key}'") def __setitem__(self, key: str, value: Any): """Allow property set using dictionary key syntax.""" @@ -208,17 +233,34 @@ def mp_settings(self): } @classmethod - def from_settings(cls, settings: Dict[str, Any]): + def from_settings(cls, settings: Dict[str, Any]) -> "WorkspaceConfig": """Create a WorkstationConfig from MSTICPY Workspace settings.""" - ws_config = cls(workspace="**DUMMY_WORKSPACE**") # type: ignore - ws_config._config = { # type: ignore - cls.CONF_WS_NAME_KEY: settings.get(cls.PKG_CONF_NAME_KEY), # type: ignore - cls.CONF_SUB_ID_KEY: settings.get(cls.PKG_CONF_SUB_KEY), # type: ignore - cls.CONF_WS_ID_KEY: settings.get(cls.PKG_CONF_WS_KEY), # type: ignore - cls.CONF_TENANT_ID_KEY: settings.get(cls.PKG_CONF_TENANT_KEY), # type: ignore - cls.CONF_RES_GROUP_KEY: settings.get(cls.PKG_CONF_RES_GROUP_KEY), # type: ignore - } - return ws_config + return cls( + config={ # type: ignore + cls.CONF_WS_NAME_KEY: settings.get(cls.PKG_CONF_NAME_KEY), # type: ignore + cls.CONF_SUB_ID_KEY: settings.get(cls.PKG_CONF_SUB_KEY), # type: ignore + cls.CONF_WS_ID_KEY: settings.get(cls.PKG_CONF_WS_KEY), # type: ignore + cls.CONF_TENANT_ID_KEY: settings.get(cls.PKG_CONF_TENANT_KEY), # type: ignore + cls.CONF_RES_GROUP_KEY: settings.get(cls.PKG_CONF_RES_GROUP_KEY), # type: ignore + } + ) + + @classmethod + def from_connection_string(cls, connection_str: str) -> "WorkspaceConfig": + """Create a WorkstationConfig from a connection string.""" + tenant_regex = r".*tenant\(\s?['\"](?P[\w]+)['\"].*" + workspace_regex = r".*workspace\(\s?['\"](?P[\w]+)['\"].*" + tenant_id = workspace_id = None + if match := re.match(tenant_regex, connection_str): + tenant_id = match.groupdict()["tenant_id"] + if match := re.match(workspace_regex, connection_str): + workspace_id = match.groupdict()["workspace_id"] + return cls( + config={ + cls.CONF_WS_ID_KEY: workspace_id, # type: ignore[dict-item] + cls.CONF_TENANT_ID_KEY: tenant_id, # type: ignore[dict-item] + } + ) @staticmethod def _read_config_values(file_path: str) -> Dict[str, str]: @@ -291,7 +333,7 @@ def _determine_config_source(self, workspace): self._read_pkg_config_values(workspace_name=workspace) if self.config_loaded: return - # Next, search for a config.json in the current director + # Next, search for a config.json in the current directory if Path("./config.json").exists(): self._config_file = "./config.json" else: @@ -301,15 +343,7 @@ def _determine_config_source(self, workspace): return # Finally, search for a msticpyconfig.yaml - if ( - os.environ.get("MSTICPYCONFIG") - and Path(os.environ.get("MSTICPYCONFIG")).exists() - ): - self._config_file = os.environ.get("MSTICPYCONFIG") - elif Path("./msticpyconfig.yaml").exists(): - self._config_file = "./msticpyconfig.yaml" - else: - self._config_file = self._search_for_file("**/msticpyconfig.yaml") + self._config_file = self._search_for_file("**/msticpyconfig.yaml") if self._config_file: os.environ["MSTICPYCONFIG"] = self._config_file refresh_config() @@ -327,44 +361,39 @@ def _determine_config_source(self, workspace): ) ) - def _read_pkg_config_values(self, workspace_name: str = None): - as_settings = get_config("AzureSentinel", {}) - if not as_settings: - return {} - ws_settings = as_settings.get("Workspaces") # type: ignore + def _read_pkg_config_values(self, workspace_name: Optional[str] = None): + """Try to find a usable config from the MSTICPy config file.""" + ws_settings = get_config("AzureSentinel", {}).get("Workspaces") # type: ignore if not ws_settings: - return {} - if workspace_name and workspace_name in ws_settings: - selected_workspace = ws_settings[workspace_name] + return + selected_workspace: Dict[str, str] = {} + if workspace_name: + selected_workspace = self._lookup_ws_name_and_id( + workspace_name, ws_settings + ) elif "Default" in ws_settings: selected_workspace = ws_settings["Default"] elif len(ws_settings) == 1: selected_workspace = next(iter(ws_settings.values())) - else: - return {} - if ( - selected_workspace - and self.PKG_CONF_WS_KEY in selected_workspace - and self.PKG_CONF_TENANT_KEY in selected_workspace - ): - self._config[self.CONF_WS_ID_KEY] = selected_workspace.get( - self.PKG_CONF_WS_KEY - ) - self._config[self.CONF_TENANT_ID_KEY] = selected_workspace.get( - self.PKG_CONF_TENANT_KEY - ) - if self.PKG_CONF_SUB_KEY in selected_workspace: - self._config[self.CONF_SUB_ID_KEY] = selected_workspace.get( - self.PKG_CONF_SUB_KEY - ) - if self.PKG_CONF_RES_GROUP_KEY in selected_workspace: - self._config[self.CONF_RES_GROUP_KEY] = selected_workspace.get( - self.PKG_CONF_RES_GROUP_KEY - ) - if self.PKG_CONF_NAME_KEY in selected_workspace: - self._config[self.CONF_WS_NAME_KEY] = selected_workspace.get( - self.PKG_CONF_NAME_KEY - ) + + if selected_workspace: + self._config = {} + for name, value in selected_workspace.items(): + tgt_name = self._SETTINGS_TO_CONFIG_NAME_MAP.get(name) + if tgt_name and value: + self._config[tgt_name] = value + + def _lookup_ws_name_and_id(self, ws_name: str, ws_configs: dict): + for name, ws_config in ws_configs.items(): + if ws_name.casefold() == name.casefold(): + return ws_config + if ws_config.get(self.PKG_CONF_WS_KEY, "").casefold() == ws_name.casefold(): + return ws_config + if ( + ws_config.get(self.PKG_CONF_NAME_KEY, "").casefold() + == ws_name.casefold() + ): + return ws_config return {} def _search_for_file(self, pattern: str) -> Optional[str]: diff --git a/msticpy/config/__init__.py b/msticpy/config/__init__.py index ffc920315..3b4344d3a 100644 --- a/msticpy/config/__init__.py +++ b/msticpy/config/__init__.py @@ -12,9 +12,21 @@ It use the ipywidgets package. """ +from ..common.utility.package import init_dir, init_getattr -from .mp_config_control import MpConfigControls +_STATIC_ATTRIBS = list(locals().keys()) +_DEFAULT_IMPORTS = { + "MpConfigControls": "msticpy.config.mp_config_control", + "MpConfigEdit": "msticpy.config.mp_config_edit", + "MpConfigFile": "msticpy.config.mp_config_file", +} -# flake8: noqa: F403 -from .mp_config_edit import MpConfigEdit -from .mp_config_file import MpConfigFile + +def __getattr__(attrib: str): + """Import and a dynamic attribute of module.""" + return init_getattr(__name__, _DEFAULT_IMPORTS, attrib) + + +def __dir__(): + """Return attribute list.""" + return init_dir(_STATIC_ATTRIBS, _DEFAULT_IMPORTS) diff --git a/msticpy/config/ce_azure_sentinel.py b/msticpy/config/ce_azure_sentinel.py index 7b4bda467..3d574c4de 100644 --- a/msticpy/config/ce_azure_sentinel.py +++ b/msticpy/config/ce_azure_sentinel.py @@ -9,7 +9,9 @@ import ipywidgets as widgets from .._version import VERSION -from ..context.azure.sentinel_core import MicrosoftSentinel +from ..common.utility.package import lazy_import + +# from ..context.azure.sentinel_core import MicrosoftSentinel from .ce_common import ( ITEM_LIST_LAYOUT, get_or_create_mpc_section, @@ -22,6 +24,8 @@ __version__ = VERSION __author__ = "Ian Hellen" +ms_sentinel = lazy_import("msticpy.context.azure.sentinel_core", "MicrosoftSentinel") + # pylint: disable=too-many-ancestors class CEAzureSentinel(CEItemsBase): @@ -216,7 +220,7 @@ def _imp_ws_from_url(self, btn): if not url: self.set_status("Please paste portal URL into ") return - self._update_settings(MicrosoftSentinel.get_workspace_details_from_url(url)) + self._update_settings(ms_sentinel().get_workspace_details_from_url(url)) def _update_settings(self, ws_settings): """Update current controls with workspace settings.""" @@ -246,11 +250,11 @@ def _resolve_settings(self, btn): return if workspace_id: self._update_settings( - MicrosoftSentinel.get_workspace_settings(workspace_id=workspace_id) + ms_sentinel().get_workspace_settings(workspace_id=workspace_id) ) else: self._update_settings( - MicrosoftSentinel.get_workspace_settings_by_name( + ms_sentinel().get_workspace_settings_by_name( workspace_name=workspace_name, subscription_id=subscription_id, resource_group=resource_group, diff --git a/msticpy/config/ce_msticpy.py b/msticpy/config/ce_msticpy.py new file mode 100644 index 000000000..c620f9899 --- /dev/null +++ b/msticpy/config/ce_msticpy.py @@ -0,0 +1,38 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Key Vault component edit.""" +from .._version import VERSION +from .ce_simple_settings import CESimpleSettings + +__version__ = VERSION +__author__ = "Ian Hellen" + + +class CEMsticpy(CESimpleSettings): + """Key Vault settings edit component.""" + + _DESCRIPTION = "MSTICPy Settings" + _COMP_PATH = "msticpy" + _HELP_TEXT = """ + Set the global parameters for MSTICPy.
+ + This section lets you set some global parameters such as:
+ +
    +
  • Paths to query files
    • +
  • Enabling/disabling friendly exceptions
    • +
  • HTTP timeouts
    • +
+ Note: Proxies cannot currently be set from this interface. + Edit the msticpyconfig.yaml file directly to set proxies + following the guidance in the link to the documentation.
+ """ + _HELP_URI = { + "MSTICPy global settings": ( + "https://msticpy.readthedocs.io/en/latest/getting_started/" + + "msticpyconfig.html#msticpy-global-settings" + ) + } diff --git a/msticpy/config/file_browser.py b/msticpy/config/file_browser.py index 2d90336ca..d3e2c1cc0 100644 --- a/msticpy/config/file_browser.py +++ b/msticpy/config/file_browser.py @@ -4,8 +4,10 @@ # license information. # -------------------------------------------------------------------------- """File Browser class.""" + +import contextlib from pathlib import Path -from typing import Any, Callable, List, Optional, Tuple +from typing import Any, Callable, List, Optional, Tuple, Union import ipywidgets as widgets @@ -16,13 +18,16 @@ __author__ = "Ian Hellen" +StrOrPath = Union[str, Path] + + # pylint: disable=too-many-instance-attributes class FileBrowser(CompEditDisplayMixin): """File system browser control.""" PARENT = ".." - def __init__(self, path: str = ".", select_cb: Callable[[str], Any] = None): + def __init__(self, path: StrOrPath = ".", select_cb: Callable[[str], Any] = None): """ Initialize the class for path and with optional callback. @@ -138,7 +143,7 @@ def _return_file(self, btn): self.action(self.file) @staticmethod - def read_folder(folder: str) -> Tuple[List[str], List[str]]: + def read_folder(folder: StrOrPath) -> Tuple[List[StrOrPath], List[StrOrPath]]: """ Return folder contents. @@ -149,16 +154,22 @@ def read_folder(folder: str) -> Tuple[List[str], List[str]]: Returns ------- - Tuple[List[str], List[str]] + Tuple[List[StrOrPath], List[StrOrPath]] List of folders and files in the folder. """ contents = list(Path(folder).glob("*")) - files = [file.parts[-1] for file in contents if file.is_file()] - folders = [fld.parts[-1] for fld in contents if fld.is_dir()] - return folders, files - - def get_folder_list(self, folders: List[str]) -> List[str]: + files = [] + folders = [] + for file in contents: + with contextlib.suppress(PermissionError, IOError): + if file.is_file(): + files.append(file) + elif file.is_dir(): + folders.append(file) + return folders, files # type: ignore[return-value] + + def get_folder_list(self, folders: List[StrOrPath]) -> List[StrOrPath]: """Return sorted list of folders with '..' inserted if not root.""" if self.current_folder != Path(self.current_folder.parts[0]): return [self.PARENT, *(sorted(folders))] @@ -170,10 +181,8 @@ def _search(self, btn): if self.txt_search.value: found_files: Optional[List[Path]] = None while found_files is None: - try: + with contextlib.suppress(FileNotFoundError): found_files = list(self.current_folder.rglob(self.txt_search.value)) - except FileNotFoundError: - pass self.select_search.options = [ str(file) for file in found_files if file.exists() ] diff --git a/msticpy/config/mp_config_edit.py b/msticpy/config/mp_config_edit.py index f1e9db9d7..b007ac184 100644 --- a/msticpy/config/mp_config_edit.py +++ b/msticpy/config/mp_config_edit.py @@ -14,9 +14,11 @@ from .ce_azure_sentinel import CEAzureSentinel from .ce_data_providers import CEDataProviders from .ce_keyvault import CEKeyVault +from .ce_msticpy import CEMsticpy from .ce_other_providers import CEOtherProviders from .ce_ti_providers import CETIProviders -from .ce_user_defaults import CEAutoLoadComps, CEAutoLoadQProvs + +# from .ce_user_defaults import CEAutoLoadComps, CEAutoLoadQProvs from .comp_edit import CETabControlDef, CompEditDisplayMixin, CompEditTabs from .mp_config_control import MpConfigControls, get_mpconfig_definitions from .mp_config_file import MpConfigFile @@ -30,14 +32,15 @@ class MpConfigEdit(CompEditDisplayMixin): """Msticpy Configuration helper class.""" _TAB_DEFINITIONS = { + "MSTICPy": CEMsticpy, "MicrosoftSentinel": CEAzureSentinel, "TI Providers": CETIProviders, "Data Providers": CEDataProviders, "GeoIP Providers": CEOtherProviders, "Key Vault": CEKeyVault, "Azure": CEAzure, - "Autoload QueryProvs": CEAutoLoadQProvs, - "Autoload Components": CEAutoLoadComps, + # "Autoload QueryProvs": CEAutoLoadQProvs, + # "Autoload Components": CEAutoLoadComps, } def __init__( diff --git a/msticpy/context/azure/sentinel_utils.py b/msticpy/context/azure/sentinel_utils.py index 2a38e14f0..b964ee55f 100644 --- a/msticpy/context/azure/sentinel_utils.py +++ b/msticpy/context/azure/sentinel_utils.py @@ -165,6 +165,7 @@ def _build_sent_res_id( The formatted resource ID. """ + print("_build_sent_res_id", sub_id, res_grp, ws_name) if not sub_id or not res_grp or not ws_name: config = self._check_config( workspace_name=ws_name, diff --git a/msticpy/context/domain_utils.py b/msticpy/context/domain_utils.py index 367c1df7a..2d0e56cac 100644 --- a/msticpy/context/domain_utils.py +++ b/msticpy/context/domain_utils.py @@ -32,7 +32,7 @@ from .._version import VERSION from ..common.exceptions import MsticpyUserConfigError -from ..common.pkg_config import get_config, get_http_timeout +from ..common.settings import get_config, get_http_timeout from ..common.utility import export, mp_ua_header __version__ = VERSION diff --git a/msticpy/data/core/data_providers.py b/msticpy/data/core/data_providers.py index 9d2d11307..661993540 100644 --- a/msticpy/data/core/data_providers.py +++ b/msticpy/data/core/data_providers.py @@ -4,26 +4,26 @@ # license information. # -------------------------------------------------------------------------- """Data provider loader.""" -import re -from collections import abc from datetime import datetime from functools import partial from itertools import tee from pathlib import Path -from typing import Any, Dict, Iterable, List, NamedTuple, Optional, Pattern, Union +from typing import Any, Dict, Iterable, List, Optional, Union import pandas as pd from tqdm.auto import tqdm from ..._version import VERSION -from ...common.exceptions import MsticpyDataQueryError from ...common.pkg_config import get_config from ...common.utility import export, valid_pyname from ...nbwidgets import QueryTime -from ..drivers import DriverBase, import_driver +from ..drivers import import_driver +from ..drivers.driver_base import DriverBase, DriverProps from .param_extractor import extract_query_params from .query_container import QueryContainer from .query_defns import DataEnvironment +from .query_provider_connections_mixin import QueryProviderConnectionsMixin +from .query_provider_utils_mixin import QueryProviderUtilsMixin from .query_source import QuerySource from .query_store import QueryStore @@ -32,26 +32,18 @@ _HELP_FLAGS = ("help", "?") _DEBUG_FLAGS = ("print", "debug_query", "print_query") -_COMPATIBLE_DRIVER_MAPPINGS = {"mssentinel": ["m365d"], "mde": ["m365d"]} - - -class QueryParam(NamedTuple): - """ - Named tuple for custom query parameters. - - name and data_type are mandatory. - description and default are optional. - - """ - - name: str - data_type: str - description: Optional[str] = None - default: Optional[str] = None +_COMPATIBLE_DRIVER_MAPPINGS = { + "mssentinel": ["m365d"], + "mde": ["m365d"], + "mssentinel_new": ["mssentinel"], + "kusto_new": ["kusto"], +} +# These are mixin classes that do not have an __init__ method +# pylint: disable=super-init-not-called @export -class QueryProvider: +class QueryProvider(QueryProviderConnectionsMixin, QueryProviderUtilsMixin): """ Container for query store and query execution provider. @@ -60,13 +52,11 @@ class QueryProvider: """ - create_param = QueryParam - def __init__( # noqa: MC0001 self, data_environment: Union[str, DataEnvironment], - driver: DriverBase = None, - query_paths: List[str] = None, + driver: Optional[DriverBase] = None, + query_paths: Optional[List[str]] = None, **kwargs, ): """ @@ -85,6 +75,13 @@ def __init__( # noqa: MC0001 kwargs : Other arguments are passed to the data provider driver. + Notes + ----- + Additional keyword arguments are passed to the data provider driver. + The driver may support additional keyword arguments, use + the QueryProvider.driver_help() method to get a list of these + parameters. + See Also -------- DataProviderBase : base class for data query providers. @@ -103,7 +100,6 @@ def __init__( # noqa: MC0001 data_environment = data_env else: raise TypeError(f"Unknown data environment {data_environment}") - self.environment = data_environment.name self._driver_kwargs = kwargs.copy() if driver is None: @@ -112,12 +108,22 @@ def __init__( # noqa: MC0001 driver = self.driver_class(data_environment=data_environment, **kwargs) else: raise LookupError( - "Could not find suitable data provider for", f" {self.environment}" + "Could not find suitable data provider for", f" {data_environment}" ) else: self.driver_class = driver.__class__ + # allow the driver to override the data environment used + # for selecting queries + self.environment = ( + driver.get_driver_property(DriverProps.EFFECTIVE_ENV) + or data_environment.name + ) + self._additional_connections: Dict[str, DriverBase] = {} self._query_provider = driver + # replace the connect method docstring with that from + # the driver's connect method + self.__class__.connect.__doc__ = driver.connect.__doc__ self.all_queries = QueryContainer() # Add any query files @@ -141,7 +147,7 @@ def __getattr__(self, name): return getattr(parent, child_name) raise AttributeError(f"{name} is not a valid attribute.") - def connect(self, connection_str: str = None, **kwargs): + def connect(self, connection_str: Optional[str] = None, **kwargs): """ Connect to data source. @@ -158,270 +164,23 @@ def connect(self, connection_str: str = None, **kwargs): for attr_name, attr in self._query_provider.public_attribs.items(): setattr(self, attr_name, attr) + refresh_query_funcs = False + # if the driver supports dynamic filtering of queries, + # filter the query store based on connect-time parameters + if self._query_provider.filter_queries_on_connect: + self.query_store.apply_query_filter(self._query_provider.query_usable) + refresh_query_funcs = True # Add any built-in or dynamically retrieved queries from driver if self._query_provider.has_driver_queries: driver_queries = self._query_provider.driver_queries self._add_driver_queries(queries=driver_queries) - # Since we're now connected, add Pivot functions - self._add_pivots(lambda: self._query_time.timespan) - - def add_connection( - self, - connection_str: Optional[str] = None, - alias: Optional[str] = None, - **kwargs, - ): - """ - Add an additional connection for the query provider. - - Parameters - ---------- - connection_str : Optional[str], optional - Connection string for the provider, by default None - alias : Optional[str], optional - Alias to use for the connection, by default None - - Other Parameters - ---------------- - kwargs : Dict[str, Any] - Other parameters passed to the driver constructor. - - Notes - ----- - Some drivers may accept types other than strings for the - `connection_str` parameter. - - """ - # create a new instance of the driver class - new_driver = self.driver_class(**(self._driver_kwargs)) - # connect - new_driver.connect(connection_str=connection_str, **kwargs) - # add to collection - driver_key = alias or str(len(self._additional_connections)) - self._additional_connections[driver_key] = new_driver - - @property - def connected(self) -> bool: - """ - Return True if the provider is connected. - - Returns - ------- - bool - True if the provider is connected. - - """ - return self._query_provider.connected - - @property - def connection_string(self) -> str: - """ - Return provider connection string. - - Returns - ------- - str - Provider connection string. - - """ - return self._query_provider.current_connection - - @property - def schema(self) -> Dict[str, Dict]: - """ - Return current data schema of connection. - - Returns - ------- - Dict[str, Dict] - Data schema of current connection. - - """ - return self._query_provider.schema - - @property - def schema_tables(self) -> List[str]: - """ - Return list of tables in the data schema of the connection. - - Returns - ------- - List[str] - Tables in the of current connection. - - """ - return list(self._query_provider.schema.keys()) - - @property - def instance(self) -> Optional[str]: - """ - Return instance name, if any for provider. - - Returns - ------- - Optional[str] - The instance name or None for drivers that do not - support multiple instances. - - """ - return self._query_provider.instance - - def import_query_file(self, query_file: str): - """ - Import a yaml data source definition. - - Parameters - ---------- - query_file : str - Path to the file to import - - """ - self.query_store.import_file(query_file) - self._add_query_functions() - - @classmethod - def list_data_environments(cls) -> List[str]: - """ - Return list of current data environments. - - Returns - ------- - List[str] - List of current data environments + refresh_query_funcs = True - """ - # pylint: disable=not-an-iterable - return [env for env in DataEnvironment.__members__ if env != "Unknown"] - # pylint: enable=not-an-iterable - - def list_queries(self, substring: Optional[str] = None) -> List[str]: - """ - Return list of family.query in the store. - - Parameters - ---------- - substring : Optional[str] - Optional pattern - will return only queries matching the pattern, - default None. - - Returns - ------- - List[str] - List of queries - - """ - if substring: - return list( - filter( - lambda x: substring in x.lower(), # type: ignore - self.query_store.query_names, - ) - ) - return list(self.query_store.query_names) - - def search( - self, - search: Union[str, Iterable[str]] = None, - table: Union[str, Iterable[str]] = None, - param: Union[str, Iterable[str]] = None, - ignore_case: bool = True, - ) -> List[str]: - """ - Search queries for match properties. - - Parameters - ---------- - search : Union[str, Iterable[str]], optional - String or iterable of search terms to match on - any property of the query, by default None. - The properties include: name, description, table, - parameter names and query_text. - table : Union[str, Iterable[str]], optional - String or iterable of search terms to match on - the query table name, by default None - param : Union[str, Iterable[str]], optional - String or iterable of search terms to match on - the query parameter names, by default None - ignore_case : bool - Use case-insensitive search, default is True. - - Returns - ------- - List[str] - A list of matched queries - - Notes - ----- - Search terms are treated as regular expressions. - Supplying multiple parameters returns the intersection - of matches for each category. For example: - `qry_prov.search(search="account", table="syslog")` will - match queries that have a table parameter of "syslog" AND - have the term "Account" somewhere in the query properties. - - """ - if not (search or table or param): - return [] - - glob_searches = _normalize_to_regex(search, ignore_case) - table_searches = _normalize_to_regex(table, ignore_case) - param_searches = _normalize_to_regex(param, ignore_case) - search_hits: List[str] = [] - for query, search_data in self.query_store.search_items.items(): - glob_match = (not glob_searches) or any( - re.search(term, prop) - for term in glob_searches - for prop in search_data.values() - ) - table_match = (not table_searches) or any( - re.search(term, search_data["table"]) for term in table_searches - ) - param_match = (not param_searches) or any( - re.search(term, search_data["params"]) for term in param_searches - ) - if glob_match and table_match and param_match: - search_hits.append(query) - return sorted(search_hits) - - def list_connections(self) -> List[str]: - """ - Return a list of current connections or the default connection. - - Returns - ------- - List[str] - The alias and connection string for each connection. - - """ - add_connections = [ - f"{alias}: {driver.current_connection}" - for alias, driver in self._additional_connections.items() - ] - return [f"Default: {self._query_provider.current_connection}", *add_connections] - - def query_help(self, query_name: str): - """ - Print help for `query_name`. - - Parameters - ---------- - query_name : str - The name of the query. + if refresh_query_funcs: + self._add_query_functions() - """ - self.query_store[query_name].help() - - def get_query(self, query_name: str) -> str: - """ - Return the raw query text for `query_name`. - - Parameters - ---------- - query_name : str - The name of the query. - - """ - return self.query_store[query_name].query + # Since we're now connected, add Pivot functions + self._add_pivots(lambda: self._query_time.timespan) def exec_query(self, query: str, **kwargs) -> Union[pd.DataFrame, Any]: """ @@ -454,123 +213,13 @@ def exec_query(self, query: str, **kwargs) -> Union[pd.DataFrame, Any]: ) if not self._additional_connections: return result - # run query against all connections - results = [result] - print(f"Running query for {len(self._additional_connections)} connections.") - for con_name, connection in self._additional_connections.items(): - print(f"{con_name}...") - try: - results.append( - connection.query(query, query_source=query_source, **query_options) - ) - except MsticpyDataQueryError: - print(f"Query {con_name} failed.") - return pd.concat(results) - - def browse_queries(self, **kwargs): - """ - Return QueryProvider query browser. - - Other Parameters - ---------------- - kwargs : - passed to SelectItem constructor. - - Returns - ------- - SelectItem - SelectItem browser for TI Data. - - """ - # pylint: disable=import-outside-toplevel - from ...vis.query_browser import browse_queries - - return browse_queries(self, **kwargs) - - # alias for browse_queries - browse = browse_queries + return self._exec_additional_connections(query, result, **kwargs) @property def query_time(self): """Return the default QueryTime control for queries.""" return self._query_time - def add_custom_query( - self, - name: str, - query: str, - family: Union[str, Iterable[str]], - description: Optional[str] = None, - parameters: Optional[Iterable[QueryParam]] = None, - ): - """ - Add a custom function to the provider. - - Parameters - ---------- - name : str - The name of the query. - query : str - The query text (optionally parameterized). - family : Union[str, Iterable[str]] - The query group/family or list of families. The query will - be added to attributes of the query provider with these - names. - description : Optional[str], optional - Optional description (for query help), by default None - parameters : Optional[Iterable[QueryParam]], optional - Optional list of parameter definitions, by default None. - If the query is parameterized you must supply definitions - for the parameters here - at least name and type. - Parameters can be the named tuple QueryParam (also - exposed as QueryProvider.Param) or a 4-value - - Examples - -------- - >>> qp = QueryProvider("MSSentinel") - >>> qp_host = qp.create_paramramram("host_name", "str", "Name of Host") - >>> qp_start = qp.create_param("start", "datetime") - >>> qp_end = qp.create_param("end", "datetime") - >>> qp_evt = qp.create_param("event_id", "int", None, 4688) - >>> - >>> query = ''' - >>> SecurityEvent - >>> | where EventID == {event_id} - >>> | where TimeGenerated between (datetime({start}) .. datetime({end})) - >>> | where Computer has "{host_name}" - >>> ''' - >>> - >>> qp.add_custom_query( - >>> name="test_host_proc", - >>> query=query, - >>> family="Custom", - >>> parameters=[qp_host, qp_start, qp_end, qp_evt] - >>> ) - - """ - if parameters: - param_dict = { - param[0]: { - "type": param[1], - "default": param[2], - "description": param[3], - } - for param in parameters - } - else: - param_dict = {} - source = { - "args": {"query": query}, - "description": description, - "parameters": param_dict, - } - metadata = {"data_families": [family] if isinstance(family, str) else family} - query_source = QuerySource( - name=name, source=source, defaults={}, metadata=metadata - ) - self.query_store.add_data_source(query_source) - self._add_query_functions() - def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: if not self._query_provider.loaded: raise ValueError("Provider is not loaded.") @@ -594,7 +243,9 @@ def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: return None params, missing = extract_query_params(query_source, *args, **kwargs) - self._check_for_time_params(params, missing) + query_options = { + "default_time_params": self._check_for_time_params(params, missing) + } if missing: query_source.help() raise ValueError(f"No values found for these parameters: {missing}") @@ -618,18 +269,22 @@ def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: if _debug_flag(*args, **kwargs): return query_str - # Handle any query options passed - query_options = _get_query_options(params, kwargs) + # Handle any query options passed and run the query + query_options.update(_get_query_options(params, kwargs)) return self.exec_query(query_str, query_source=query_source, **query_options) - def _check_for_time_params(self, params, missing): + def _check_for_time_params(self, params, missing) -> bool: """Fall back on builtin query time if no time parameters were supplied.""" + defaults_added = False if "start" in missing: missing.remove("start") params["start"] = self._query_time.start + defaults_added = True if "end" in missing: missing.remove("end") params["end"] = self._query_time.end + defaults_added = True + return defaults_added def _get_query_folder_for_env(self, root_path: str, environment: str) -> List[Path]: """Return query folder for current environment.""" @@ -834,18 +489,8 @@ def _get_query_options( if not query_options: # Any kwargs left over we send to the query provider driver query_options = {key: val for key, val in kwargs.items() if key not in params} + query_options["time_span"] = { + "start": params.get("start"), + "end": params.get("end"), + } return query_options - - -def _normalize_to_regex( - search_term: Union[str, Iterable[str], None], ignore_case: bool -) -> List[Pattern[str]]: - """Return iterable or str search term as list of compiled reg expressions.""" - if not search_term: - return [] - regex_opts = [re.IGNORECASE] if ignore_case else [] - if isinstance(search_term, str): - return [re.compile(search_term, *regex_opts)] - if isinstance(search_term, abc.Iterable): - return [re.compile(term, *regex_opts) for term in set(search_term)] - return [] diff --git a/msticpy/data/core/query_defns.py b/msticpy/data/core/query_defns.py index 282d5848c..a42626e7d 100644 --- a/msticpy/data/core/query_defns.py +++ b/msticpy/data/core/query_defns.py @@ -106,6 +106,8 @@ class DataEnvironment(Enum): Cybereason = 12 Elastic = 14 OSQueryLogs = 15 + MSSentinel_New = 16 + Kusto_New = 17 @classmethod def parse(cls, value: Union[str, int]) -> "DataEnvironment": diff --git a/msticpy/data/core/query_provider_connections_mixin.py b/msticpy/data/core/query_provider_connections_mixin.py new file mode 100644 index 000000000..e436a3144 --- /dev/null +++ b/msticpy/data/core/query_provider_connections_mixin.py @@ -0,0 +1,98 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Query Provider additional connection methods.""" +from typing import Any, Dict, List, Optional, Protocol + +import pandas as pd + +from ..._version import VERSION +from ...common.exceptions import MsticpyDataQueryError +from ..drivers.driver_base import DriverBase + +__version__ = VERSION +__author__ = "Ian Hellen" + + +# pylint: disable=too-few-public-methods +class QueryProviderProtocol(Protocol): + """Protocol for required properties of QueryProvider class.""" + + driver_class: Any + _driver_kwargs: Dict[str, Any] + _additional_connections: Dict[str, Any] + _query_provider: DriverBase + + +# pylint: disable=super-init-not-called +class QueryProviderConnectionsMixin(QueryProviderProtocol): + """Mixin additional connection handling QueryProvider class.""" + + def add_connection( + self, + connection_str: Optional[str] = None, + alias: Optional[str] = None, + **kwargs, + ): + """ + Add an additional connection for the query provider. + + Parameters + ---------- + connection_str : Optional[str], optional + Connection string for the provider, by default None + alias : Optional[str], optional + Alias to use for the connection, by default None + + Other Parameters + ---------------- + kwargs : Dict[str, Any] + Other parameters passed to the driver constructor. + + Notes + ----- + Some drivers may accept types other than strings for the + `connection_str` parameter. + + """ + # create a new instance of the driver class + new_driver = self.driver_class(**(self._driver_kwargs)) + # connect + new_driver.connect(connection_str=connection_str, **kwargs) + # add to collection + driver_key = alias or str(len(self._additional_connections)) + self._additional_connections[driver_key] = new_driver + + def list_connections(self) -> List[str]: + """ + Return a list of current connections or the default connection. + + Returns + ------- + List[str] + The alias and connection string for each connection. + + """ + add_connections = [ + f"{alias}: {driver.current_connection}" + for alias, driver in self._additional_connections.items() + ] + return [f"Default: {self._query_provider.current_connection}", *add_connections] + + def _exec_additional_connections(self, query, result, **kwargs) -> pd.DataFrame: + """Return results of query run query against additional connections.""" + query_source = kwargs.get("query_source") + query_options = kwargs.get("query_options", {}) + results = [result] + print(f"Running query for {len(self._additional_connections)} connections.") + for con_name, connection in self._additional_connections.items(): + print(f"{con_name}...") + try: + results.append( + connection.query(query, query_source=query_source, **query_options) + ) + except MsticpyDataQueryError: + print(f"Query {con_name} failed.") + return pd.concat(results) diff --git a/msticpy/data/core/query_provider_utils_mixin.py b/msticpy/data/core/query_provider_utils_mixin.py new file mode 100644 index 000000000..858d34cc9 --- /dev/null +++ b/msticpy/data/core/query_provider_utils_mixin.py @@ -0,0 +1,375 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Query Provider mixin methods.""" +import re +from collections import abc +from typing import Dict, Iterable, List, NamedTuple, Optional, Pattern, Protocol, Union + +from ..._version import VERSION +from ...common.utility.package import lazy_import +from ..drivers.driver_base import DriverBase +from .query_defns import DataEnvironment +from .query_source import QuerySource +from .query_store import QueryStore + +__version__ = VERSION +__author__ = "Ian Hellen" + +query_browser = lazy_import("msticpy.vis.query_browser", "browse_queries") + + +# pylint: disable=too-few-public-methods +class QueryProviderProtocol(Protocol): + """Protocol for required properties of QueryProvider class.""" + + _query_provider: DriverBase + query_store: QueryStore + + def _add_query_functions(self): + pass + + +class QueryParam(NamedTuple): + """ + Named tuple for custom query parameters. + + name and data_type are mandatory. + description and default are optional. + + """ + + name: str + data_type: str + description: Optional[str] = None + default: Optional[str] = None + + +# pylint: disable=super-init-not-called +class QueryProviderUtilsMixin(QueryProviderProtocol): + """Mixin utility methods for QueryProvider class.""" + + create_param = QueryParam + + @property + def connected(self) -> bool: + """ + Return True if the provider is connected. + + Returns + ------- + bool + True if the provider is connected. + + """ + return self._query_provider.connected + + @property + def connection_string(self) -> str: + """ + Return provider connection string. + + Returns + ------- + str + Provider connection string. + + """ + return self._query_provider.current_connection + + @property + def schema(self) -> Dict[str, Dict]: + """ + Return current data schema of connection. + + Returns + ------- + Dict[str, Dict] + Data schema of current connection. + + """ + return self._query_provider.schema + + @property + def schema_tables(self) -> List[str]: + """ + Return list of tables in the data schema of the connection. + + Returns + ------- + List[str] + Tables in the of current connection. + + """ + return list(self._query_provider.schema.keys()) + + @property + def instance(self) -> Optional[str]: + """ + Return instance name, if any for provider. + + Returns + ------- + Optional[str] + The instance name or None for drivers that do not + support multiple instances. + + """ + return self._query_provider.instance + + def import_query_file(self, query_file: str): + """ + Import a yaml data source definition. + + Parameters + ---------- + query_file : str + Path to the file to import + + """ + self.query_store.import_file(query_file) + self._add_query_functions() + + def driver_help(self): + """Display help for the driver.""" + print(self._query_provider.__doc__) + + @classmethod + def list_data_environments(cls) -> List[str]: + """ + Return list of current data environments. + + Returns + ------- + List[str] + List of current data environments + + """ + # pylint: disable=not-an-iterable + return [env for env in DataEnvironment.__members__ if env != "Unknown"] + # pylint: enable=not-an-iterable + + def list_queries(self, substring: Optional[str] = None) -> List[str]: + """ + Return list of family.query in the store. + + Parameters + ---------- + substring : Optional[str] + Optional pattern - will return only queries matching the pattern, + default None. + + Returns + ------- + List[str] + List of queries + + """ + if substring: + return list( + filter( + lambda x: substring in x.lower(), # type: ignore + self.query_store.query_names, + ) + ) + return list(self.query_store.query_names) + + def search( + self, + search: Union[str, Iterable[str]] = None, + table: Union[str, Iterable[str]] = None, + param: Union[str, Iterable[str]] = None, + ignore_case: bool = True, + ) -> List[str]: + """ + Search queries for match properties. + + Parameters + ---------- + search : Union[str, Iterable[str]], optional + String or iterable of search terms to match on + any property of the query, by default None. + The properties include: name, description, table, + parameter names and query_text. + table : Union[str, Iterable[str]], optional + String or iterable of search terms to match on + the query table name, by default None + param : Union[str, Iterable[str]], optional + String or iterable of search terms to match on + the query parameter names, by default None + ignore_case : bool + Use case-insensitive search, default is True. + + Returns + ------- + List[str] + A list of matched queries + + Notes + ----- + Search terms are treated as regular expressions. + Supplying multiple parameters returns the intersection + of matches for each category. For example: + `qry_prov.search(search="account", table="syslog")` will + match queries that have a table parameter of "syslog" AND + have the term "Account" somewhere in the query properties. + + """ + if not (search or table or param): + return [] + + glob_searches = _normalize_to_regex(search, ignore_case) + table_searches = _normalize_to_regex(table, ignore_case) + param_searches = _normalize_to_regex(param, ignore_case) + search_hits: List[str] = [] + for query, search_data in self.query_store.search_items.items(): + glob_match = (not glob_searches) or any( + re.search(term, prop) + for term in glob_searches + for prop in search_data.values() + ) + table_match = (not table_searches) or any( + re.search(term, search_data["table"]) for term in table_searches + ) + param_match = (not param_searches) or any( + re.search(term, search_data["params"]) for term in param_searches + ) + if glob_match and table_match and param_match: + search_hits.append(query) + return sorted(search_hits) + + def query_help(self, query_name: str): + """ + Print help for `query_name`. + + Parameters + ---------- + query_name : str + The name of the query. + + """ + self.query_store[query_name].help() + + def get_query(self, query_name: str) -> str: + """ + Return the raw query text for `query_name`. + + Parameters + ---------- + query_name : str + The name of the query. + + """ + return self.query_store[query_name].query + + def browse_queries(self, **kwargs): + """ + Return QueryProvider query browser. + + Other Parameters + ---------------- + kwargs : + passed to SelectItem constructor. + + Returns + ------- + SelectItem + SelectItem browser for TI Data. + + """ + return query_browser()(self, **kwargs) + + # alias for browse_queries + browse = browse_queries + + def add_custom_query( + self, + name: str, + query: str, + family: Union[str, Iterable[str]], + description: Optional[str] = None, + parameters: Optional[Iterable[QueryParam]] = None, + ): + """ + Add a custom function to the provider. + + Parameters + ---------- + name : str + The name of the query. + query : str + The query text (optionally parameterized). + family : Union[str, Iterable[str]] + The query group/family or list of families. The query will + be added to attributes of the query provider with these + names. + description : Optional[str], optional + Optional description (for query help), by default None + parameters : Optional[Iterable[QueryParam]], optional + Optional list of parameter definitions, by default None. + If the query is parameterized you must supply definitions + for the parameters here - at least name and type. + Parameters can be the named tuple QueryParam (also + exposed as QueryProvider.Param) or a 4-value + + Examples + -------- + >>> qp = QueryProvider("MSSentinel") + >>> qp_host = qp.create_param("host_name", "str", "Name of Host") + >>> qp_start = qp.create_param("start", "datetime") + >>> qp_end = qp.create_param("end", "datetime") + >>> qp_evt = qp.create_param("event_id", "int", None, 4688) + >>> + >>> query = ''' + >>> SecurityEvent + >>> | where EventID == {event_id} + >>> | where TimeGenerated between (datetime({start}) .. datetime({end})) + >>> | where Computer has "{host_name}" + >>> ''' + >>> + >>> qp.add_custom_query( + >>> name="test_host_proc", + >>> query=query, + >>> family="Custom", + >>> parameters=[qp_host, qp_start, qp_end, qp_evt] + >>> ) + + """ + if parameters: + param_dict = { + param[0]: { + "type": param[1], + "default": param[2], + "description": param[3], + } + for param in parameters + } + else: + param_dict = {} + source = { + "args": {"query": query}, + "description": description, + "parameters": param_dict, + } + metadata = {"data_families": [family] if isinstance(family, str) else family} + query_source = QuerySource( + name=name, source=source, defaults={}, metadata=metadata + ) + self.query_store.add_data_source(query_source) + self._add_query_functions() + + +def _normalize_to_regex( + search_term: Union[str, Iterable[str], None], ignore_case: bool +) -> List[Pattern[str]]: + """Return iterable or str search term as list of compiled reg expressions.""" + if not search_term: + return [] + regex_opts = [re.IGNORECASE] if ignore_case else [] + if isinstance(search_term, str): + return [re.compile(search_term, *regex_opts)] + if isinstance(search_term, abc.Iterable): + return [re.compile(term, *regex_opts) for term in set(search_term)] + return [] diff --git a/msticpy/data/core/query_source.py b/msticpy/data/core/query_source.py index 3bd823210..6aa0f16c0 100644 --- a/msticpy/data/core/query_source.py +++ b/msticpy/data/core/query_source.py @@ -87,20 +87,16 @@ def __init__( that joins data from several sources) """ + self.show = True self.name = name self._source: Dict[str, Any] = source or {} self.defaults: Dict[str, Any] = defaults or {} self._global_metadata: Dict[str, Any] = dict(metadata) if metadata else {} self.query_store: Optional["QueryStore"] = None # type: ignore # noqa: F821 - # consolidate source metadata - source-specifc + # consolidate source metadata - source-specific # overrides global # add an empty dict in case neither has defined params - # self.metadata = ChainMap( - # _value_or_default(self._source, "metadata", {}), - # _value_or_default(self.defaults, "metadata", {}), - # self._global_metadata, - # ) self.metadata = collapse_dicts( self._global_metadata, self.defaults.get("metadata", {}), @@ -109,12 +105,6 @@ def __init__( # make ChainMap for parameters from with source # higher priority than default # add an empty dict in case neither has defined params - # self.params = ChainMap( - # _value_or_default(self._source, "parameters", {}), - # _value_or_default(self.defaults, "parameters", {}), - # # self._source.get("parameters", {}), - # # self.defaults.get("parameters", {}), - # ) self.params = collapse_dicts( self.defaults.get("parameters", {}), self._source.get("parameters", {}), diff --git a/msticpy/data/core/query_store.py b/msticpy/data/core/query_store.py index e4537e82c..2075c15d9 100644 --- a/msticpy/data/core/query_store.py +++ b/msticpy/data/core/query_store.py @@ -7,7 +7,7 @@ from collections import defaultdict from functools import cached_property from os import path -from typing import Any, Dict, Iterable, List, Optional, Set, Union +from typing import Any, Callable, Dict, Iterable, List, Optional, Set, Union from ..._version import VERSION from ...common.exceptions import MsticpyUserConfigError @@ -79,6 +79,7 @@ def __init__(self, environment: str): self.environment: str = environment self.data_families: Dict[str, Dict[str, QuerySource]] = defaultdict(dict) self.data_family_defaults: Dict[str, Dict[str, Any]] = defaultdict(dict) + self._all_sources: List[QuerySource] = [] def __getattr__(self, name: str): """Return the item in dot-separated path `name`.""" @@ -102,7 +103,8 @@ def query_names(self) -> Iterable[str]: for family in sorted(self.data_families): yield from [ f"{family}.{query}" - for query in sorted(self.data_families[family].keys()) + for query, query_source in sorted(self.data_families[family].items()) + if query_source.show ] @cached_property @@ -132,6 +134,7 @@ def add_data_source(self, source: QuerySource): """ source.query_store = self + self._all_sources.append(source) for family in source.data_families: self.data_families[family][source.name] = source # we want to update any new defaults for the data family @@ -218,6 +221,20 @@ def import_file(self, query_file: str): new_source = QuerySource(source_name, source, defaults, metadata) self.add_data_source(new_source) + def apply_query_filter(self, query_filter: Callable[[QuerySource], bool]): + """ + Apply a filter to the query sources. + + Parameters + ---------- + query_filter : Callable[[bool], QuerySource] + A function that takes a QuerySource and returns True + if the query should be displayed. + + """ + for source in self._all_sources: + source.show = query_filter(source) + @classmethod # noqa: MC0001 def import_files( # noqa: MC0001 cls, diff --git a/msticpy/data/core/query_template.py b/msticpy/data/core/query_template.py new file mode 100644 index 000000000..4dc909e66 --- /dev/null +++ b/msticpy/data/core/query_template.py @@ -0,0 +1,73 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""MSTICPy query template definition.""" +from dataclasses import dataclass, field +from typing import Any, Dict, List, Optional + +from ..._version import VERSION + +__version__ = VERSION +__author__ = "Ian Hellen" + + +@dataclass +class Metadata: + """Metadata for query definitions.""" + + version: int + description: str + data_environments: List[str] + data_families: List[str] + database: Optional[str] = None + cluster: Optional[str] = None + clusters: Optional[List[str]] = None + tags: List[str] = field(default_factory=list) + data_source: Optional[str] = None + + +@dataclass +class Parameter: + """Query parameter.""" + + description: str + datatype: str + default: Any = None + required: Optional[bool] = None + + +@dataclass +class Defaults: + """Default values for query definitions.""" + + metadata: Optional[Dict[str, Any]] = None + parameters: Dict[str, Parameter] = field(default_factory=dict) + + +@dataclass +class Args: + """Query arguments.""" + + query: str = "" + + +@dataclass +class Query: + """A Query definition.""" + + description: str + metadata: Dict[str, Any] = field(default_factory=dict) + args: Args = field(default_factory=Args) + parameters: Dict[str, Parameter] = field(default_factory=dict) + + +@dataclass +class QueryCollection: + """Query Collection class - a query template.""" + + file_name: str + metadata: Metadata + defaults: Optional[Defaults] = None + sources: Dict[str, Query] = field(default_factory=dict) diff --git a/msticpy/data/drivers/__init__.py b/msticpy/data/drivers/__init__.py index b0be230d4..976896d1c 100644 --- a/msticpy/data/drivers/__init__.py +++ b/msticpy/data/drivers/__init__.py @@ -31,6 +31,8 @@ DataEnvironment.M365D: ("mdatp_driver", "MDATPDriver"), DataEnvironment.Cybereason: ("cybereason_driver", "CybereasonDriver"), DataEnvironment.Elastic: ("elastic_driver", "ElasticDriver"), + DataEnvironment.MSSentinel_New: ("azure_monitor_driver", "AzureMonitorDriver"), + DataEnvironment.Kusto_New: ("azure_kusto_driver", "AzureKustoDriver"), } diff --git a/msticpy/data/drivers/azure_kusto_driver.py b/msticpy/data/drivers/azure_kusto_driver.py new file mode 100644 index 000000000..81fe4b0e0 --- /dev/null +++ b/msticpy/data/drivers/azure_kusto_driver.py @@ -0,0 +1,880 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Kusto Driver subclass.""" +import dataclasses +import json +import logging +from datetime import timedelta +from functools import partial +from typing import Any, Dict, List, NamedTuple, Optional, Tuple, Union + +import pandas as pd +from azure.kusto.data import ( + ClientRequestProperties, + KustoClient, + KustoConnectionStringBuilder, +) + +from ..._version import VERSION +from ...auth.azure_auth import az_connect, get_default_resource_name +from ...auth.cloud_mappings import AzureCloudConfig +from ...common.exceptions import ( + MsticpyDataQueryError, + MsticpyMissingDependencyError, + MsticpyNotConnectedError, + MsticpyParameterError, +) +from ...common.provider_settings import ProviderArgs, get_protected_setting +from ...common.settings import get_config, get_http_proxies +from ...common.utility import export +from ..core.query_defns import DataEnvironment +from ..core.query_source import QuerySource +from .driver_base import DriverBase, DriverProps + +try: + from azure.kusto.data.exceptions import KustoApiError, KustoServiceError + from azure.kusto.data.helpers import dataframe_from_result_table + from azure.kusto.data.response import KustoResponseDataSet +except ImportError as imp_err: + raise MsticpyMissingDependencyError( + "Cannot use this feature without Azure Kusto client installed", + title="Error importing azure.kusto.data", + packages="azure-kusto-data", + ) from imp_err + +__version__ = VERSION +__author__ = "Ian Hellen" + +_HELP_URL = "https://msticpy.readthedocs.io/en/latest/DataProviders/DataProv-Kusto.html" + +logger = logging.getLogger(__name__) + + +@dataclasses.dataclass +class KustoConfig: + """Kusto configuration class.""" + + name: str + cluster: str + alias: str + path: str + args: ProviderArgs = dataclasses.field(default_factory=ProviderArgs) + tenant_id: Optional[str] = None + integrated_auth: bool = False + cluster_groups: List[str] = dataclasses.field(default_factory=list) + + @dataclasses.dataclass + class ConfigFields: + """Kusto configuration fields.""" + + CLUSTER = "Cluster" + TENANT_ID = "TenantId" + INTEG_AUTH = "IntegratedAuth" + DEFAULTS = "ClusterDefaults" + CLIENT_ID = "ClientId" + CLIENT_SEC = "ClientSecret" + ARGS = "Args" + CLUSTER_GROUPS = "ClusterGroups" + + # pylint: disable=no-member + @property + def default_db(self): + """Return default database for this cluster.""" + return self.args.get("Database", self.args.get("DefaultDatabase")) + + # pylint: disable=unsubscriptable-object, unsupported-membership-test + def __getattr__(self, attrib): + """Return attribute from args if not in self.""" + if attrib in self.args: + return self.args[attrib] + raise AttributeError(f"Invalid attribute '{attrib}'") + + def __contains__(self, attrib): + """Return True if attribute in self or args.""" + return attrib in self.__dict__ or attrib in self.args + + +@dataclasses.dataclass +class QuerySourceFields: + """Kusto query source/yaml query fields.""" + + CLUSTER = "cluster" + CLUSTERS = "clusters" + CLUSTER_GROUPS = "cluster_groups" + DATA_ENVS = "data_environments" + DATA_FAMILIES = "data_families" + + +class AuthParams(NamedTuple): + """NamedTuple for auth parameters.""" + + method: str + params: Dict[str, Any] + uri: str + + +KFields = KustoConfig.ConfigFields + +_DEFAULT_TIMEOUT = 60 * 4 +_MAX_TIMEOUT = 60 * 60 + + +# pylint: disable=too-many-instance-attributes +@export +class AzureKustoDriver(DriverBase): + """Kusto Driver class to execute kql queries for Azure Data Explorer.""" + + def __init__(self, connection_str: Optional[str] = None, **kwargs): + """ + Instantiate KustoDriver. + + Parameters + ---------- + connection_str : str, optional + Connection string + + Other Parameters + ---------------- + debug : bool + print out additional diagnostic information. + timeout : int + Query timeout in seconds, default is 240 seconds (4 minutes) + Maximum is 3600 seconds (1 hour). + (can be set here or in connect and overridden in query methods) + proxies : Dict[str, str] + Proxy settings for log analytics queries. + Dictionary format is {protocol: proxy_url} + Where protocol is https, http, etc. and proxy_url can contain + optional authentication information in the format + "https://username:password@proxy_host:port" + If you have a proxy configuration in msticpyconfig.yaml and + you do not want to use it, set this to an empty dictionary. + (can be overridden in connect method) + + """ + super().__init__(**kwargs) + if kwargs.get("debug", False): + logger.setLevel(logging.DEBUG) + self.environment = kwargs.get("data_environment", DataEnvironment.Kusto) + self._strict_query_match = kwargs.get("strict_query_match", False) + self._kusto_settings: Dict[str, Dict[str, KustoConfig]] = _get_kusto_settings() + self._default_database: Optional[str] = None + self.current_connection: Optional[str] = connection_str + self._current_config: Optional[KustoConfig] = None + self.client: Optional[KustoClient] = None + self._az_auth_types: Optional[List[str]] = None + self._az_tenant_id: Optional[str] = None + self._def_timeout = min(kwargs.pop("timeout", _DEFAULT_TIMEOUT), _MAX_TIMEOUT) + self._def_proxies = kwargs.get("proxies", get_http_proxies()) + + self.add_query_filter("data_environments", "Kusto") + self.set_driver_property(DriverProps.PUBLIC_ATTRS, self._set_public_attribs()) + self.set_driver_property(DriverProps.FILTER_ON_CONNECT, True) + self.set_driver_property(DriverProps.EFFECTIVE_ENV, DataEnvironment.Kusto.name) + + def _set_public_attribs(self): + """Expose subset of attributes via query_provider.""" + return { + "get_database_names": self.get_database_names, + "get_database_schema": self.get_database_schema, + "configured_clusters": self.configured_clusters, + "cluster_uri": self.cluster_uri, + "cluster_name": self.cluster_name, + "cluster_config_name": self.cluster_config_name, + "set_cluster": self.set_cluster, + "set_database": self.set_database, + } + + @property + def cluster_uri(self) -> str: + """Return current cluster URI.""" + if not self._current_config: + return "" + return self._current_config.cluster + + @property + def cluster_name(self) -> str: + """Return current cluster URI.""" + if not self._current_config: + return "" + return self._current_config.name + + @property + def cluster_config_name(self) -> str: + """Return current cluster URI.""" + if isinstance(self._current_config, KustoConfig): + return self._current_config.alias + return "not defined" + + @property + def schema(self) -> Dict[str, Dict]: + """Return schema for current database.""" + try: + return self.get_database_schema() + except ValueError: + print("Default database not set - unable to retrieve schema.") + except MsticpyNotConnectedError: + print("Not connected to a cluster - unable to retrieve schema.") + except MsticpyDataQueryError: + print("Kusto Error retrieving the schema.") + return {} + + @property + def configured_clusters(self) -> Dict[str, KustoConfig]: + """Return current Kusto config settings.""" + return self._kusto_settings["id"] + + def set_cluster(self, cluster: str): + """Set the current cluster to `cluster` and connect.""" + self.connect(cluster=cluster) + + def set_database(self, database: str): + """Set the default database to `database`.""" + self._default_database = database + + def connect(self, connection_str: Optional[str] = None, **kwargs): + """ + Connect to data source. + + Either a connection string or a cluster name must be specified. + The cluster name can be a short name or a full URI. If a short name, + the cluster must be defined in the msticpyconfig.yaml file. + In this case, the short name can be either the key of the cluster + definition the host name part of the cluster URI. + + Parameters + ---------- + connection_str : str + Connect to a data source + cluster : str, optional + Short name or URI of cluster to connect to. + + Other Parameters + ---------------- + database : str, optional + Name to set the default database to. + tenant_id : str, optional + Azure tenant ID for the cluster. + connection_str : str, optional + Kusto connection string, including authentication credentials. + auth_types: Union[str, list], optional + Credential type or types to use for authentication. + Use `msticpy.auth.azure_auth.list_auth_methods()` to get a list + of available methods. + mp_az_auth : Union[bool, str, list, None], optional + Deprecated parameter to use MSTICPy Azure authentication. + Values can be: + True or "default": use the settings in msticpyconfig.yaml 'Azure' section + str: single auth method name + List[str]: list of acceptable auth methods + mp_az_tenant_id: str, optional + alias for `tenant_id`. + timeout : int + Query timeout in seconds, default is 240 seconds (4 minutes) + Maximum is 3600 seconds (1 hour). + (can be overridden in query methods) + + See Also + -------- + msticpy.auth.azure_auth.list_auth_methods + + """ + logger.info( + "Connecting to Kusto cluster: connection_str=%s, args=%s", + connection_str, + kwargs, + ) + self._default_database = kwargs.pop("database", None) + self._def_timeout = min(kwargs.pop("timeout", self._def_timeout), _MAX_TIMEOUT) + az_auth_types = kwargs.pop("auth_types", kwargs.pop("mp_az_auth", None)) + if isinstance(az_auth_types, bool): + self._az_auth_types = None + elif isinstance(az_auth_types, str): + self._az_auth_types = [az_auth_types] + else: + self._az_auth_types = az_auth_types + self._az_tenant_id = kwargs.pop( + "tenant_id", kwargs.pop("mp_az_tenant_id", None) + ) + + cluster = kwargs.pop("cluster", None) + if not connection_str and not cluster: + raise MsticpyParameterError( + "Must specify either a connection string or a cluster name", + parameter=["connection_str", "cluster"], + ) + + if cluster: + self._current_config = self._lookup_cluster_settings(cluster) + logger.info( + "Using cluster id: %s, retrieved %s", + cluster, + self.cluster_uri, + ) + kusto_cs = self._get_connection_string_for_cluster(self._current_config) + else: + logger.info("Using connection string %s", connection_str) + kusto_cs = connection_str + + self.client = KustoClient(kusto_cs) + proxies = kwargs.get("proxies", self._def_proxies) + proxy_url = proxies.get("https") if proxies else None + if proxy_url: + logger.info( + "Using proxy: %s", + proxy_url + if "@" not in proxy_url # don't log proxy credentials + else "****" + proxy_url.split("@")[-1], + ) + self.client.set_proxy(proxy_url) + self._connected = True + + def query( + self, query: str, query_source: Optional[QuerySource] = None, **kwargs + ) -> Union[pd.DataFrame, Any]: + """ + Execute query string and return DataFrame of results. + + Parameters + ---------- + query : str + The query to execute + query_source : QuerySource + The query definition object + + Other Parameters + ---------------- + database : str, Optional + Supply or override the Kusto database name + timeout : int + Query timeout in seconds, default is 240 seconds (4 minutes) + Maximum is 3600 seconds (1 hour). + + Returns + ------- + Union[pd.DataFrame, results.ResultSet] + A DataFrame (if successful) or + the underlying provider result if an error. + + """ + data, result = self.query_with_results( + query, query_source=query_source, **kwargs + ) + return data if data is not None else result + + def query_with_results( + self, query: str, **kwargs + ) -> Tuple[Optional[pd.DataFrame], Any]: + """ + Return query results as a DataFrame and the result status. + + Parameters + ---------- + query : str + The query string + + Returns + ------- + Tuple[Optional[pd.DataFrame], Any] + DataFrame of results and the result status. + + Raises + ------ + MsticpyNotConnectedError + If there is no connection to the data source. + MsticpyDataQueryError + If no database is specified in the query or parameters + and there is no default database. + + """ + if not self._connected: + _raise_not_connected_error() + query_source = kwargs.pop("query_source", None) + + if query_source and not self.query_usable(query_source): + query_spec = self._get_cluster_spec_from_query_source(query_source) + raise MsticpyDataQueryError( + "Invalid query source - for this connection.", + f"Connected cluster is: {self.cluster_uri} ({self.cluster_config_name})", + "The cluster in the query definition is:", + *[f"{name}: {value}" for name, value in query_spec.items()], + title="Mismatched cluster for query.", + help_uri=_HELP_URL, + ) + + database = self._get_query_database_name(query_source=query_source, **kwargs) + data: Optional[pd.DataFrame] = None + status = {"success": False} + connection_props = ClientRequestProperties() + connection_props.set_option( + ClientRequestProperties.request_timeout_option_name, + timedelta(seconds=kwargs.get("timeout", self._def_timeout)), + ) + + try: + logger.info("Query executed query=%s, database=%s", query, database) + response = self.client.execute( # type: ignore[union-attr] + database=database, query=query, properties=connection_props + ) + data = dataframe_from_result_table(response.primary_results[0]) + status = _parse_query_status(response) + logger.info("Query completed: %s", str(status)) + except KustoApiError as err: + logger.exception("Query failed: %s", err) + _raise_kusto_error(err) + except KustoServiceError as err: + logger.exception("Query failed: %s", err) + _raise_unknown_query_error(err) + return data, status + + def get_database_names(self) -> List[str]: + """Get a list of database names from the connected cluster.""" + if self.client is None: + _raise_not_connected_error() + try: + connection_props = ClientRequestProperties() + connection_props.set_option( + ClientRequestProperties.request_timeout_option_name, + timedelta(seconds=self._def_timeout), + ) + logger.info("Get database names cluster: %s", self.cluster_uri) + response = self.client.execute_mgmt( # type: ignore[union-attr] + database="NetDefaultDB", + query=".show databases", + properties=connection_props, + ) + + # Convert the result to a DataFrame + databases_df = dataframe_from_result_table(response.primary_results[0]) + return databases_df["DatabaseName"].tolist() + except KustoServiceError as err: + raise MsticpyDataQueryError( + "Error getting database names", + err, + title="Kusto error", + help_uri=_HELP_URL, + ) from err + + def get_database_schema( + self, database: Optional[str] = None + ) -> Dict[str, Dict[str, str]]: + """ + Get table names and schema from the connected cluster/database. + + Parameters + ---------- + database : Optional[str] + Name of the database to get schema for. + The default is the last connected database. + + Returns + ------- + Dict[str, Dict[str, str]] + Dictionary of table names, each with a dictionary of + column names and types. + + Raises + ------ + ValueError : + No database name specified or set as the default. + MsticpyNotConnectedError : + Not connected to a cluster. + MsticpyDataQueryError : + Error querying the cluster. + + """ + db_name = database or self._default_database + if self.client is None: + _raise_not_connected_error() + if not db_name: + raise ValueError("No database name specified") + + query = f".show database {db_name} schema" + try: + # Execute the query + logger.info("Get database schema: %s", db_name) + response = self.client.execute_mgmt(db_name, query) # type: ignore[union-attr] + # Convert the result to a DataFrame + schema_dataframe = dataframe_from_result_table(response.primary_results[0]) + except KustoServiceError as err: + raise MsticpyDataQueryError( + "Error getting database schema", + err, + title="Kusto error", + help_uri=_HELP_URL, + ) from err + + return { + str(table): { + col_name: col_type.replace("System.", "") + for col_name, col_type in cols[["ColumnName", "ColumnType"]].values + if col_name is not None + } + for table, cols in schema_dataframe.groupby("TableName") + } + + def _get_cluster_spec_from_query_source( + self, query_source: QuerySource + ) -> Dict[str, str]: + """Return cluster details from query source.""" + return { + QuerySourceFields.CLUSTER: query_source.metadata.get( + QuerySourceFields.CLUSTER, "NA" + ), + QuerySourceFields.CLUSTERS: query_source.metadata.get( + QuerySourceFields.CLUSTERS, "NA" + ), + QuerySourceFields.CLUSTER_GROUPS: query_source.metadata.get( + QuerySourceFields.CLUSTER_GROUPS, "NA" + ), + } + + def _get_connection_string_for_cluster( + self, cluster_config: KustoConfig + ) -> KustoConnectionStringBuilder: + """Return full cluster URI and credential for cluster name or URI.""" + auth_params = self._get_auth_params_from_config(cluster_config) + if auth_params.method == "clientsecret": + connect_auth_types = self._az_auth_types or AzureCloudConfig().auth_methods + if "clientsecret" not in connect_auth_types: + connect_auth_types.append("clientsecret") + credential = az_connect( + auth_types=connect_auth_types, **(auth_params.params) + ) + else: + credential = az_connect( + auth_types=self._az_auth_types, **(auth_params.params) + ) + logger.info("Credentials obtained %s", type(credential.modern).__name__) + token = credential.modern.get_token(get_default_resource_name(auth_params.uri)) + logger.info("Token obtained for %s", auth_params.uri) + return KustoConnectionStringBuilder.with_aad_user_token_authentication( + connection_string=auth_params.uri, + user_token=token.token, + ) + + def _get_auth_params_from_config(self, cluster_config: KustoConfig) -> AuthParams: + """Get authentication parameters for cluster from KustoConfig values.""" + method = "integrated" + auth_params_dict = {} + if KFields.CLIENT_SEC in cluster_config and KFields.CLIENT_ID in cluster_config: + method = "clientsecret" + auth_params_dict["client_id"] = cluster_config.ClientId + auth_params_dict["client_secret"] = cluster_config.ClientSecret + logger.info( + "Using client secret authentication because client_secret in config" + ) + elif KFields.INTEG_AUTH in cluster_config: + logger.info("Using integrated auth.") + auth_params_dict["tenant_id"] = cluster_config.tenant_id + return AuthParams(method, auth_params_dict, cluster_config.cluster) + + def _lookup_cluster_settings(self, cluster: str) -> KustoConfig: + """Return cluster URI from config if cluster name is passed.""" + cluster_key = cluster.casefold().strip() + if cluster_key in self._kusto_settings["url"]: + return self._kusto_settings["url"][cluster_key] + if cluster_key in self._kusto_settings["name"]: + return self._kusto_settings["name"][cluster_key] + if cluster_key in self._kusto_settings["id"]: + return self._kusto_settings["id"][cluster_key] + if cluster_key.startswith("https://"): + return KustoConfig( + cluster=cluster, + name=cluster.replace("https://", "").split(".")[0], + alias="no_config_found", + path="", + tenant_id=self._az_tenant_id, + ) + + raise MsticpyDataQueryError( + f"Cluster '{cluster}' not found in msticpyconfig.yaml", + "or is not in the correct format for a a cluster URI", + "The cluster must be a key, cluster short name of an entry defined", + "in the 'KustoClusters' section of the config file,", + "or it must be a valid cluster URI.", + title="Unusable cluster identifier", + help_uri=_HELP_URL, + ) + + def _get_query_database_name( + self, query_source: Optional[QuerySource] = None, **kwargs + ) -> str: + """Get the database name from query source or kwargs.""" + if database := kwargs.get("database"): + logger.info("Using database %s from parameter.", database) + return database + if query_source: + return self._get_db_from_query_source(query_source) + # check if database is specified in the current config + if isinstance(self._current_config, KustoConfig): + database = self._current_config.default_db + if database: + logger.info("Using database %s from current config.", database) + return database + if self._default_database: + logger.info("Using database %s from _default_database.", database) + return self._default_database + _raise_no_db_error() + return "" # pragma: no cover + + def query_usable(self, query_source) -> bool: + """Return True if query source is valid for current cluster.""" + if not query_source or not isinstance(query_source, QuerySource): + return False + if not self._current_config: + # This probably won't work but we can just try the query + # and see if it works + return True + # Check if query source has cluster information + if not ( + query_source.metadata.keys() + & { + QuerySourceFields.CLUSTER_GROUPS, + QuerySourceFields.CLUSTERS, + QuerySourceFields.CLUSTER, + } + ): + if self._strict_query_match: + return False + logger.info( + ( + "Query source %s has no cluster information. " + "The query may fail on the current cluster." + ), + query_source.name, + ) + return True + # Check for matches on cluster groups or cluster id + result = False + result |= self._cluster_groups_match(query_source) + if result: + return result + result |= self._cluster_id_matches(query_source) + return result + + def _cluster_groups_match(self, query_source: QuerySource) -> bool: + """Return True if query source cluster group is valid for current cluster.""" + source_cluster_groups = query_source.metadata.get( + QuerySourceFields.CLUSTER_GROUPS, [] + ) + if ( + source_cluster_groups + and self._current_config.cluster_groups # type: ignore[union-attr] + ): + driver_groups = { + group.casefold() + for group in self._current_config.cluster_groups # type: ignore[union-attr] + } + query_groups = {group.casefold() for group in source_cluster_groups} + return bool(driver_groups.intersection(query_groups)) + return False + + def _cluster_id_matches(self, query_source: QuerySource) -> bool: + """Return True if query source cluster is valid for current cluster.""" + # Get different representations of the cluster name + result = False + cluster_ids = { + self.cluster_uri.casefold(), + self.cluster_name.casefold(), + self.cluster_config_name.casefold(), + } + source_clusters = query_source.metadata.get(QuerySourceFields.CLUSTERS, []) + if source_clusters: + query_source_clusters = {cluster.casefold() for cluster in source_clusters} + result |= bool(cluster_ids.intersection(query_source_clusters)) + if result: + return result + source_cluster = query_source.metadata.get(QuerySourceFields.CLUSTER) + if source_cluster: + result |= source_cluster.casefold() in cluster_ids + return result + + @staticmethod + def _get_db_from_query_source(query_source: QuerySource) -> str: + """Get the database name from query source metadata.""" + if database := query_source.metadata.get("database"): + return database + data_families = query_source.metadata.get("data_families") + if not data_families: + logger.info("Could not find database name in query source metadata.") + _raise_no_db_error(query_source) + + if "." in data_families[0]: # type: ignore + _, database = data_families[0].split(".", maxsplit=1) # type: ignore + else: + # Not expected but we can still use a DB value with no dot + database = data_families[0] # type: ignore + logger.info("Using database %s from query source metadata.", database) + return database + + +def _get_kusto_settings() -> Dict[str, Dict[str, KustoConfig]]: + """Return a dictionary of Kusto cluster settings from msticpyconfig.yaml.""" + kusto_config = { + kusto_entry: kusto_config + for kusto_entry, kusto_config in get_config("DataProviders", {}).items() + if kusto_entry.startswith("Kusto") + } + kusto_clusters = {} + # handle legacy configuration + for config_id, cluster_conf in kusto_config.items(): + name = config_id.replace("Kusto-", "") + kusto_clusters[name] = cluster_conf + kusto_clusters[name]["path"] = f"DataProviders.{config_id}" + + kusto_new_conf = { + config_id: {**cluster_conf, "path": f"KustoClusters.{config_id}"} + for config_id, cluster_conf in get_config("KustoClusters", {}).items() + } + defaults: Dict[str, Any] = kusto_new_conf.pop(KFields.DEFAULTS, {}).get( + KFields.ARGS, {} # type: ignore[assignment] + ) + kusto_clusters.update(kusto_new_conf) + + cluster_by_url = _create_cluster_config( + kusto_clusters=kusto_clusters, defaults=defaults + ) + return { + "url": cluster_by_url, + "id": {conf.alias.casefold(): conf for conf in cluster_by_url.values()}, + "name": {conf.name.casefold(): conf for conf in cluster_by_url.values()}, + } + + +def _create_cluster_config( + kusto_clusters: Dict[str, Any], defaults: Dict[str, Any] +) -> Dict[str, KustoConfig]: + """Return a dictionary of Kusto cluster settings from msticpyconfig.yaml.""" + return { + config[KFields.ARGS] + .get(KFields.CLUSTER) + .casefold(): KustoConfig( + tenant_id=_setting_or_default( + config[KFields.ARGS], KFields.TENANT_ID, defaults + ), + integrated_auth=_setting_or_default( + config[KFields.ARGS], KFields.INTEG_AUTH, defaults + ) + or False, + args=_create_protected_args( + _section_or_default(config[KFields.ARGS], defaults), config["path"] + ), + cluster=config[KFields.ARGS].get(KFields.CLUSTER), + alias=name, + name=get_cluster_name(config[KFields.ARGS].get(KFields.CLUSTER)), + path=config["path"], + cluster_groups=config.get(KFields.CLUSTER_GROUPS), + ) + for name, config in kusto_clusters.items() + } + + +def _setting_or_default(settings: Dict[str, Any], name: str, default: Dict[str, Any]): + """Return a setting from the settings dictionary or the default.""" + return settings.get(name, default.get(name)) + + +def _section_or_default(settings: Dict[str, Any], default: Dict[str, Any]): + """Return a combined dictionary from the settings dictionary or the default.""" + return { + key: settings.get(key, default.get(key)) + for key in (settings.keys() | default.keys()) + } + + +def _create_protected_args(args: Dict[str, Any], path: str) -> ProviderArgs: + """Return a dictionary of protected settings for Kusto args config.""" + args_dict = { + key_name: partial( + get_protected_setting, config_path=f"{path}.Args", setting_name=key_name + ) + if isinstance(value, dict) + and (value.get("EnvironmentVar") or value.get("KeyVault")) + else value + for key_name, value in args.items() + } + return ProviderArgs(**args_dict) + + +def get_cluster_name(cluster_uri): + """Return the cluster name from the cluster uri.""" + return cluster_uri.replace("https://", "").split(".")[0] + + +def _parse_query_status(response: KustoResponseDataSet) -> Dict[str, Any]: + """Parse the query status from the Kusto response.""" + try: + query_info_idx = response.tables_names.index("QueryCompletionInformation") + except ValueError: + return { + "status": "Failed", + "message": "QueryCompletionInformation not found in response", + } + + df_status = dataframe_from_result_table(response.tables[query_info_idx]) + results = df_status[["EventTypeName", "Payload"]].to_dict(orient="records") + return { + row.get("EventTypeName", "Unknown_field"): json.loads( + row.get("Payload", "No Payload") + ) + for row in results + } + + +def _raise_kusto_error(error): + """Raise a Kusto error.""" + if isinstance(error, KustoApiError): + raise MsticpyDataQueryError( + error.error.description, + f"error code: {error.error.code}", + title=error.error.message, + help_uri=_HELP_URL, + ) from error + + +def _raise_no_db_error(query_source=None): + """Raise an error if no database is specified.""" + if query_source: + messages = ( + "No database found in the query definition", + ( + "Correct the query definition or use the 'database' parameter" + "or set the default database when connecting to the cluster." + ), + ) + else: + messages = ( + ( + "No database specified. Use the 'database' parameter or set the " + "default database when connecting to the cluster." + ), + ) + raise MsticpyDataQueryError( + *messages, + title="No database specified", + help_uri=_HELP_URL, + ) + + +def _raise_not_connected_error(): + """Raise an error if not connected to a cluster.""" + raise MsticpyNotConnectedError( + "Please connect to the cluster before executing a query.", + title="Not connected to cluster", + help_uri=_HELP_URL, + ) + + +def _raise_unknown_query_error(err): + """Raise an error if unknown exception raised.""" + raise MsticpyDataQueryError( + "Unknown exception when executing query.", + f"Exception type: {type(err)}", + *err.args, + title="Unknown exception during query execution", + help_uri=_HELP_URL, + ) from err diff --git a/msticpy/data/drivers/azure_monitor_driver.py b/msticpy/data/drivers/azure_monitor_driver.py new file mode 100644 index 000000000..9d9b4d620 --- /dev/null +++ b/msticpy/data/drivers/azure_monitor_driver.py @@ -0,0 +1,608 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +""" +Azure monitor/Log Analytics KQL Driver class. + +See Also +-------- +Azure SDK code: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/monitor + +Azure SDK docs: https://learn.microsoft.com/python/api/overview/ +azure/monitor-query-readme?view=azure-python + +""" +import logging +from datetime import datetime +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union, cast + +import httpx +import pandas as pd +from azure.core.exceptions import HttpResponseError +from azure.core.pipeline.policies import UserAgentPolicy + +from ..._version import VERSION +from ...auth.azure_auth import AzureCloudConfig, az_connect +from ...common.exceptions import ( + MsticpyDataQueryError, + MsticpyKqlConnectionError, + MsticpyMissingDependencyError, + MsticpyNoDataSourceError, + MsticpyNotConnectedError, +) +from ...common.settings import get_http_proxies, get_http_timeout +from ...common.timespan import TimeSpan +from ...common.utility import export, mp_ua_header +from ...common.wsconfig import WorkspaceConfig +from ..core.query_defns import DataEnvironment +from .driver_base import DriverBase, DriverProps, QuerySource + +logger = logging.getLogger(__name__) + +try: + from azure.monitor.query import ( + LogsQueryClient, + LogsQueryPartialResult, + LogsQueryResult, + ) +except ImportError as imp_err: + raise MsticpyMissingDependencyError( + "Cannot use this feature without Azure monitor client installed", + title="Error importing azure.monitor.query", + packages="azure-monitor-query", + ) from imp_err + +__version__ = VERSION +__author__ = "Ian Hellen" + + +_KQL_CLOUD_MAP = { + "global": "public", + "cn": "china", + "usgov": "government", + "de": "germany", +} + +_LOGANALYTICS_URL_BY_CLOUD = { + "global": "https://api.loganalytics.io/", + "cn": "https://api.loganalytics.azure.cn/", + "usgov": "https://api.loganalytics.us/", + "de": "https://api.loganalytics.de/", +} + + +_HELP_URL = ( + "https://msticpy.readthedocs.io/en/latest/data_acquisition/DataProv-MSSentinel.html" +) +# pylint: disable=too-many-instance-attributes + + +@export +class AzureMonitorDriver(DriverBase): + """KqlDriver class to execute kql queries.""" + + _DEFAULT_TIMEOUT = 300 + + def __init__(self, connection_str: Optional[str] = None, **kwargs): + """ + Instantiate KqlDriver and optionally connect. + + Parameters + ---------- + connection_str : str, optional + Connection string + + Other Parameters + ---------------- + debug : bool + print out additional diagnostic information. + timeout : int (seconds) + Specify a timeout for queries. Default is 300 seconds. + (can be set here or in connect and overridden in query methods) + proxies : Dict[str, str] + Proxy settings for log analytics queries. + Dictionary format is {protocol: proxy_url} + Where protocol is https, http, etc. and proxy_url can contain + optional authentication information in the format + "https://username:password@proxy_host:port" + If you have a proxy configuration in msticpyconfig.yaml and + you do not want to use it, set this to an empty dictionary. + (can be overridden in connect method) + + """ + if kwargs.get("debug", False): + logger.setLevel(logging.DEBUG) + super().__init__(**kwargs) + + self._schema: Dict[str, Any] = {} + self.set_driver_property( + DriverProps.FORMATTERS, + {"datetime": self._format_datetime, "list": self._format_list}, + ) + self._loaded = True + self._ua_policy = UserAgentPolicy(user_agent=mp_ua_header()["UserAgent"]) + self._def_timeout = kwargs.get( + "timeout", kwargs.get("server_timeout", self._DEFAULT_TIMEOUT) + ) + self._def_proxies = kwargs.get("proxies", get_http_proxies()) + self._query_client: Optional[LogsQueryClient] = None + self._az_tenant_id: Optional[str] = None + self._ws_config: Optional[WorkspaceConfig] = None + self._workspace_id: Optional[str] = None + self._workspace_ids: List[str] = [] + self._def_connection_str: Optional[str] = connection_str + self._connect_auth_types: Optional[List[str]] = None + self.add_query_filter( + "data_environments", ("MSSentinel", "LogAnalytics", "AzureSentinel") + ) + self.set_driver_property( + DriverProps.EFFECTIVE_ENV, DataEnvironment.MSSentinel.name + ) + logger.info( + "AzureMonitorDriver loaded. connect_str %s, kwargs: %s", + connection_str, + kwargs, + ) + + @property + def url_endpoint(self) -> str: + """Return the current URL endpoint for Azure Monitor.""" + return _LOGANALYTICS_URL_BY_CLOUD.get( + AzureCloudConfig().cloud, _LOGANALYTICS_URL_BY_CLOUD["global"] + ) + + def connect(self, connection_str: Optional[str] = None, **kwargs): + """ + Connect to data source. + + Parameters + ---------- + connection_str : Union[str, WorkspaceConfig, None] + Connection string or WorkspaceConfig for the Sentinel Workspace. + + Other Parameters + ---------------- + auth_types: Iterable [str] + Authentication (credential) types to use. By default the + values configured in msticpyconfig.yaml are used. If not set, + it will use the msticpy defaults. + mp_az_auth : Union[bool, str, list, None], optional + Deprecated parameter directing driver to use MSTICPy Azure authentication. + Values can be: + True or "default": use the settings in msticpyconfig.yaml 'Azure' section + str: single auth method name + List[str]: list of acceptable auth methods from + Use `auth_types` parameter instead. + tenant_id: str, optional + Optional parameter specifying a Tenant ID for use by MSTICPy Azure + authentication. By default, the tenant_id for the workspace. + workspace : str, optional + Alternative to supplying a WorkspaceConfig object as the connection_str + parameter. Giving a workspace name will fetch the workspace + settings from msticpyconfig.yaml. + workspaces : Iterable[str], optional + List of workspaces to run the queries against, each workspace name + must have an entry in msticpyconfig.yaml + workspace_ids: Iterable[str], optional + List of workspace IDs to run the queries against. Must be supplied + along with a `tenant_id`. + timeout : int (seconds) + Specify a timeout for queries. Default is 300 seconds. + (can be overridden query method) + proxies : Dict[str, str] + Proxy settings for log analytics queries. + Dictionary format is {protocol: proxy_url} + Where protocol is https, http, etc. and proxy_url can contain + optional authentication information in the format + "https://username:password@proxy_host:port" + If you have a proxy configuration in msticpyconfig.yaml and + you do not want to use it, set this to an empty dictionary. + + Notes + ----- + When using the `workspaces` or `workspace_ids` parameters, some + functionality will be reduced - e.g. no schema will be available for + the workspaces. As an alternative to using multiple workspaces here + you can create multiple workspace connections + + """ + self._connected = False + self._query_client = self._create_query_client(connection_str, **kwargs) + + # get the schema + self._schema = self._get_schema() + self._connected = True + print("connected") + + return self._connected + + # pylint: disable=too-many-branches + + @property + def schema(self) -> Dict[str, Dict]: + """ + Return current data schema of connection. + + Returns + ------- + Dict[str, Dict] + Data schema of current connection. + + """ + return self._schema + + def query( + self, query: str, query_source: Optional[QuerySource] = None, **kwargs + ) -> Union[pd.DataFrame, Any]: + """ + Execute query string and return DataFrame of results. + + Parameters + ---------- + query : str + The query to execute + query_source : Optional[QuerySource] + The query definition object + + Other Parameters + ---------------- + timeout : int (seconds) + Specify a timeout for the query. Default is 300 seconds. + + Returns + ------- + Union[pd.DataFrame, results.ResultSet] + A DataFrame (if successful) or + the underlying provider result if an error. + + """ + if not self._connected or self._query_client is None: + raise MsticpyNotConnectedError( + "Please run connect() to connect to the workspace", + "before running a query.", + title="Workspace not connected.", + help_uri=_HELP_URL, + ) + if query_source: + self._check_table_exists(query_source) + data, result = self.query_with_results(query, **kwargs) + return data if data is not None else result + + # pylint: disable=too-many-branches + def query_with_results( + self, query: str, **kwargs + ) -> Tuple[pd.DataFrame, Dict[str, Any]]: + """ + Execute query string and return DataFrame of results. + + Parameters + ---------- + query : str + The kql query to execute + + Returns + ------- + Tuple[pd.DataFrame, Dict[str, Any]] + A DataFrame (if successful) and + Query status dictionary. + + """ + if not self._connected or self._query_client is None: + raise MsticpyNotConnectedError( + "Please run connect() to connect to the workspace", + "before running a query.", + title="Workspace not connected.", + help_uri=_HELP_URL, + ) + logger.info("Query to run %s", query) + time_span_value = self._get_time_span_value(**kwargs) + + server_timeout = kwargs.pop("timeout", self._def_timeout) + + workspace_id = next(iter(self._workspace_ids), None) or self._workspace_id + additional_workspaces = self._workspace_ids[1:] if self._workspace_ids else None + try: + result = self._query_client.query_workspace( + workspace_id=workspace_id, # type: ignore[arg-type] + query=query, + timespan=time_span_value, # type: ignore[arg-type] + server_timeout=server_timeout, + additional_workspaces=additional_workspaces, + ) + except HttpResponseError as http_err: + result = None + self._raise_query_failure(query, http_err) + # We might get an unknown exception type from azure.monitor.query + except Exception as unknown_err: # pylint: disable=broad-except + result = None + self._raise_unknown_error(unknown_err) + result = cast(LogsQueryResult, result) + status = self._get_query_status(result) + logger.info("query status %s", repr(status)) + + if isinstance(result, LogsQueryPartialResult): + table = result.partial_data[0] # type: ignore[attr-defined] + else: + table = result.tables[0] # type: ignore[attr-defined] + data_frame = pd.DataFrame(table.rows, columns=table.columns) + logger.info("Dataframe returned with %d rows", len(data_frame)) + return data_frame, status + + def _create_query_client(self, connection_str, **kwargs): + """Create the query client.""" + az_auth_types = kwargs.get("auth_types", kwargs.get("mp_az_auth")) + if isinstance(az_auth_types, bool): + az_auth_types = None + if isinstance(az_auth_types, str): + az_auth_types = [az_auth_types] + self._connect_auth_types = az_auth_types + + self._def_timeout = kwargs.get("timeout", self._DEFAULT_TIMEOUT) + self._def_proxies = kwargs.get("proxies", self._def_proxies) + self._get_workspaces(connection_str, **kwargs) + + credentials = az_connect( + auth_methods=az_auth_types, tenant_id=self._az_tenant_id + ) + logger.info( + "Created query client. Auth type: %s, Url: %s, Proxies: %s", + type(credentials.modern) if credentials else "None", + self.url_endpoint, + kwargs.get("proxies", self._def_proxies), + ) + return LogsQueryClient( + credential=credentials.modern, + endpoint=self.url_endpoint, + proxies=kwargs.get("proxies", self._def_proxies), + ) + + def _get_workspaces(self, connection_str: Optional[str] = None, **kwargs): + """Get workspace or workspaces to connect to.""" + self._az_tenant_id = kwargs.get("tenant_id", kwargs.get("mp_az_tenant_id")) + # multiple workspace IDs + if workspaces := kwargs.pop("workspaces", None): + self._get_workspaces_by_name(workspaces) + return + if workspace_ids := kwargs.pop("workspace_ids", None): + self._get_workspaces_by_id(workspace_ids) + return + + # standard - single-workspace configuration + workspace_name = kwargs.get("workspace") + ws_config: Optional[WorkspaceConfig] = None + connection_str = connection_str or self._def_connection_str + if workspace_name or connection_str is None: + ws_config = WorkspaceConfig(workspace=workspace_name) # type: ignore + logger.info( + "WorkspaceConfig created from workspace name %s", workspace_name + ) + elif isinstance(connection_str, str): + self._def_connection_str = connection_str + ws_config = WorkspaceConfig.from_connection_string(connection_str) + logger.info( + "WorkspaceConfig created from connection_str %s", connection_str + ) + elif isinstance(connection_str, WorkspaceConfig): + logger.info("WorkspaceConfig as parameter %s", connection_str.workspace_id) + ws_config = connection_str + + if not ws_config: + logger.warning("No workspace set") + raise MsticpyKqlConnectionError( + "A workspace name, config or connection string is needed" + " to connect to a workspace.", + title="No connection details", + help_uri=_HELP_URL, + ) + if ws_config.workspace_id is None or ws_config.tenant_id is None: + logger.warning("Unable to get workspace ID or tenant ID") + raise MsticpyKqlConnectionError( + "The workspace config or connection string did not have" + "the required parameters to connect to a workspace.", + "At least a workspace ID and tenant ID are required.", + title="No connection details", + help_uri=_HELP_URL, + ) + self._ws_config = ws_config + if not self._az_tenant_id and WorkspaceConfig.CONF_TENANT_ID_KEY in ws_config: + self._az_tenant_id = ws_config[WorkspaceConfig.CONF_TENANT_ID_KEY] + self._workspace_id = ws_config[WorkspaceConfig.CONF_WS_ID_KEY] + + def _get_workspaces_by_id(self, workspace_ids): + if not self._az_tenant_id: + raise MsticpyKqlConnectionError( + "You must supply a tenant_id with the workspace_ids parameter", + title="No tenant_id supplied.", + help_uri=_HELP_URL, + ) + self._workspace_ids = workspace_ids + logger.info( + "%d configured workspaces: %s", + len(self._workspace_ids), + ", ".join(self._workspace_ids), + ) + + def _get_workspaces_by_name(self, workspaces): + workspace_configs = { + WorkspaceConfig(workspace)[WorkspaceConfig.CONF_WS_ID_KEY]: WorkspaceConfig( + workspace + )[WorkspaceConfig.CONF_TENANT_ID_KEY] + for workspace in workspaces + } + if len(set(workspace_configs.values())) > 1: + raise ValueError("All workspaces must have the same tenant ID.") + self._az_tenant_id = next(iter(workspace_configs.values())) + self._workspace_ids = list(set(workspace_configs)) + logger.info( + "%d configured workspaces: %s", + len(self._workspace_ids), + ", ".join(self._workspace_ids), + ) + + def _get_time_span_value(self, **kwargs): + """Return the timespan for the query API call.""" + default_time_params = kwargs.get("default_time_params", False) + time_params = kwargs.get("time_span", {}) + if ( + default_time_params + or "start" not in time_params + or "end" not in time_params + ): + time_span_value = None + logger.info("No time parameters supplied.") + else: + time_span = TimeSpan( + start=time_params["start"], + end=time_params["end"], + ) + time_span_value = time_span.start, time_span.end + logger.info("Time parameters set %s", str(time_span)) + return time_span_value + + def _check_table_exists(self, query_source): + """Check that query table is in the workspace schema.""" + if not self.schema: + return + try: + table = query_source.params.get("table", {}).get("default") + except KeyError: + table = None + if table: + if " " in table.strip(): + table = table.strip().split(" ")[0] + if table not in self.schema: + raise MsticpyNoDataSourceError( + f"The table {table} for this query is not in your workspace", + "or database schema. Please check your query.", + title=f"{table} not found.", + help_uri=_HELP_URL, + ) + + @staticmethod + def _get_query_status(result) -> Dict[str, Any]: + status = { + "status": result.status.name, + "tables": len(result.tables) if result.tables else 0, + } + + if isinstance(result, LogsQueryPartialResult): + status["partial"] = "partial results returned" + status["tables"] = (len(result.partial_data) if result.partial_data else 0,) + return status + + def _get_schema(self) -> Dict[str, Dict]: + """Return the workspace schema.""" + if not self._ws_config: + logger.info("No workspace config - cannot get schema") + return {} + mgmt_endpoint = AzureCloudConfig().endpoints.resource_manager + + url_tables = ( + "{endpoint}subscriptions/{sub_id}/resourcegroups/" + "{res_group}/providers/Microsoft.OperationalInsights/workspaces/" + "{ws_name}/tables?api-version=2021-12-01-preview" + ) + try: + ws_name = self._ws_config.workspace_name + except AttributeError: + ws_name = self._ws_config.workspace_key + + if not ws_name or ws_name == "Default": + logger.info("No workspace name - cannot get schema") + return {} + try: + fmt_url = url_tables.format( + endpoint=mgmt_endpoint, + sub_id=self._ws_config.subscription_id, + res_group=self._ws_config.resource_group, + ws_name=self._ws_config.workspace_name, + ) + except AttributeError: + logger.info("Not all workspace config available - cannot get schema") + return {} + + credentials = az_connect( + auth_methods=self._connect_auth_types, tenant_id=self._az_tenant_id + ) + token = credentials.modern.get_token(f"{mgmt_endpoint}/.default") + headers = {"Authorization": f"Bearer {token.token}", **mp_ua_header()} + logger.info("Schema request to %s", fmt_url) + response = httpx.get( + fmt_url, + headers=headers, + timeout=get_http_timeout(), + proxies=self._def_proxies or {}, + ) + if response.status_code != 200: + logger.info("Schema request failed. Status code: %d", response.status_code) + return {} + tables = response.json() + logger.info( + "Schema retrieved from workspace. %d tables found.", + len(tables.get("value", 0)), + ) + return _schema_format_tables(tables) + + @staticmethod + def _format_datetime(date_time: datetime) -> str: + """Return datetime-formatted string.""" + return date_time.isoformat(sep="T") + "Z" + + @staticmethod + def _format_list(param_list: Iterable[Any]): + """Return formatted list parameter.""" + fmt_list = [] + for item in param_list: + if isinstance(item, str): + fmt_list.append(f"'{item}'") + else: + fmt_list.append(f"{item}") + return ", ".join(fmt_list) + + @staticmethod + def _raise_query_failure(query, http_err): + """Raise query failure exception.""" + err_contents = [] + if hasattr(http_err, "message"): + err_contents = http_err.message.split("\n") + if not err_contents: + err_contents = ["Unknown query error"] + + err_contents.append(f"Query:\n{query}") + raise MsticpyDataQueryError( + *err_contents, title="Query Failure", help_uri=_HELP_URL + ) from http_err + + @staticmethod + def _raise_unknown_error(exception): + """Raise an unknown exception.""" + raise MsticpyDataQueryError( + "An unknown exception was returned by the service", + *exception.args, + f"Full exception:\n{exception}", + title="connection failed", + help_uri=_HELP_URL, + ) from exception + + +def _schema_format_tables( + ws_tables: Dict[str, Iterable[Dict[str, Any]]] +) -> Dict[str, Dict[str, str]]: + """Return a sorted dictionary of table names and column names/types.""" + table_schema = { + table["name"]: _schema_format_columns(table["properties"]["schema"]) + for table in ws_tables["value"] + } + return dict(sorted(table_schema.items())) + + +def _schema_format_columns(table_schema: Dict[str, Any]) -> Dict[str, str]: + """Return a sorted dictionary of column names and types.""" + columns = { + col["name"]: col["type"] for col in table_schema.get("standardColumns", {}) + } + for col in table_schema.get("customColumns", []): + columns[col["name"]] = col["type"] + return dict(sorted(columns.items())) diff --git a/msticpy/data/drivers/cybereason_driver.py b/msticpy/data/drivers/cybereason_driver.py index 6d1c49a81..ad8115fb3 100644 --- a/msticpy/data/drivers/cybereason_driver.py +++ b/msticpy/data/drivers/cybereason_driver.py @@ -18,7 +18,7 @@ from ...common.provider_settings import ProviderArgs, get_provider_settings from ...common.utility import mp_ua_header from ..core.query_defns import Formatters -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource __version__ = VERSION __author__ = "Florian Bracq" @@ -87,11 +87,14 @@ def __init__(self, **kwargs): timeout=self.get_http_timeout(timeout=timeout, def_timeout=120), headers=mp_ua_header(), ) - self.formatters = { - Formatters.PARAM_HANDLER: self._custom_param_handler, - Formatters.DATETIME: self._format_datetime, - Formatters.LIST: self._format_list, - } + self.set_driver_property( + DriverProps.FORMATTERS, + { + Formatters.PARAM_HANDLER: self._custom_param_handler, + Formatters.DATETIME: self._format_datetime, + Formatters.LIST: self._format_list, + }, + ) self._debug = kwargs.get("debug", False) diff --git a/msticpy/data/drivers/driver_base.py b/msticpy/data/drivers/driver_base.py index a60bc7b9f..b4c5a6def 100644 --- a/msticpy/data/drivers/driver_base.py +++ b/msticpy/data/drivers/driver_base.py @@ -7,7 +7,7 @@ import abc from abc import ABC from collections import defaultdict -from typing import Any, Callable, Dict, Iterable, Optional, Set, Tuple, Union +from typing import Any, Dict, Iterable, Optional, Set, Tuple, Union import pandas as pd @@ -15,12 +15,61 @@ from ...common.exceptions import MsticpyNotConnectedError from ...common.pkg_config import get_http_timeout from ...common.provider_settings import ProviderSettings, get_provider_settings +from ..core.query_defns import DataEnvironment from ..core.query_source import QuerySource __version__ = VERSION __author__ = "Ian Hellen" +class DriverProps: + """Defined driver properties.""" + + PUBLIC_ATTRS = "public_attribs" + FORMATTERS = "formatters" + USE_QUERY_PATHS = "use_query_paths" + HAS_DRIVER_QUERIES = "has_driver_queries" + EFFECTIVE_ENV = "effective_environment" + SUPPORTS_THREADING = "supports_threading" + SUPPORTS_ASYNC = "supports_async" + MAX_PARALLEL = "max_parallel" + FILTER_ON_CONNECT = "filter_queries_on_connect" + + PROPERTY_TYPES: Dict[str, Any] = { + PUBLIC_ATTRS: dict, + FORMATTERS: dict, + USE_QUERY_PATHS: bool, + HAS_DRIVER_QUERIES: bool, + EFFECTIVE_ENV: (str, DataEnvironment), + SUPPORTS_THREADING: bool, + SUPPORTS_ASYNC: bool, + MAX_PARALLEL: int, + FILTER_ON_CONNECT: bool, + } + + @classmethod + def defaults(cls): + """Return default values for driver properties.""" + return { + cls.PUBLIC_ATTRS: {}, + cls.FORMATTERS: {}, + cls.USE_QUERY_PATHS: True, + cls.HAS_DRIVER_QUERIES: False, + cls.EFFECTIVE_ENV: None, + cls.SUPPORTS_THREADING: False, + cls.SUPPORTS_ASYNC: False, + cls.MAX_PARALLEL: 4, + cls.FILTER_ON_CONNECT: False, + } + + @classmethod + def valid_type(cls, property_name: str, value: Any) -> bool: + """Return expected property type.""" + if property_name not in cls.PROPERTY_TYPES: + return True + return isinstance(value, cls.PROPERTY_TYPES[property_name]) + + # pylint: disable=too-many-instance-attributes class DriverBase(ABC): """Base class for data providers.""" @@ -31,14 +80,29 @@ def __init__(self, **kwargs): self._loaded = False self._connected = False self.current_connection = None - self.public_attribs: Dict[str, Any] = {} - self.formatters: Dict[str, Callable] = {} - self.use_query_paths = True - self.has_driver_queries = False + # self.public_attribs: Dict[str, Any] = {} + # self.formatters: Dict[str, Callable] = {} + # self.use_query_paths = True + # self.has_driver_queries = False self._previous_connection = False self.data_environment = kwargs.get("data_environment") self._query_filter: Dict[str, Set[str]] = defaultdict(set) self._instance: Optional[str] = None + self.properties = DriverProps.defaults() + self.set_driver_property( + name=DriverProps.EFFECTIVE_ENV, + value=( + self.data_environment.name + if isinstance(self.data_environment, DataEnvironment) + else self.data_environment or "" + ), + ) + + def __getattr__(self, attrib): + """Return item from the properties dictionary as an attribute.""" + if attrib in self.properties: + return self.properties[attrib] + raise AttributeError(f"{self.__class__.__name__} has no attribute '{attrib}'") @property def loaded(self) -> bool: @@ -119,7 +183,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): @abc.abstractmethod def query( - self, query: str, query_source: QuerySource = None, **kwargs + self, query: str, query_source: Optional[QuerySource] = None, **kwargs ) -> Union[pd.DataFrame, Any]: """ Execute query string and return DataFrame of results. @@ -140,7 +204,7 @@ def query( Returns ------- Union[pd.DataFrame, Any] - A DataFrame (if successfull) or + A DataFrame (if successful) or the underlying provider result if an error. """ @@ -195,7 +259,7 @@ def query_attach_spec(self) -> Dict[str, Set[str]]: """Parameters that determine whether a query is relevant for the driver.""" return self._query_filter - def add_query_filter(self, name, query_filter): + def add_query_filter(self, name: str, query_filter: Union[str, Iterable]): """Add an expression to the query attach filter.""" allowed_names = {"data_environments", "data_families", "data_sources"} if name not in allowed_names: @@ -203,7 +267,28 @@ def add_query_filter(self, name, query_filter): f"'name' {name} must be one of:", ", ".join(f"'{name}'" for name in allowed_names), ) - self._query_filter[name].add(query_filter) + if isinstance(query_filter, str): + self._query_filter[name].add(query_filter) + else: + self._query_filter[name].update(query_filter) + + def set_driver_property(self, name: str, value: Any): + """Set an item in driver properties.""" + if not DriverProps.valid_type(name, value): + raise TypeError( + f"Property '{name}' is not the correct type.", + f"Expected: '{DriverProps.PROPERTY_TYPES[name]}'.", + ) + self.properties[name] = value + + def get_driver_property(self, name: str) -> Any: + """Return value or KeyError from driver properties.""" + return self.properties[name] + + def query_usable(self, query_source: QuerySource) -> bool: + """Return True if query should be exposed for this driver.""" + del query_source + return True # Read values from configuration @staticmethod diff --git a/msticpy/data/drivers/elastic_driver.py b/msticpy/data/drivers/elastic_driver.py index 6e1e2d95b..6d72042d4 100644 --- a/msticpy/data/drivers/elastic_driver.py +++ b/msticpy/data/drivers/elastic_driver.py @@ -14,7 +14,7 @@ from ...common.exceptions import MsticpyUserConfigError from ...common.utility import check_kwargs, export from ..core.query_defns import Formatters -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource __version__ = VERSION __author__ = "Neil Desai, Ian Hellen" @@ -42,11 +42,14 @@ def __init__(self, **kwargs): self._connected = False self._debug = kwargs.get("debug", False) - self.formatters = { - Formatters.PARAM_HANDLER: self._custom_param_handler, - Formatters.DATETIME: self._format_datetime, - Formatters.LIST: self._format_list, - } + self.set_driver_property( + DriverProps.FORMATTERS, + { + Formatters.PARAM_HANDLER: self._custom_param_handler, + Formatters.DATETIME: self._format_datetime, + Formatters.LIST: self._format_list, + }, + ) def connect(self, connection_str: str = None, **kwargs): """ diff --git a/msticpy/data/drivers/kql_driver.py b/msticpy/data/drivers/kql_driver.py index fd3300230..f2dc09f9a 100644 --- a/msticpy/data/drivers/kql_driver.py +++ b/msticpy/data/drivers/kql_driver.py @@ -30,7 +30,7 @@ from ...common.utility import MSTICPY_USER_AGENT, export from ...common.wsconfig import WorkspaceConfig from ..core.query_defns import DataEnvironment -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource _KQL_ENV_OPTS = "KQLMAGIC_CONFIGURATION" @@ -118,7 +118,10 @@ def __init__(self, connection_str: str = None, **kwargs): self._debug = kwargs.get("debug", False) super().__init__(**kwargs) - self.formatters = {"datetime": self._format_datetime, "list": self._format_list} + self.set_driver_property( + DriverProps.FORMATTERS, + {"datetime": self._format_datetime, "list": self._format_list}, + ) self._loaded = self._is_kqlmagic_loaded() os.environ["KQLMAGIC_LOAD_MODE"] = "silent" diff --git a/msticpy/data/drivers/kusto_driver.py b/msticpy/data/drivers/kusto_driver.py index 668868a65..8a5cb4dec 100644 --- a/msticpy/data/drivers/kusto_driver.py +++ b/msticpy/data/drivers/kusto_driver.py @@ -134,7 +134,7 @@ def query( Returns ------- Union[pd.DataFrame, results.ResultSet] - A DataFrame (if successfull) or + A DataFrame (if successful) or the underlying provider result if an error. """ @@ -214,12 +214,12 @@ def _create_connection(self, cluster, database): "of your msticyconfig.yaml", "Expected format:", "Kusto[-instance_name]:", - " args:", + " Args:", " Cluster: cluster_uri", " Integrated: True", "or", "Kusto[-instance_name]:", - " args:", + " Args:", " Cluster: cluster_uri", " TenantId: tenant_uuid", " ClientId: tenant_uuid", diff --git a/msticpy/data/drivers/mdatp_driver.py b/msticpy/data/drivers/mdatp_driver.py index 68173c3c0..19cbe0d39 100644 --- a/msticpy/data/drivers/mdatp_driver.py +++ b/msticpy/data/drivers/mdatp_driver.py @@ -50,9 +50,7 @@ def __init__(self, connection_str: str = None, instance: str = "Default", **kwar api_uri, oauth_uri, api_suffix = _select_api_uris( self.data_environment, self.cloud ) - self.add_query_filter("data_environments", "MDE") - self.add_query_filter("data_environments", "M365D") - self.add_query_filter("data_environments", "MDATP") + self.add_query_filter("data_environments", ("MDE", "M365D", "MDATP")) self.req_body = { "client_id": None, diff --git a/msticpy/data/drivers/mordor_driver.py b/msticpy/data/drivers/mordor_driver.py index 6708c857f..77956654d 100644 --- a/msticpy/data/drivers/mordor_driver.py +++ b/msticpy/data/drivers/mordor_driver.py @@ -24,7 +24,7 @@ from ...common.pkg_config import get_config from ...common.utility import mp_ua_header from ..core.query_source import QuerySource -from .driver_base import DriverBase +from .driver_base import DriverBase, DriverProps __version__ = VERSION __author__ = "Ian Hellen" @@ -41,8 +41,8 @@ _MTR_TAC_CAT_URI = "https://attack.mitre.org/tactics/{cat}/" _MTR_TECH_CAT_URI = "https://attack.mitre.org/techniques/{cat}/" -MITRE_TECHNIQUES: pd.DataFrame = None -MITRE_TACTICS: pd.DataFrame = None +MITRE_TECHNIQUES: Optional[pd.DataFrame] = None +MITRE_TACTICS: Optional[pd.DataFrame] = None _MITRE_TECH_CACHE = "mitre_tech_cache.pkl" _MITRE_TACTICS_CACHE = "mitre_tact_cache.pkl" @@ -108,12 +108,15 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): self.mdr_idx_tech, self.mdr_idx_tact = _build_mdr_indexes(self.mordor_data) self._connected = True - self.public_attribs = { - "mitre_techniques": self.mitre_techniques, - "mitre_tactics": self.mitre_tactics, - "driver_queries": self.driver_queries, - "search_queries": self.search_queries, - } + self.set_driver_property( + DriverProps.PUBLIC_ATTRS, + { + "mitre_techniques": self.mitre_techniques, + "mitre_tactics": self.mitre_tactics, + "driver_queries": self.driver_queries, + "search_queries": self.search_queries, + }, + ) # pylint: enable=global-statement @@ -345,8 +348,13 @@ def technique_name(self) -> Optional[str]: Name of the Mitre technique """ - if not self._technique_name and self.technique in MITRE_TECHNIQUES.index: - self._technique_name = MITRE_TECHNIQUES.loc[self.technique].Name + if ( + not self._technique_name + and self.technique in MITRE_TECHNIQUES.index # type: ignore[union-attr] + ): + self._technique_name = MITRE_TECHNIQUES.loc[ # type: ignore[union-attr] + self.technique + ].Name return self._technique_name @property @@ -360,8 +368,13 @@ def technique_desc(self) -> Optional[str]: Technique description """ - if not self._technique_desc and self.technique in MITRE_TECHNIQUES.index: - self._technique_desc = MITRE_TECHNIQUES.loc[self.technique].Description + if ( + not self._technique_desc + and self.technique in MITRE_TECHNIQUES.index # type: ignore[union-attr] + ): + self._technique_desc = MITRE_TECHNIQUES.loc[ # type: ignore + self.technique + ].Description return self._technique_desc @property @@ -392,9 +405,9 @@ def tactics_full(self) -> List[Tuple[str, str, str, str]]: if not self._tactics_full and self.tactics: for tactic in self.tactics: tactic_name = tactic_desc = "unknown" - if tactic in MITRE_TACTICS.index: - tactic_name = MITRE_TACTICS.loc[tactic].Name - tactic_desc = MITRE_TACTICS.loc[tactic].Description + if tactic in MITRE_TACTICS.index: # type: ignore[union-attr] + tactic_name = MITRE_TACTICS.loc[tactic].Name # type: ignore[union-attr] + tactic_desc = MITRE_TACTICS.loc[tactic].Description # type: ignore[union-attr] tactic_uri = self.MTR_TAC_URI.format(tactic_id=tactic) self._tactics_full.append( (tactic, tactic_name, tactic_desc, tactic_uri) diff --git a/msticpy/data/drivers/splunk_driver.py b/msticpy/data/drivers/splunk_driver.py index 5d8e4e0fe..68099da6a 100644 --- a/msticpy/data/drivers/splunk_driver.py +++ b/msticpy/data/drivers/splunk_driver.py @@ -19,7 +19,7 @@ ) from ...common.utility import check_kwargs, export from ..core.query_defns import Formatters -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource try: import splunklib.client as sp_client @@ -74,17 +74,23 @@ def __init__(self, **kwargs): self._loaded = True self._connected = False self._debug = kwargs.get("debug", False) - self.public_attribs = { - "client": self.service, - "saved_searches": self._saved_searches, - "fired_alerts": self._fired_alerts, - } - self.formatters = { - Formatters.DATETIME: self._format_datetime, - Formatters.LIST: self._format_list, - } - - def connect(self, connection_str: str = None, **kwargs): + self.set_driver_property( + DriverProps.PUBLIC_ATTRS, + { + "client": self.service, + "saved_searches": self._saved_searches, + "fired_alerts": self._fired_alerts, + }, + ) + self.set_driver_property( + DriverProps.FORMATTERS, + { + Formatters.DATETIME: self._format_datetime, + Formatters.LIST: self._format_list, + }, + ) + + def connect(self, connection_str: Optional[str] = None, **kwargs): """ Connect to Splunk via splunk-sdk. @@ -173,7 +179,7 @@ def _get_connect_args( return cs_dict def query( - self, query: str, query_source: QuerySource = None, **kwargs + self, query: str, query_source: Optional[QuerySource] = None, **kwargs ) -> Union[pd.DataFrame, Any]: """ Execute splunk query and retrieve results via OneShot or async search mode. diff --git a/msticpy/data/drivers/sumologic_driver.py b/msticpy/data/drivers/sumologic_driver.py index 9afbc74b2..a17d8cd6d 100644 --- a/msticpy/data/drivers/sumologic_driver.py +++ b/msticpy/data/drivers/sumologic_driver.py @@ -22,7 +22,7 @@ ) from ...common.provider_settings import ProviderSettings, get_provider_settings from ...common.utility import check_kwargs, export -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource __version__ = VERSION __author__ = "juju4" @@ -62,10 +62,10 @@ def __init__(self, **kwargs): self._loaded = True self._connected = False self._debug = kwargs.get("debug", False) - self.public_attribs = { - "client": self.service, - } - self.formatters = {"datetime": self._format_datetime} + self.set_driver_property(DriverProps.PUBLIC_ATTRS, {"client": self.service}) + self.set_driver_property( + DriverProps.FORMATTERS, {"datetime": self._format_datetime} + ) self.checkinterval = self._DEF_CHECKINTERVAL self.timeout = self._DEF_TIMEOUT diff --git a/msticpy/init/azure_ml_tools.py b/msticpy/init/azure_ml_tools.py index 5b74510ac..cb15c2312 100644 --- a/msticpy/init/azure_ml_tools.py +++ b/msticpy/init/azure_ml_tools.py @@ -23,7 +23,7 @@ from .._version import VERSION from ..common.pkg_config import _HOME_PATH, refresh_config from ..common.utility import search_for_file -from ..config import MpConfigFile +from ..config import MpConfigFile # pylint: disable=no-name-in-module __version__ = VERSION diff --git a/msticpy/init/nbinit.py b/msticpy/init/nbinit.py index 6aa0df212..d3d0b767d 100644 --- a/msticpy/init/nbinit.py +++ b/msticpy/init/nbinit.py @@ -59,7 +59,6 @@ from IPython import get_ipython from IPython.core.interactiveshell import InteractiveShell from IPython.display import HTML, display -from matplotlib import MatplotlibDeprecationWarning try: import seaborn as sns @@ -70,7 +69,13 @@ from ..auth.azure_auth_core import AzureCliStatus, check_cli_credentials from ..common.check_version import check_version from ..common.exceptions import MsticpyException, MsticpyUserError -from ..common.pkg_config import _HOME_PATH, get_config, refresh_config, validate_config +from ..common.pkg_config import _HOME_PATH +from ..common.settings import ( + current_config_path, + get_config, + refresh_config, + validate_config, +) from ..common.utility import ( check_and_install_missing_packages, check_kwargs, @@ -182,8 +187,6 @@ def _verbose(verbosity: Optional[int] = None) -> int: dict(pkg="IPython.display", tgt="Markdown"), dict(pkg="ipywidgets", alias="widgets"), dict(pkg="pathlib", tgt="Path"), - dict(pkg="matplotlib.pyplot", alias="plt"), - dict(pkg="matplotlib", tgt="MatplotlibDeprecationWarning"), dict(pkg="numpy", alias="np"), ] if sns is not None: @@ -212,8 +215,6 @@ def _verbose(verbosity: Optional[int] = None) -> int: _MP_IMPORT_ALL: List[Dict[str, str]] = [ dict(module_name="msticpy.datamodel.entities"), - dict(module_name="msticpy.nbtools"), - dict(module_name="msticpy.sectools"), ] # pylint: enable=use-dict-literal @@ -670,6 +671,7 @@ def _use_custom_config(config_file: str): def _get_or_create_config() -> bool: # Cases + # 0. Current config file exists -> return ok # 1. Env var set and mpconfig exists -> goto 4 # 2. Env var set and mpconfig file not exists - warn and continue # 3. search_for_file finds mpconfig -> goto 4 @@ -677,6 +679,8 @@ def _get_or_create_config() -> bool: # 5. search_for_file(config.json) # 6. If aml user try to import config.json into mpconfig and save # 7. Error - no Microsoft Sentinel config + if current_config_path() is not None: + return True mp_path = os.environ.get("MSTICPYCONFIG") if mp_path and not Path(mp_path).is_file(): _err_output(_MISSING_MPCONFIG_ENV_ERR) @@ -727,9 +731,6 @@ def _set_nb_options(namespace): "style": {"description_width": "initial"}, } - # Some of our dependencies (networkx) still use deprecated Matplotlib - # APIs - we can't do anything about it, so suppress them from view - warnings.simplefilter("ignore", category=MatplotlibDeprecationWarning) warnings.filterwarnings("ignore", category=DeprecationWarning) if sns: sns.set() diff --git a/msticpy/resources/mpconfig_defaults.yaml b/msticpy/resources/mpconfig_defaults.yaml index 4d6a84862..f04fc5fe1 100644 --- a/msticpy/resources/mpconfig_defaults.yaml +++ b/msticpy/resources/mpconfig_defaults.yaml @@ -112,6 +112,11 @@ TIProviders: Args: ApiID: str() AuthKey: *cred_key + Pulsedive: + Args: + AuthKey: *cred_key + Primary: bool(default=False) + Provider: "Pulsedive" OtherProviders: GeoIPLite: Args: @@ -193,6 +198,11 @@ DataProviders: ClientId: str(required=False, format=uuid) # [SuppressMessage("Microsoft.Security", "CS002:SecretInNextLine", Justification="Test code")] ClientSecret: *cred_key_opt +msticpy: + FriendlyExceptions: bool(default=True, required=False) + QueryDefinitions: list(required=False) + http_timeout: int(required=False) + # Proxies key not yet supported AzureCLI: # Deprecated section - use DataProviders.AzureCLI Args: diff --git a/requirements-all.txt b/requirements-all.txt index a82655fd5..d35ac4ec4 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -3,6 +3,7 @@ azure-common>=1.1.18 azure-core>=1.24.0 azure-identity>=1.10.0 azure-keyvault-secrets>=4.0.0 +azure-kusto-data>=4.0.0 azure-mgmt-compute>=4.6.2 azure-mgmt-core>=1.2.1 azure-mgmt-keyvault>=2.0.0 @@ -11,6 +12,7 @@ azure-mgmt-network>=2.7.0 azure-mgmt-resource>=16.1.0 azure-mgmt-resourcegraph>=8.0.0 azure-mgmt-subscription>=3.0.0 +azure-monitor-query>=1.0.0 azure-storage-blob>=12.5.0 beautifulsoup4>=4.0.0 bokeh>=1.4.0, <=2.4.3 diff --git a/requirements.txt b/requirements.txt index d11fd1d98..376b65ca7 100644 --- a/requirements.txt +++ b/requirements.txt @@ -17,7 +17,6 @@ ipython >= 7.23.1; python_version >= "3.8" ipywidgets>=7.4.2, <8.0.0 KqlmagicCustom[jupyter-basic,auth_code_clipboard]>=0.1.114.post22 lxml>=4.6.5 -matplotlib>=3.0.0 msal>=1.12.0 msal_extensions>=0.3.0 msrest>=0.6.0 diff --git a/setup.py b/setup.py index ca0383b93..aa6fc3ed8 100644 --- a/setup.py +++ b/setup.py @@ -41,12 +41,21 @@ def _combine_extras(extras: list) -> list: "azure-storage-blob>=12.5.0", "azure-mgmt-resourcegraph>=8.0.0", ], + "azure_query": [ + "azure-kusto-data>=4.0.0", + "azure-monitor-query>=1.0.0", + ], "keyvault": [ "azure-keyvault-secrets>=4.0.0", "azure-mgmt-keyvault>=2.0.0", "keyring>=13.2.1", # needed by Key Vault package ], - "ml": ["scikit-learn>=1.0.0", "scipy>=1.1.0", "statsmodels>=0.11.1"], + "ml": [ + "scikit-learn>=1.0.0", + "scipy>=1.1.0", + "statsmodels>=0.11.1", + "matplotlib>=3.0.0", + ], "sql2kql": ["mo-sql-parsing>=8, <9.0.0"], "riskiq": ["passivetotal>=2.5.3"], "panel": ["panel>=0.14.4"], @@ -61,7 +70,7 @@ def _combine_extras(extras: list) -> list: _combine_extras(list({name for name in EXTRAS if name != "dev"})) ) -EXTRAS["azure"] = sorted(_combine_extras(["_azure_core", "keyvault"])) +EXTRAS["azure"] = sorted(_combine_extras(["_azure_core", "keyvault", "azure_query"])) EXTRAS["test"] = sorted(_combine_extras(["all", "dev"])) EXTRAS["azsentinel"] = sorted(_combine_extras(["azure", "kql", "keyvault"])) EXTRAS["azuresentinel"] = sorted(_combine_extras(["azure", "kql", "keyvault"])) diff --git a/tests/analysis/polling_detection/test_periodogram_polling_detector.py b/tests/analysis/polling_detection/test_periodogram_polling_detector.py index 13c312ed7..d84792176 100644 --- a/tests/analysis/polling_detection/test_periodogram_polling_detector.py +++ b/tests/analysis/polling_detection/test_periodogram_polling_detector.py @@ -10,9 +10,8 @@ The code for this is located at https://github.com/cran/GeneCycle/blob/master/R/fisher.g.test.R """ -from datetime import datetime - import unittest.mock as mock +from datetime import datetime import numpy as np import pandas as pd @@ -22,6 +21,7 @@ __author__ = "Daniel Yates" + ## #### ## ## init ## ## #### ## diff --git a/tests/common/test_wsconfig.py b/tests/common/test_wsconfig.py index aaf2411a2..df35d5715 100644 --- a/tests/common/test_wsconfig.py +++ b/tests/common/test_wsconfig.py @@ -5,9 +5,12 @@ # -------------------------------------------------------------------------- """WorkspaceConfig test class.""" import io -import unittest from contextlib import redirect_stdout from pathlib import Path +from typing import NamedTuple + +import pytest +import pytest_check as check from msticpy.common import pkg_config from msticpy.common.wsconfig import WorkspaceConfig @@ -19,120 +22,138 @@ # pylint: disable=protected-access -class TestPkgConfig(unittest.TestCase): - """Unit test class.""" - - def test_wsconfig_default_ws(self): - """Test WorkspaceConfig.""" - test_config1 = Path(_TEST_DATA).joinpath(pkg_config._CONFIG_FILE) - with custom_mp_config(test_config1): - # Default workspace - _DEF_WS = { - "WorkspaceId": "52b1ab41-869e-4138-9e40-2a4457f09bf3", - "TenantId": "72f988bf-86f1-41af-91ab-2d7cd011db49", - } - ws_config = WorkspaceConfig() - self.assertIn("workspace_id", ws_config) - self.assertEqual(ws_config["workspace_id"], _DEF_WS["WorkspaceId"]) - self.assertIn("tenant_id", ws_config) - self.assertEqual(ws_config["tenant_id"], _DEF_WS["TenantId"]) - self.assertIsNotNone(ws_config.code_connect_str) - self.assertTrue( - ws_config.code_connect_str.startswith("loganalytics://code().tenant(") - and _DEF_WS["WorkspaceId"] in ws_config.code_connect_str - and _DEF_WS["TenantId"] in ws_config.code_connect_str - ) - - def test_wsconfig_named_ws(self): - """Test WorkspaceConfig.""" - test_config1 = Path(_TEST_DATA).joinpath(pkg_config._CONFIG_FILE) - with custom_mp_config(test_config1): - # Named workspace - _NAMED_WS = { - "WorkspaceId": "a927809c-8142-43e1-96b3-4ad87cfe95a3", - "TenantId": "69d28fd7-42a5-48bc-a619-af56397b9f28", - } - wstest_config = WorkspaceConfig(workspace="MyTestWS") - self.assertIn("workspace_id", wstest_config) - self.assertIsNotNone(wstest_config["workspace_id"]) - self.assertEqual(wstest_config["workspace_id"], _NAMED_WS["WorkspaceId"]) - self.assertIn("tenant_id", wstest_config) - self.assertEqual(wstest_config["tenant_id"], _NAMED_WS["TenantId"]) - self.assertIsNotNone(wstest_config.code_connect_str) - self.assertTrue( - wstest_config.code_connect_str.startswith( - "loganalytics://code().tenant(" - ) - and _NAMED_WS["WorkspaceId"] in wstest_config.code_connect_str - and _NAMED_WS["TenantId"] in wstest_config.code_connect_str - ) - - def test_wsconfig_config_json_fallback(self): - # Fallback to config.json - test_config2 = Path(_TEST_DATA).joinpath("msticpyconfig-noAzSentSettings.yaml") - with custom_mp_config(test_config2): - _NAMED_WS = { - "WorkspaceId": "9997809c-8142-43e1-96b3-4ad87cfe95a3", - "TenantId": "99928fd7-42a5-48bc-a619-af56397b9f28", - } - wrn_mssg = io.StringIO() - with redirect_stdout(wrn_mssg): - wstest_config = WorkspaceConfig() - self.assertIn( - "Could not find Microsoft Sentinel settings", wrn_mssg.getvalue() - ) - self.assertIn("workspace_id", wstest_config) - self.assertIsNotNone(wstest_config["workspace_id"]) - self.assertEqual(wstest_config["workspace_id"], _NAMED_WS["WorkspaceId"]) - self.assertIn("tenant_id", wstest_config) - self.assertEqual(wstest_config["tenant_id"], _NAMED_WS["TenantId"]) - self.assertIsNotNone(wstest_config.code_connect_str) - self.assertTrue( - wstest_config.code_connect_str.startswith( - "loganalytics://code().tenant(" - ) - and _NAMED_WS["WorkspaceId"] in wstest_config.code_connect_str - and _NAMED_WS["TenantId"] in wstest_config.code_connect_str - ) - - def test_wsconfig_misc_funcs(self): - """Test miscellaneous functions.""" - test_config1 = Path(_TEST_DATA).joinpath(pkg_config._CONFIG_FILE) - with custom_mp_config(test_config1): - ws_dict = WorkspaceConfig.list_workspaces() - self.assertIn("Default", ws_dict) - self.assertEqual( - ws_dict["Default"]["WorkspaceId"], - "52b1ab41-869e-4138-9e40-2a4457f09bf3", - ) - self.assertEqual( - ws_dict["MyTestWS"]["WorkspaceId"], - "a927809c-8142-43e1-96b3-4ad87cfe95a3", - ) - ws_config = WorkspaceConfig() - ws_config.prompt_for_ws() - - def test_wsconfig_single_ws(self): - test_config3 = Path(_TEST_DATA).joinpath( - "msticpyconfig-SingleAzSentSettings.yaml" +def test_wsconfig_default_ws(): + """Test WorkspaceConfig - no parameter.""" + test_config1 = Path(_TEST_DATA).joinpath(pkg_config._CONFIG_FILE) + with custom_mp_config(test_config1): + # Default workspace + _DEF_WS = { + "WorkspaceId": "52b1ab41-869e-4138-9e40-2a4457f09bf3", + "TenantId": "72f988bf-86f1-41af-91ab-2d7cd011db49", + } + ws_config = WorkspaceConfig() + check.is_in("workspace_id", ws_config) + check.equal(ws_config["workspace_id"], _DEF_WS["WorkspaceId"]) + check.is_in("tenant_id", ws_config) + check.equal(ws_config["tenant_id"], _DEF_WS["TenantId"]) + check.is_not_none(ws_config.code_connect_str) + check.is_true( + ws_config.code_connect_str.startswith("loganalytics://code().tenant(") + and _DEF_WS["WorkspaceId"] in ws_config.code_connect_str + and _DEF_WS["TenantId"] in ws_config.code_connect_str + ) + + +class WsTestCase(NamedTuple): + """Test case for named workspaces.""" + + ws_name: str + ws_id: str + tenant_id: str + + +Workspace2 = WsTestCase( + "Workspace2", + "a927809c-8142-43e1-96b3-4ad87cfe95a4", + "69d28fd7-42a5-48bc-a619-af56397b9f28", +) +DefaultWS = WsTestCase( + "Workspace1", + "52b1ab41-869e-4138-9e40-2a4457f09bf3", + "72f988bf-86f1-41af-91ab-2d7cd011db49", +) + +WS_TEST_CASES = ( + ("MyTestWS2", Workspace2), + ("mytestws2", Workspace2), + ("Workspace2", Workspace2), + ("workspace2", Workspace2), + (Workspace2.ws_id, Workspace2), + (None, DefaultWS), +) + + +@pytest.mark.parametrize("name, expected", WS_TEST_CASES) +def test_wsconfig_named_ws(name, expected): + """Test WorkspaceConfig.""" + test_config1 = get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + with custom_mp_config(test_config1): + # Named workspace + + wstest_config = WorkspaceConfig(workspace=name) + check.is_in("workspace_id", wstest_config) + check.equal(wstest_config["workspace_id"], expected.ws_id) + check.equal(wstest_config["tenant_id"], expected.tenant_id) + check.equal(wstest_config["workspace_name"], expected.ws_name) + check.is_not_none(wstest_config.code_connect_str) + check.is_true( + wstest_config.code_connect_str.startswith("loganalytics://code().tenant(") + and expected.ws_id in wstest_config.code_connect_str + and expected.tenant_id in wstest_config.code_connect_str ) - with custom_mp_config(test_config3): - # Single workspace - _NAMED_WS = { - "WorkspaceId": "a927809c-8142-43e1-96b3-4ad87cfe95a3", - "TenantId": "69d28fd7-42a5-48bc-a619-af56397b9f28", - } + + +def test_wsconfig_config_json_fallback(): + """Test fallback to config.json if no AzSent settings in config.yaml.""" + test_config2 = Path(_TEST_DATA).joinpath("msticpyconfig-noAzSentSettings.yaml") + with custom_mp_config(test_config2): + _NAMED_WS = { + "WorkspaceId": "9997809c-8142-43e1-96b3-4ad87cfe95a3", + "TenantId": "99928fd7-42a5-48bc-a619-af56397b9f28", + } + wrn_mssg = io.StringIO() + with redirect_stdout(wrn_mssg): wstest_config = WorkspaceConfig() - self.assertIn("workspace_id", wstest_config) - self.assertIsNotNone(wstest_config["workspace_id"]) - self.assertEqual(wstest_config["workspace_id"], _NAMED_WS["WorkspaceId"]) - self.assertIn("tenant_id", wstest_config) - self.assertEqual(wstest_config["tenant_id"], _NAMED_WS["TenantId"]) - self.assertIsNotNone(wstest_config.code_connect_str) - self.assertTrue( - wstest_config.code_connect_str.startswith( - "loganalytics://code().tenant(" - ) - and _NAMED_WS["WorkspaceId"] in wstest_config.code_connect_str - and _NAMED_WS["TenantId"] in wstest_config.code_connect_str - ) + check.is_in("Could not find Microsoft Sentinel settings", wrn_mssg.getvalue()) + check.is_in("workspace_id", wstest_config) + check.is_not_none(wstest_config["workspace_id"]) + check.equal(wstest_config["workspace_id"], _NAMED_WS["WorkspaceId"]) + check.is_in("tenant_id", wstest_config) + check.equal(wstest_config["tenant_id"], _NAMED_WS["TenantId"]) + check.is_not_none(wstest_config.code_connect_str) + check.is_true( + wstest_config.code_connect_str.startswith("loganalytics://code().tenant(") + and _NAMED_WS["WorkspaceId"] in wstest_config.code_connect_str + and _NAMED_WS["TenantId"] in wstest_config.code_connect_str + ) + + +def test_wsconfig_misc_funcs(): + """Test miscellaneous functions.""" + test_config1 = Path(_TEST_DATA).joinpath(pkg_config._CONFIG_FILE) + with custom_mp_config(test_config1): + ws_dict = WorkspaceConfig.list_workspaces() + check.is_in("Default", ws_dict) + check.equal( + ws_dict["Default"]["WorkspaceId"], + "52b1ab41-869e-4138-9e40-2a4457f09bf3", + ) + check.equal( + ws_dict["MyTestWS"]["WorkspaceId"], + "a927809c-8142-43e1-96b3-4ad87cfe95a3", + ) + ws_config = WorkspaceConfig() + ws_config.prompt_for_ws() + + +def test_wsconfig_single_ws(): + """Test single workspace config.""" + test_config3 = Path(_TEST_DATA).joinpath("msticpyconfig-SingleAzSentSettings.yaml") + with custom_mp_config(test_config3): + # Single workspace + _NAMED_WS = { + "WorkspaceId": "a927809c-8142-43e1-96b3-4ad87cfe95a3", + "TenantId": "69d28fd7-42a5-48bc-a619-af56397b9f28", + } + wstest_config = WorkspaceConfig() + check.is_in("workspace_id", wstest_config) + check.is_not_none(wstest_config["workspace_id"]) + check.equal(wstest_config["workspace_id"], _NAMED_WS["WorkspaceId"]) + check.is_in("tenant_id", wstest_config) + check.equal(wstest_config["tenant_id"], _NAMED_WS["TenantId"]) + check.is_not_none(wstest_config.code_connect_str) + check.is_true( + wstest_config.code_connect_str.startswith("loganalytics://code().tenant(") + and _NAMED_WS["WorkspaceId"] in wstest_config.code_connect_str + and _NAMED_WS["TenantId"] in wstest_config.code_connect_str + ) diff --git a/tests/context/azure/sentinel_test_fixtures.py b/tests/context/azure/sentinel_test_fixtures.py index 00a13d82c..7ae3fb0c5 100644 --- a/tests/context/azure/sentinel_test_fixtures.py +++ b/tests/context/azure/sentinel_test_fixtures.py @@ -11,6 +11,8 @@ from msticpy import VERSION from msticpy.context.azure import MicrosoftSentinel +from ...unit_test_lib import custom_mp_config, get_test_data_path + __version__ = VERSION __author__ = "Ian Hellen" @@ -20,12 +22,23 @@ def _set_default_workspace(self, sub_id, workspace=None): """Mock set_default_workspace for MSSentinel.""" del sub_id, workspace - self._default_workspace = ( - "WSName", - "/subscriptions/cd928da3-dcde-42a3-aad7-d2a1268c2f48/" - "resourceGroups/RG/providers/" - "Microsoft.OperationalInsights/workspaces/WSName", - ) + self._default_workspace = "Default" + # ( + # "WSName", + # "/subscriptions/cd928da3-dcde-42a3-aad7-d2a1268c2f48/" + # "resourceGroups/RG/providers/" + # "Microsoft.OperationalInsights/workspaces/WSName", + # ) + + +# @pytest.fixture(scope="module") +# def sentinel_instance(): +# """Generate MicrosoftSentinel instance for testing.""" +# with custom_mp_config(get_test_data_path().parent.joinpath("msticpyconfig.yaml")): +# return MicrosoftSentinel( +# # sub_id="fd09863b-5cec-4833-ab9c-330ad07b0c1a", res_grp="RG", ws_name="WSName" +# workspace="WSName" +# ) @pytest.fixture @@ -38,10 +51,14 @@ def sent_loader(mock_creds, get_token, monkeypatch): ) mock_creds.return_value = None get_token.return_value = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" - sent = MicrosoftSentinel( - sub_id="fd09863b-5cec-4833-ab9c-330ad07b0c1a", res_grp="RG", ws_name="WSName" - ) - sent.connect() - sent.connected = True - sent.token = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" - return sent + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + sentinel = MicrosoftSentinel( + # sub_id="fd09863b-5cec-4833-ab9c-330ad07b0c1a", res_grp="RG", ws_name="WSName" + workspace="WSName" + ) + sentinel.connect() + sentinel.connected = True + sentinel.token = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" + return sentinel diff --git a/tests/context/azure/test_sentinel_core.py b/tests/context/azure/test_sentinel_core.py index 534857874..dbd701dc2 100644 --- a/tests/context/azure/test_sentinel_core.py +++ b/tests/context/azure/test_sentinel_core.py @@ -12,6 +12,8 @@ from msticpy.context.azure import AzureData, MicrosoftSentinel +from ...unit_test_lib import custom_mp_config, get_test_data_path + # pylint: disable=redefined-outer-name, protected-access _RESOURCES = pd.DataFrame( @@ -25,20 +27,24 @@ "resource_id": ["123", "456", "789"], } ) -_RESOURCE_DETAILS = {"properties": {"workspaceResourceId": "ABC"}} _RES_ID = ( "subscriptions/123/resourceGroups/RG/providers/" "Microsoft.OperationalInsights/workspaces/WSNAME" ) +_RESOURCE_DETAILS = {"properties": {"workspaceResourceId": _RES_ID}} + def test_azuresent_init(): """Test class initialization.""" - sentinel_inst = MicrosoftSentinel(sub_id="123", res_grp="RG", ws_name="WSName") - assert isinstance(sentinel_inst, MicrosoftSentinel) - sentinel_inst = MicrosoftSentinel(res_id=_RES_ID) - assert isinstance(sentinel_inst, MicrosoftSentinel) + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + sentinel_inst = MicrosoftSentinel(sub_id="123", res_grp="RG", ws_name="WSName") + assert isinstance(sentinel_inst, MicrosoftSentinel) + sentinel_inst = MicrosoftSentinel(res_id=_RES_ID) + assert isinstance(sentinel_inst, MicrosoftSentinel) @patch(MicrosoftSentinel.__module__ + ".AzureData.connect") @@ -47,29 +53,34 @@ def test_azuresent_connect_token(get_token: Mock, az_data_connect: Mock): """Test connect success.""" token = "12398120398" - sentinel_inst = MicrosoftSentinel(res_id=_RES_ID) - - setattr(sentinel_inst, "set_default_workspace", MagicMock()) - sentinel_inst.connect(auth_methods=["env"], token=token) - - tenant_id = sentinel_inst._check_config(["tenant_id"])["tenant_id"] - assert sentinel_inst.token == token - az_data_connect.assert_called_once_with( - auth_methods=["env"], tenant_id=tenant_id, silent=False - ) + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + sentinel_inst = MicrosoftSentinel(res_id=_RES_ID) - get_token.return_value = token - token = "12398120398" - sentinel_inst = MicrosoftSentinel( - res_id=_RES_ID, - ) - setattr(sentinel_inst, "set_default_workspace", MagicMock()) - sentinel_inst.connect(auth_methods=["env"]) + setattr(sentinel_inst, "set_default_workspace", MagicMock()) + sentinel_inst.connect(auth_methods=["env"], token=token) + + tenant_id = sentinel_inst._check_config(["tenant_id"])["tenant_id"] + assert sentinel_inst.token == token + az_data_connect.assert_called_once_with( + auth_methods=["env"], tenant_id=tenant_id, silent=False + ) + + get_token.return_value = token + token = "12398120398" + sentinel_inst = MicrosoftSentinel( + res_id=_RES_ID, + ) + setattr(sentinel_inst, "set_default_workspace", MagicMock()) + sentinel_inst.connect(auth_methods=["env"]) - assert sentinel_inst.token == token - get_token.assert_called_once_with( - sentinel_inst.credentials, tenant_id=tenant_id, cloud=sentinel_inst.user_cloud - ) + assert sentinel_inst.token == token + get_token.assert_called_once_with( + sentinel_inst.credentials, + tenant_id=tenant_id, + cloud=sentinel_inst.user_cloud, + ) @patch(MicrosoftSentinel.__module__ + ".AzureData.connect") @@ -86,11 +97,14 @@ def test_azuresent_connect_fail(az_data_connect: Mock): def sentinel_inst_loader(mock_creds): """Generate MicrosoftSentinel for testing.""" mock_creds.return_value = None - sentinel_inst = MicrosoftSentinel(sub_id="123", res_grp="RG", ws_name="WSName") - sentinel_inst.connect() - sentinel_inst.connected = True - sentinel_inst.token = "123" - return sentinel_inst + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + sentinel_inst = MicrosoftSentinel(sub_id="123", res_grp="RG", ws_name="WSName") + sentinel_inst.connect() + sentinel_inst.connected = True + sentinel_inst.token = "123" + return sentinel_inst @patch(AzureData.__module__ + ".AzureData.get_resources") @@ -99,6 +113,9 @@ def test_azuresent_workspaces(mock_res_dets, mock_res, sentinel_inst_loader): """Test Sentinel workspaces feature.""" mock_res.return_value = _RESOURCES mock_res_dets.return_value = _RESOURCE_DETAILS - workspaces = sentinel_inst_loader.get_sentinel_workspaces(sub_id="123") - assert isinstance(workspaces, dict) - assert workspaces["ABC"] == "ABC" + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + workspaces = sentinel_inst_loader.get_sentinel_workspaces(sub_id="123") + assert isinstance(workspaces, dict) + assert workspaces["WSNAME"] == _RES_ID diff --git a/tests/context/azure/test_sentinel_dynamic_summary.py b/tests/context/azure/test_sentinel_dynamic_summary.py index 4e86d8772..8997018d4 100644 --- a/tests/context/azure/test_sentinel_dynamic_summary.py +++ b/tests/context/azure/test_sentinel_dynamic_summary.py @@ -15,9 +15,9 @@ import pytest import pytest_check as check import respx +import yaml from msticpy.common.exceptions import MsticpyAzureConnectionError -from msticpy.common.pkg_config import get_config from msticpy.common.wsconfig import WorkspaceConfig from msticpy.context.azure import MicrosoftSentinel from msticpy.context.azure.sentinel_dynamic_summary import SentinelQueryProvider @@ -26,7 +26,7 @@ DynamicSummary, ) -from ...unit_test_lib import get_test_data_path +from ...unit_test_lib import custom_mp_config, get_test_data_path # pylint: disable=redefined-outer-name, protected-access @@ -234,7 +234,10 @@ def list_responses(): def _get_test_ws_settings(): - az_ws_settings = get_config("AzureSentinel.Workspaces") + """Get test workspace settings from config file.""" + test_config = get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + settings = yaml.safe_load(test_config.read_text(encoding="utf-8")) + az_ws_settings = settings.get("AzureSentinel", {}).get("Workspaces", {}) return next( iter((key, val) for key, val in az_ws_settings.items() if key != "Default") ) @@ -260,15 +263,20 @@ def sentinel_loader(mock_creds, get_token, monkeypatch): get_token.return_value = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" mock_creds.return_value = None ws_key, settings = _get_test_ws_settings() - sent = MicrosoftSentinel( - sub_id=settings.get("SubscriptionId", "fd09863b-5cec-4833-ab9c-330ad07b0c1a"), - res_grp=settings.get("ResourceGroup", "RG"), - ws_name=settings.get("WorkspaceName", "Default"), - ) - sent._default_workspace = ws_key - sent.connect(workspace=ws_key) - sent.connected = True - sent.token = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" # nosec + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + sent = MicrosoftSentinel( + sub_id=settings.get( + "SubscriptionId", "fd09863b-5cec-4833-ab9c-330ad07b0c1a" + ), + res_grp=settings.get("ResourceGroup", "RG"), + ws_name=settings.get("WorkspaceName", "Default"), + ) + sent._default_workspace = ws_key + sent.connect(workspace=ws_key) + sent.connected = True + sent.token = "fd09863b-5cec-4833-ab9c-330ad07b0c1a" # nosec return sent diff --git a/tests/data/drivers/test_azure_kusto_driver.py b/tests/data/drivers/test_azure_kusto_driver.py new file mode 100644 index 000000000..2a405a4b8 --- /dev/null +++ b/tests/data/drivers/test_azure_kusto_driver.py @@ -0,0 +1,1165 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Azure Kusto driver query test class.""" +from datetime import timedelta +from itertools import product +from typing import Callable, List, NamedTuple, Optional + +import pandas as pd +import pytest +import pytest_check as check +import yaml +from azure.kusto.data.exceptions import KustoApiError, KustoServiceError + +from msticpy.auth.azure_auth_core import AzCredentials +from msticpy.common import proxy_settings +from msticpy.common.exceptions import ( + MsticpyDataQueryError, + MsticpyNotConnectedError, + MsticpyParameterError, +) +from msticpy.data.core.data_providers import QueryProvider +from msticpy.data.core.query_defns import DataEnvironment +from msticpy.data.core.query_source import QuerySource +from msticpy.data.drivers import azure_kusto_driver +from msticpy.data.drivers.azure_kusto_driver import AzureKustoDriver + +from ...unit_test_lib import custom_get_config, get_test_data_path +from .test_azure_monitor_driver import QUERY_SOURCE + +# from azure.kusto.data.request import KustoConnectionStringBuilder + +# pylint: disable=protected-access, unused-argument, redefined-outer-name + + +_KUSTO_CONFIG_SETTINGS_TEXT = """ +DataProviders: + Kusto-MDE: + Args: + Cluster: https://test.kusto.windows.net + IntegratedAuth: true + Kusto-MSTICTI: + Args: + Cluster: https://msticti.kusto.windows.net + IntegratedAuth: true + Kusto-Help: + Args: + Cluster: https://help.kusto.windows.net + IntegratedAuth: true + +KustoClusters: + ClusterDefaults: + Args: + TenantId: 72f988bf-86f1-41af-91ab-2d7cd011db47 + MDE: + ClusterGroups: [Group1] + Args: + Cluster: https://test.kusto.windows.net + ClientSecret: + EnvironmentVar: COMPUTERNAME + MSTICTI: + ClusterGroups: [Group1] + Args: + Cluster: https://msticti.kusto.windows.net + Help: + Args: + Cluster: https://help.kusto.windows.net + IntegratedAuth: true + G3Cluster: + ClusterGroups: [Group3] + Args: + Cluster: https://g3.kusto.windows.net + ClientId: random_client_id + ClientSecret: [PLACEHOLDER] + Database: G3DB + G2Cluster: + ClusterGroups: [Group2] + Args: + Cluster: https://g2.kusto.windows.net + +""" +_KUSTO_CONFIG_SETTINGS = yaml.safe_load(_KUSTO_CONFIG_SETTINGS_TEXT) +_KUSTO_LEGACY_CONFIG_SETTINGS = { + "DataProviders": _KUSTO_CONFIG_SETTINGS["DataProviders"] +} +_KUSTO_NEW_CONFIG_SETTINGS = {"KustoClusters": _KUSTO_CONFIG_SETTINGS["KustoClusters"]} + + +def get_test_df(): + return pd.read_csv(get_test_data_path().joinpath("host_logons.csv"), index_col=0) + + +def test_init(): + # Test that __init__ sets the current_connection property correctly + driver = AzureKustoDriver(connection_str="https://test.kusto.windows.net") + assert driver.current_connection == "https://test.kusto.windows.net" + + # Test that __init__ sets the _connection_props property correctly + driver = AzureKustoDriver(timeout=300) + assert driver._def_timeout == 300 + + driver = AzureKustoDriver(proxies={"https": "https://test.proxy.com"}) + assert driver._def_proxies == {"https": "https://test.proxy.com"} + + +class MPConfig: + """Mock MPConfig.""" + + def __init__(self, config=None): + """Initialize mock MPConfig.""" + self._config = config or {} + + def get(self, key, default=None): + """Return a Kusto config dictionary.""" + return self._config.get(key, default) + + +_TEST_CONFIG = ( + (_KUSTO_CONFIG_SETTINGS, ["MDE", "MSTICTI", "Help"], 5, "combined"), + (_KUSTO_LEGACY_CONFIG_SETTINGS, ["MDE", "MSTICTI", "Help"], 3, "legacy"), + (_KUSTO_NEW_CONFIG_SETTINGS, ["MDE", "MSTICTI", "Help"], 5, "new"), + ({}, [], 0, "empty"), +) + + +@pytest.mark.parametrize("conf, keys, items, name", _TEST_CONFIG) +def test_get_kusto_config(conf, keys, items, name, monkeypatch): + """Test creation of Kusto config entries.""" + config = MPConfig(conf) + monkeypatch.setattr(azure_kusto_driver, "get_config", config.get) + + driver = AzureKustoDriver(connection_str="https://test.kusto.windows.net") + for index in ("url", "name", "id"): + check.equal(len(driver._kusto_settings[index]), items) + for key in [key.casefold() for key in keys]: + check.is_in(key, driver._kusto_settings["id"]) + check.is_instance( + driver._kusto_settings["id"][key], azure_kusto_driver.KustoConfig + ) + + for _, k_config in driver._kusto_settings["id"].items(): + check.is_in("Cluster", k_config.args) + if "IntegratedAuth" in k_config.args: + check.equal(k_config.args["IntegratedAuth"], k_config.integrated_auth) + if "TenantId" in k_config.args: + check.equal(k_config.args["TenantId"], k_config.tenant_id) + check.is_not_none(k_config.path) + check.is_not_none(k_config.cluster) + check.is_not_none(k_config.name) + if "ClusterGroups" in k_config: + check.is_in("Group1", k_config.cluster_groups) + + +class Token: + """Mock token class.""" + + token = "test_token" + + +class Credentials: + """Mock credentials class.""" + + def get_token(self, *args, **kwargs): + """Mock get_token.""" + return Token() + + +def az_connect(*args, **kwargs): + """Mock az_connect.""" + return AzCredentials(legacy=[args, kwargs], modern=Credentials()) + + +class AzConnectState: + """Mock az_connect to capture args.""" + + def __init__(self): + """Initialize mock az_connect state.""" + self.connect_args = [] + self.connect_kwargs = {} + + def az_connect(self, *args, **kwargs): + """Mock az_connect.""" + self.connect_args = args + self.connect_kwargs = kwargs + return az_connect(*args, **kwargs) + + +class ConnectTest(NamedTuple): + """Test configuration for Kusto connection tests.""" + + init_args: dict + connect_args: dict + name: str + tests: List[Callable] + exception: Optional[type] = None + additional_config: Optional[dict] = None + + +_TEST_CONNECT_ARGS = ( + ConnectTest( + name="connection_str", + init_args={"connection_str": "https://test1.kusto.windows.net"}, + connect_args={"connection_str": "https://test.kusto.windows.net"}, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://test.kusto.windows.net" + ], + ), + ConnectTest( + name="cluster-no-init", + init_args={}, + connect_args={"cluster": "https://test.kusto.windows.net"}, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://test.kusto.windows.net", + lambda driver: driver._current_config.cluster + == "https://test.kusto.windows.net", + ], + ), + ConnectTest( + name="cluster-id-no-init", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "Help", + "auth_types": ["device_code"], + "tenant_id": "test_tenant_id", + }, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://help.kusto.windows.net", + lambda driver: driver._current_config.cluster + == "https://help.kusto.windows.net", + lambda driver: driver._az_tenant_id == "test_tenant_id", + lambda driver: driver._az_auth_types == ["device_code"], + ], + ), + ConnectTest( + name="cluster-name-no-init", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "help", + "auth_types": ["device_code"], + "tenant_id": "test_tenant_id", + "database": "test_db", + }, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://help.kusto.windows.net", + lambda driver: driver._current_config.cluster + == "https://help.kusto.windows.net", + lambda driver: driver._az_tenant_id == "test_tenant_id", + lambda driver: driver._az_auth_types == ["device_code"], + lambda driver: driver._default_database == "test_db", + ], + ), + ConnectTest( + name="mp_az_auth-bool", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "Help", + "mp_az_auth": True, + }, + tests=[ + lambda driver: driver._az_auth_types is None, + ], + ), + ConnectTest( + name="mp_az_auth-str", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "Help", + "mp_az_auth": "msi", + }, + tests=[ + lambda driver: driver._az_auth_types == ["msi"], + ], + ), + ConnectTest( + name="mp_az_auth-list", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "Help", + "mp_az_auth": ["msi", "cli"], + }, + tests=[ + lambda driver: driver._az_auth_types == ["msi", "cli"], + ], + ), + ConnectTest( + name="mp_az_auth-client_id-override", + init_args={ + "cluster": "https://test.kusto.windows.net", + }, + connect_args={ + "cluster": "G3", + "mp_az_auth": ["msi", "cli"], + }, + tests=[ + lambda az_connect: "clientsecret" + in az_connect.connect_kwargs["auth_types"], + ], + ), + ConnectTest( + name="no-cluster-or-connection_str", + init_args={}, + connect_args={}, + tests=[], + exception=MsticpyParameterError, + ), + ConnectTest( + name="connection_str-with-proxies", + init_args={}, + connect_args={"connection_str": "https://test.kusto.windows.net"}, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://test.kusto.windows.net", + lambda driver: driver.client._proxy_url == "https://test.com", + ], + additional_config={ + "msticpy": {"Proxies": {"https": {"Url": "https://test.com"}}} + }, + ), + ConnectTest( + name="cluster-not-in-config", + init_args={}, + connect_args={"cluster": "https://random.kusto.windows.net"}, + tests=[ + lambda driver: driver.client._kusto_cluster + == "https://random.kusto.windows.net" + ], + ), + ConnectTest( + name="cluster-not-in-config-and-not-url", + init_args={}, + connect_args={"cluster": "random_name"}, + tests=[], + exception=MsticpyDataQueryError, + ), +) + + +def unit_test_ids(tests): + """Test Kusto connection.""" + return [test.name for test in tests] + + +@pytest.mark.parametrize( + "test_config", _TEST_CONNECT_ARGS, ids=unit_test_ids(_TEST_CONNECT_ARGS) +) +def test_kusto_connect(test_config, monkeypatch): + """Test Kusto connect function.""" + config_settings = { + **_KUSTO_NEW_CONFIG_SETTINGS, + **(test_config.additional_config or {}), + } + + with custom_get_config( + monkeypatch=monkeypatch, + settings=config_settings, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + az_connect_obj = AzConnectState() + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect_obj.az_connect) + driver = AzureKustoDriver(**(test_config.init_args)) + if test_config.exception: + with pytest.raises(test_config.exception): + driver.connect(**(test_config.connect_args)) + return + driver.connect(**(test_config.connect_args)) + for test in test_config.tests: + print( + "Testcase:", + test_config.name, + test.__code__.co_filename, + "line:", + test.__code__.co_firstlineno, + ) + if "driver" in test.__code__.co_varnames: + check.is_true(test(driver)) + elif "az_connect" in test.__code__.co_varnames: + check.is_true(test(az_connect_obj)) + + cluster_name = test_config.connect_args.get("cluster") + if cluster_name: + driver.set_cluster(cluster_name) + if cluster_name.startswith("https://"): + exp_cluster = cluster_name + else: + exp_cluster = f"https://{cluster_name.casefold()}.kusto.windows.net" + check.equal(driver.client._kusto_cluster, exp_cluster) + + +class KustoResponseDataSet: + """Mock Kusto response dataset.""" + + def __init__(self, data, query_status=None): + """Initialize dataset.""" + self._data = data + self._query_status = query_status or self._df_status() + + @property + def primary_results(self): + """Mock primary_results.""" + return [self._data] + + @property + def tables_names(self): + """Mock table_names.""" + return ["primary", "QueryCompletionInformation"] + + @property + def tables(self): + """Mock tables.""" + return [self._data, self._query_status] + + def to_dataframe(self): + """Mock to_dataframe.""" + return self._data + + def _df_status(self): + """Mock _df_status.""" + return pd.DataFrame( + data=[["TestStatus", '{"payload": "TestPayload"}']], + columns=["EventTypeName", "Payload"], + ) + + +_TEST_DF = get_test_df() + + +class KustoClient: + """Mock Kusto client.""" + + injected_exception = None + + def __init__(self, *args, **kwargs): + """Initialize client.""" + self.args = args + self.kwargs = kwargs + self._data = _TEST_DF + + def execute(self, query, database=None, *args, **kwargs): + """Mock execute.""" + if self.injected_exception: + if self.injected_exception == KustoApiError: + raise self.injected_exception({"error": "Query failed"}) + raise self.injected_exception("Query failed") + if query is None: + raise ValueError("Query cannot be None") + if database is None: + raise ValueError("Database cannot be None") + if query == "test": + return KustoResponseDataSet(self._data) + + raise ValueError(f"Invalid query {query}") + + def execute_mgmt(self, database, query, *args, **kwargs): + """Mock execute_mgmt.""" + if self.injected_exception: + if self.injected_exception == KustoApiError: + raise self.injected_exception({"error": "Query failed"}) + raise self.injected_exception("Query failed") + if query == "service_error": + raise KustoServiceError("Query failed") + if query == ".show databases": + return KustoResponseDataSet( + pd.DataFrame(["dv1", "dv2"], columns=["DatabaseName"]) + ) + if query.endswith("schema"): + return KustoResponseDataSet( + pd.DataFrame( + data=[ + ["Table1", "TimeGenerated", "System.datetime"], + ["Table1", "TestColumn", "System.string"], + ["Table2", "TimeGenerated", "System.datetime"], + ["Table2", "TestColumn2", "System.string"], + ], + columns=["TableName", "ColumnName", "ColumnType"], + ) + ) + + pytest.fail("Unexpected test case") + + +def create_query_source( + data_families=None, database=None, cluster=None, clusters=None, cluster_groups=None +): + """Create custom query source for testing.""" + metadata = { + "data_environments": ["Kusto"], + } + if cluster: + metadata["cluster"] = cluster + if clusters: + metadata["clusters"] = clusters + if cluster_groups: + metadata["cluster_groups"] = cluster_groups + if data_families: + metadata["data_families"] = data_families + if database: + metadata["database"] = database + return QuerySource( + name="test", + source=QUERY_SOURCE, + defaults={}, + metadata=metadata, + ) + + +class QueryTest(NamedTuple): + """Test configuration for Kusto query tests.""" + + connect_args: dict + query_args: dict + name: str + tests: List[Callable] + exception: Optional[type] = None + injected_exception: Optional[type] = None + + +_TEST_QUERY_ARGS = ( + QueryTest( + name="query-no-db", + connect_args={"connection_str": "https://test.kusto.windows.net"}, + query_args={"query": "test"}, + tests=[], + exception=MsticpyDataQueryError, + ), + QueryTest( + name="simple-query", + connect_args={"cluster": "https://test.kusto.windows.net"}, + query_args={"query": "test", "database": "test_db"}, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-get-db-from-source", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "cluster": "https://ignore.kusto.windows.net", + "query_source": { + "database": "test_db", + "cluster": "https://help.kusto.windows.net", + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-get-db-family-from-source", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "data_families": ["MyKusto.param_test_db"], + "cluster": "help", + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-get-db-family-no-dot-from-source", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "data_families": ["param_test_db"], + "cluster": "help", + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-no-db-or-families-in-source", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "cluster": "help", + }, + }, + tests=[], + exception=MsticpyDataQueryError, + ), + QueryTest( + name="query-get-db-and-cluster-from-source", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster": "https://help.kusto.windows.net", + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-not-connected", + connect_args={}, + query_args={"query": "test"}, + tests=[], + exception=MsticpyNotConnectedError, + ), + QueryTest( + name="kusto-api-error", + connect_args={"cluster": "https://test.kusto.windows.net"}, + query_args={"query": "test", "database": "test_db"}, + tests=[], + injected_exception=KustoApiError, + exception=MsticpyDataQueryError, + ), + QueryTest( + name="kusto-service-error", + connect_args={"cluster": "https://test.kusto.windows.net"}, + query_args={"query": "test", "database": "test_db"}, + tests=[], + injected_exception=KustoServiceError, + exception=MsticpyDataQueryError, + ), + QueryTest( + name="no-db-error", + connect_args={"cluster": "https://test.kusto.windows.net"}, + query_args={"query": "test"}, + tests=[], + injected_exception=KustoServiceError, + exception=MsticpyDataQueryError, + ), + QueryTest( + name="db-specified-in-config", + connect_args={"cluster": "https://g3.kusto.windows.net"}, + query_args={"query": "test"}, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="no-db-error-query-source", + connect_args={"cluster": "https://test.kusto.windows.net"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster": "https://help.kusto.windows.net", + }, + }, + tests=[], + injected_exception=KustoServiceError, + exception=MsticpyDataQueryError, + ), + QueryTest( + name="query-cluster-in-clusters", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "clusters": [ + "https://test.kusto.windows.net", + "https://help.kusto.windows.net", + ], + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-cluster-in-cluster-group", + connect_args={"cluster": "MDE"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster_groups": ["Group1", "Group2"], + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), + QueryTest( + name="query-cluster-mismatch", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster": "https://test.kusto.windows.net", + }, + }, + tests=[], + exception=MsticpyDataQueryError, + ), + QueryTest( + name="query-cluster-not-in-clusters", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "clusters": [ + "https://test.kusto.windows.net", + "https://msticti.kusto.windows.net", + ], + }, + }, + tests=[], + exception=MsticpyDataQueryError, + ), + QueryTest( + name="query-cluster-not-in-cluster-group", + connect_args={"cluster": "help"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster_groups": ["Group1", "Group2"], + }, + }, + tests=[], + exception=MsticpyDataQueryError, + ), + QueryTest( + name="query-cluster-not-in-cluster-group2", + connect_args={"cluster": "g3cluster"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + "cluster_groups": ["Group1", "Group2"], + }, + }, + tests=[], + exception=MsticpyDataQueryError, + ), + # This succeeds because we optimistically try the query + # if the query has no cluster metadata + QueryTest( + name="query-no-query_source-cluster-metadata", + connect_args={"cluster": "g3cluster"}, + query_args={ + "query": "test", + "query_source": { + "database": "test_db", + }, + }, + tests=[ + lambda result: isinstance(result, pd.DataFrame), + ], + ), +) + + +@pytest.mark.parametrize( + "query_config", _TEST_QUERY_ARGS, ids=unit_test_ids(_TEST_QUERY_ARGS) +) +def test_kusto_query(query_config, monkeypatch): + """Test Kusto query with different parameters.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + if query_config.injected_exception: + monkeypatch.setattr( + KustoClient, "injected_exception", query_config.injected_exception + ) + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + test_config = _TEST_CONNECT_ARGS[2] + driver = AzureKustoDriver(**(test_config.init_args)) + if query_config.connect_args: + driver.connect(**(query_config.connect_args)) + + if query_config.exception: + with pytest.raises(query_config.exception): + query_args = query_config.query_args + if "query_source" in query_args: + query_args["query_source"] = create_query_source( + **query_args["query_source"] + ) + driver.query(**query_args) + else: + query_args = query_config.query_args + if "query_source" in query_args: + query_args["query_source"] = create_query_source( + **query_args["query_source"] + ) + result = driver.query(**query_args) + for test in query_config.tests: + print( + "Testcase:", + query_config.name, + test.__code__.co_filename, + "line:", + test.__code__.co_firstlineno, + ) + check.is_true(test(result)) + check.is_instance(result, pd.DataFrame) + + +def test_set_database_query(monkeypatch): + """Test that we can set the database via the set_database method.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + test_config = _TEST_CONNECT_ARGS[2] + driver = AzureKustoDriver(**(test_config.init_args)) + query_config = QueryTest( + name="query-no-db", + connect_args={"connection_str": "https://test.kusto.windows.net"}, + query_args={"query": "test"}, + tests=[], + ) + if query_config.connect_args: + driver.connect(**(query_config.connect_args)) + + driver.set_database("test_db") + query_args = query_config.query_args + result = driver.query(**query_args) + check.is_instance(result, pd.DataFrame) + + +_TEST_CLUSTERS_SUCC = ("MDE", "https://test.kusto.windows.net", "test") + + +def create_cluster_combo_tests(connect_names, query_source_names): + """Create tests for cluster name combos.""" + for cluster in connect_names: + for connect_cl, query_cl in product([cluster], query_source_names): + yield QueryTest( + name=f"success-combo-{connect_cl}-{query_cl}", + connect_args={"cluster": connect_cl}, + query_args={ + "query": "test", + "query_source": { + "data_families": ["MyKusto.param_test_db"], + "cluster": query_cl, + }, + }, + tests=[lambda result: isinstance(result, pd.DataFrame)], + exception=ValueError, + ) + + +_TEST_CLUSTER_ID_COMBOS = tuple( + create_cluster_combo_tests(_TEST_CLUSTERS_SUCC, _TEST_CLUSTERS_SUCC) +) + + +@pytest.mark.parametrize( + "query_config", _TEST_CLUSTER_ID_COMBOS, ids=unit_test_ids(_TEST_CLUSTER_ID_COMBOS) +) +def test_cluster_name_combos(query_config, monkeypatch): + """Test combinations of different cluster name formats.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + driver = AzureKustoDriver() + driver.connect(**(query_config.connect_args)) + + query_args = query_config.query_args + if "query_source" in query_args: + query_args["query_source"] = create_query_source( + **query_args["query_source"] + ) + result = driver.query(**query_args) + for test in query_config.tests: + print( + "Testcase:", + query_config.name, + test.__code__.co_filename, + "line:", + test.__code__.co_firstlineno, + ) + check.is_true(test(result)) + + +_TEST_CLUSTERS_FAIL = ("Help", "https://help.kusto.windows.net", "help") + +_TEST_CLUSTER_ID_COMBOS_FAIL = tuple( + create_cluster_combo_tests(_TEST_CLUSTERS_SUCC, _TEST_CLUSTERS_FAIL) +) + + +@pytest.mark.parametrize( + "query_config", + _TEST_CLUSTER_ID_COMBOS_FAIL, + ids=unit_test_ids(_TEST_CLUSTER_ID_COMBOS_FAIL), +) +def test_cluster_name_combos_fail(query_config, monkeypatch): + """Test failing combinations of different cluster name formats.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + driver = AzureKustoDriver() + driver.connect(**(query_config.connect_args)) + + query_args = query_config.query_args + if "query_source" in query_args: + query_args["query_source"] = create_query_source( + **query_args["query_source"] + ) + with pytest.raises(MsticpyDataQueryError): + driver.query(**query_args) + + +_TEST_GET_DB_NAMES = ( + ({"cluster": "MDE"}, None, None), + (None, None, MsticpyNotConnectedError), + ({"cluster": "MDE"}, KustoServiceError, MsticpyDataQueryError), +) + + +@pytest.mark.parametrize("connect_args, inj_exc, exp_exc", _TEST_GET_DB_NAMES) +def test_kusto_get_database_names(connect_args, inj_exc, exp_exc, monkeypatch): + """Test get_database_names method.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + KustoClient.injected_exception = inj_exc + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + driver = AzureKustoDriver() + if connect_args: + driver.connect(**connect_args) + + if exp_exc: + with pytest.raises(exp_exc): + driver.get_database_names() + else: + result = driver.get_database_names() + check.is_instance(result, list) + check.equal(len(result), 2) + + KustoClient.injected_exception = None + + +_TEST_GET_DB_SCHEMA = ( + ({"cluster": "MDE", "database": "test"}, None, None), + (None, None, MsticpyNotConnectedError), + ({"cluster": "MDE"}, None, ValueError), + ({"cluster": "MDE", "database": "test"}, KustoServiceError, MsticpyDataQueryError), +) + + +@pytest.mark.parametrize("connect_args, inj_exc, exp_exc", _TEST_GET_DB_SCHEMA) +def test_kusto_get_database_schema(connect_args, inj_exc, exp_exc, monkeypatch): + """Test get_database_schema method.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + KustoClient.injected_exception = inj_exc + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + driver = AzureKustoDriver() + if connect_args: + driver.connect(**connect_args) + + if exp_exc: + with pytest.raises(exp_exc): + driver.get_database_schema() + + check.equal(driver.schema, {}) + else: + result = driver.get_database_schema() + check.is_instance(result, dict) + check.equal(len(result), 2) + check.is_in("Table1", result) + check.is_in("Table2", result) + check.equal(result["Table1"]["TimeGenerated"], "datetime") + check.equal(result, driver.schema) + + KustoClient.injected_exception = None + + +_TOTAL_QUERIES = 55 + +_TEST_QUERY_LOAD = ( + ConnectTest( + name="connect-cluster-mde", + init_args={}, + connect_args={"cluster": "MDE"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 22, + lambda connect_queries: all( + q[:2] in ("T1", "T2", "T3", "T4") for q in connect_queries + ), + ], + ), + ConnectTest( + name="connect-cluster-msticti", + init_args={}, + connect_args={"cluster": "msticti"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 18, + lambda connect_queries: all( + q[:2] in ("T2", "T3", "T4") for q in connect_queries + ), + ], + ), + ConnectTest( + name="connect-cluster-msticti-strict", + init_args={"strict_query_match": True}, + connect_args={"cluster": "msticti"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 14, + lambda connect_queries: all(q[:2] in ("T2", "T3") for q in connect_queries), + ], + ), + ConnectTest( + name="connect-cluster-help", + init_args={}, + connect_args={"cluster": "help"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 8, + lambda connect_queries: all(q[:2] in ("T4", "T5") for q in connect_queries), + ], + ), + ConnectTest( + name="connect-cluster-g2", + init_args={}, + connect_args={"cluster": "G2Cluster"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 11, + lambda connect_queries: all(q[:2] in ("T4", "T6") for q in connect_queries), + ], + ), + ConnectTest( + name="connect-cluster-g2-strict", + init_args={"strict_query_match": True}, + connect_args={"cluster": "G2Cluster"}, + tests=[ + lambda load_queries: len(load_queries) == _TOTAL_QUERIES, + lambda connect_queries: len(connect_queries) == 7, + lambda connect_queries: all(q[:2] in ("T6") for q in connect_queries), + ], + ), +) + + +@pytest.mark.parametrize("connect_test", _TEST_QUERY_LOAD, ids=lambda x: x.name) +def test_provider_load_queries(connect_test, monkeypatch): + """Test provider load queries for different cluster configs.""" + monkeypatch.setattr(azure_kusto_driver, "az_connect", az_connect) + monkeypatch.setattr( + azure_kusto_driver, "dataframe_from_result_table", lambda resp: resp + ) + monkeypatch.setattr(azure_kusto_driver, "KustoClient", KustoClient) + + test_queries = get_test_data_path().joinpath("kusto") + with custom_get_config( + monkeypatch=monkeypatch, + settings=_KUSTO_NEW_CONFIG_SETTINGS, + add_modules=[ + "msticpy.common.proxy_settings", + "msticpy.data.drivers.azure_kusto_driver", + ], + ): + qry_prov = QueryProvider( + "Kusto_New", query_paths=[test_queries], **connect_test.init_args + ) + load_queries = qry_prov.list_queries() + qry_prov.connect(**connect_test.connect_args) + connect_queries = qry_prov.list_queries() + print(len(load_queries), len(connect_queries)) + print(connect_queries) + + for test in connect_test.tests: + print( + "Testcase:", + connect_test.name, + test.__code__.co_filename, + "line:", + test.__code__.co_firstlineno, + ) + if "load_queries" in test.__code__.co_varnames: + check.is_true(test(load_queries)) + if "connect_queries" in test.__code__.co_varnames: + check.is_true(test(connect_queries)) diff --git a/tests/data/drivers/test_azure_monitor_driver.py b/tests/data/drivers/test_azure_monitor_driver.py new file mode 100644 index 000000000..50a6586dc --- /dev/null +++ b/tests/data/drivers/test_azure_monitor_driver.py @@ -0,0 +1,433 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""KQL driver query test class.""" +import json +import pickle +import re +from datetime import datetime, timedelta, timezone +from unittest.mock import patch + +import pandas as pd +import pytest +import pytest_check as check +import respx +from azure.core.exceptions import HttpResponseError + +from msticpy.auth.azure_auth_core import AzCredentials +from msticpy.common.exceptions import ( + MsticpyDataQueryError, + MsticpyKqlConnectionError, + MsticpyNoDataSourceError, + MsticpyNotConnectedError, +) +from msticpy.data.core.data_providers import QueryProvider +from msticpy.data.core.query_defns import DataEnvironment +from msticpy.data.core.query_source import QuerySource +from msticpy.data.drivers import azure_monitor_driver +from msticpy.data.drivers.azure_monitor_driver import AzureMonitorDriver + +from ...unit_test_lib import custom_mp_config, get_test_data_path + +# pylint: disable=protected-access, unused-argument, redefined-outer-name + + +@pytest.fixture(scope="module") +def read_schema(): + """Read schema file.""" + with open( + get_test_data_path().joinpath("azmondata/az_mon_schema.json"), + "r", + encoding="utf-8", + ) as schema_file: + file_text = schema_file.read() + print("read test schema length", len(file_text)) + return json.loads(file_text) + + +@pytest.fixture(scope="module") +def read_query_response(): + """Read query file.""" + with open( + get_test_data_path().joinpath("/azmondata/query_response.pkl"), "rb" + ) as query_file: + return pickle.loads(query_file.read()) + + +def get_test_ids(test_params): + """Return test ids for parameterized tests.""" + return [f"{next(iter(param), 'No-params')}" for param, _ in test_params] + + +_TEST_INIT_PARAMS = ( + ({"connection_str": "test"}, ("_def_connection_str", "test")), + ( + {"data_environment": DataEnvironment.MSSentinel_New}, + ("effective_environment", DataEnvironment.MSSentinel.name), + ), +) + + +@pytest.mark.parametrize( + "params, expected", _TEST_INIT_PARAMS, ids=get_test_ids(_TEST_INIT_PARAMS) +) +def test_azmon_driver_init(params, expected): + """Test KqlDriverAZMon init.""" + azmon_driver = AzureMonitorDriver(**params) + check.is_false(azmon_driver.connected) + check.is_none(azmon_driver._query_client) + check.is_none(azmon_driver._ws_config) + check.is_none(azmon_driver._connect_auth_types) + check.is_true(azmon_driver._ua_policy._user_agent.startswith("MSTICPy")) + check.equal(getattr(azmon_driver, expected[0]), expected[1]) + + +class Token: + """Mock token class.""" + + token = "test_token" + + +class Credentials: + """Mock credentials class.""" + + def get_token(self, *args, **kwargs): + """Mock get_token.""" + return Token() + + +_WS_IDS = [ + "a927809c-8142-43e1-96b3-4ad87cfe95a3", + "a927809c-8142-43e1-96b3-4ad87cfe95a4", +] + +_TEST_CONNECT_PARAMS = ( + ( + {"auth_types": ["cli", "environment", "msi"]}, + [("_connect_auth_types", ["cli", "environment", "msi"])], + ), + ({"auth_types": "cli"}, [("_connect_auth_types", ["cli"])]), + ({"tenant_id": "test"}, [("_az_tenant_id", "test")]), + ({"connection_str": "test"}, [(None, MsticpyKqlConnectionError)]), + ( + {"mp_az_auth": ["cli", "environment", "msi"]}, + [("_connect_auth_types", ["cli", "environment", "msi"])], + ), + ({"mp_az_auth": True}, [("_connect_auth_types", None)]), + ( + {"workspace": "MyTestWS"}, + [("_workspace_id", "a927809c-8142-43e1-96b3-4ad87cfe95a3")], + ), + ({}, [("_workspace_id", "52b1ab41-869e-4138-9e40-2a4457f09bf3")]), + ( + {"workspaces": ["MyTestWS", "MyTestWS2"]}, + [ + ("_workspace_ids", _WS_IDS), + ("_az_tenant_id", "69d28fd7-42a5-48bc-a619-af56397b9f28"), + ], + ), + ( + {"workspace_ids": _WS_IDS, "tenant_id": "69d28fd7-42a5-48bc-a619-af56397b9f28"}, + [ + ("_workspace_ids", _WS_IDS), + ("_az_tenant_id", "69d28fd7-42a5-48bc-a619-af56397b9f28"), + ], + ), + ({"workspace_ids": _WS_IDS}, [(None, MsticpyKqlConnectionError)]), +) + + +@respx.mock +@pytest.mark.parametrize( + "params, expected", _TEST_CONNECT_PARAMS, ids=get_test_ids(_TEST_CONNECT_PARAMS) +) +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_azmon_driver_connect(az_connect, params, expected, read_schema): + """Test KqlDriverAZMon connect.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=200, json=read_schema + ) + respx.get(re.compile(r"https://api\.loganalytics\.io.*")).respond( + status_code=200, json=read_schema + ) + + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + azmon_driver = AzureMonitorDriver() + exception_expected = next(iter(expected))[1] + if isinstance(exception_expected, type) and issubclass( + exception_expected, Exception + ): + with pytest.raises(exception_expected): + azmon_driver.connect(**params) + else: + azmon_driver.connect(**params) + + check.is_true(azmon_driver.connected) + check.is_not_none(azmon_driver._query_client) + # check.is_not_none(azmon_driver._ws_config) + + # check.is_none(azmon_driver._connect_auth_types) + for exp in expected: + if isinstance(exp[1], list): + for item in exp[1]: + check.is_in(item, getattr(azmon_driver, exp[0])) + else: + check.equal(getattr(azmon_driver, exp[0]), exp[1]) + + +@respx.mock +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_get_schema(az_connect, read_schema, monkeypatch): + """Test KqlDriverAZMon get_schema.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=200, json=read_schema + ) + respx.get(re.compile(r"https://api\.loganalytics\.io.*")).respond( + status_code=200, json=read_schema + ) + monkeypatch.setattr(azure_monitor_driver, "LogsQueryClient", LogsQueryClient) + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + azmon_driver = AzureMonitorDriver(debug=True) + azmon_driver.connect(workspace="MyTestWS") + assert isinstance(azmon_driver._query_client, LogsQueryClient) + check.is_not_none(azmon_driver.schema) + check.equal(len(azmon_driver.schema), 17) + check.is_in("AppServiceAntivirusScanAuditLogs", azmon_driver.schema) + check.is_in("AzureActivity", azmon_driver.schema) + for col in ("TenantId", "SourceSystem", "CallerIpAddress", "OperationId"): + check.is_in(col, azmon_driver.schema["AzureActivity"]) + check.is_in(azmon_driver.schema["AzureActivity"][col], ("string", "guid")) + + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=404, content=b"not found" + ) + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + azmon_driver = AzureMonitorDriver(debug=True) + azmon_driver.connect(workspace="MyTestWS") + check.is_false(azmon_driver.schema) + + +def test_query_not_connected(): + """Test KqlDriverAZMon query when not connected.""" + with pytest.raises(MsticpyNotConnectedError): + azmon_driver = AzureMonitorDriver() + azmon_driver.query("AzureActivity") + + +QUERY_SOURCE = { + "args": { + "query": " let start = datetime({start}); let end = datetime({end}); " + "{table} {event_filter} {query_project} | where " + "{subscription_filter} | where Computer {host_op} " + '"{host_name}" | where TimeGenerated >= start | where ' + "TimeGenerated <= end {add_query_items}" + }, + "description": "Retrieves list of processes on a host", + "metadata": { + "pivot": {"direct_func_entities": ["Host"], "short_name": "processes"} + }, + "parameters": { + "host_name": {"description": "Name of host", "type": "str"}, + "host_op": { + "default": "has", + "description": "The hostname match operator", + "type": "str", + }, + "table": { + "default": "SecurityEvent", + "description": "The table to query", + "type": "str", + }, + }, +} + + +@respx.mock +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_query_unknown_table(az_connect, read_schema, monkeypatch): + """Test KqlDriverAZMon query when not connected.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=200, json=read_schema + ) + respx.get(re.compile(r"https://api\.loganalytics\.io.*")).respond( + status_code=200, json=read_schema + ) + monkeypatch.setattr(azure_monitor_driver, "LogsQueryClient", LogsQueryClient) + with pytest.raises(MsticpyNoDataSourceError): + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + azmon_driver = AzureMonitorDriver(debug=True) + azmon_driver.connect(workspace="MyTestWS") + assert azmon_driver.schema is not None + assert isinstance(azmon_driver._query_client, LogsQueryClient) + query_source = QuerySource( + name="my_test_query", source=QUERY_SOURCE, defaults={}, metadata={} + ) + azmon_driver.query(query="UnknownTable", query_source=query_source) + + +@respx.mock +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_load_provider(az_connect, read_schema, monkeypatch): + """Test KqlDriverAZMon query when not connected.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=200, json=read_schema + ) + respx.get(re.compile(r"https://api\.loganalytics\.io.*")).respond( + status_code=200, json=read_schema + ) + + monkeypatch.setattr(azure_monitor_driver, "LogsQueryClient", LogsQueryClient) + + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + query_prov = QueryProvider("MSSentinel_New", debug=True) + query_prov.connect(workspace="MyTestWS") + assert isinstance(query_prov._query_provider._query_client, LogsQueryClient) + + check.greater(len(query_prov.list_queries()), 100) + check.equal(len(query_prov.schema), 17) + + +class LogsQueryClient: + """Mock LogsQueryClient class.""" + + with open( + get_test_data_path().joinpath("azmondata/query_response.pkl"), "rb" + ) as query_file: + _data = pickle.loads(query_file.read()) + + def __init__(self, *args, **kwargs): + """Mock LogsQueryClient class.""" + self.args = args + self.kwargs = kwargs + self.query_workspace_args = {} + + def query_workspace(self, query, timespan=None, **kwargs): + """Mock query_workspace method.""" + self.query_workspace_args = {"query": query, "timespan": timespan, **kwargs} + + if query == "HttpResponseError": + raise HttpResponseError( + message="LA response error", + ) + if query == "Exception": + raise ValueError("Unknown exception") + return self._data + + +@respx.mock +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_queries(az_connect, read_schema, monkeypatch): + """Test KqlDriverAZMon queries.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + respx.get(re.compile(r"https://management\.azure\.com/.*")).respond( + status_code=200, json=read_schema + ) + respx.get(re.compile(r"https://api\.loganalytics\.io.*")).respond( + status_code=200, json=read_schema + ) + monkeypatch.setattr(azure_monitor_driver, "LogsQueryClient", LogsQueryClient) + + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + query_prov = QueryProvider("MSSentinel_New", debug=True) + query_prov.connect(workspace="MyTestWS") + assert isinstance(query_prov._query_provider._query_client, LogsQueryClient) + query = "Testtable | take 10" + results = query_prov.exec_query(query) + + check.is_in("credential", query_prov._query_provider._query_client.kwargs) + check.is_in("endpoint", query_prov._query_provider._query_client.kwargs) + check.is_in("proxies", query_prov._query_provider._query_client.kwargs) + + query_ws_args = query_prov._query_provider._query_client.query_workspace_args + check.equal(query_ws_args["query"], query) + check.is_not_none(query_ws_args["workspace_id"]) + check.equal(query_ws_args["server_timeout"], 300) + check.is_none(query_ws_args["additional_workspaces"]) + check.is_instance(results, pd.DataFrame) + check.equal(len(results), 3) + + # fail because not in schema + with pytest.raises(MsticpyNoDataSourceError): + query_prov.Azure.get_vmcomputer_for_ip( + ip_address="192.1.2.3", + start=datetime.now(tz=timezone.utc) - timedelta(1), + end=datetime.now(tz=timezone.utc), + ) + # should succeed + results = query_prov.Azure.list_azure_activity_for_ip( + ip_address_list=["192.1.2.3"], + start=datetime.now(tz=timezone.utc) - timedelta(1), + end=datetime.now(tz=timezone.utc), + ) + check.is_instance(results, pd.DataFrame) + check.equal(len(results), 3) + + # Fail due to service errors + with pytest.raises(MsticpyDataQueryError): + query_prov.exec_query("HttpResponseError") + + with pytest.raises(MsticpyDataQueryError): + query_prov.exec_query("Exception") + + +@patch("msticpy.data.drivers.azure_monitor_driver.az_connect") +def test_query_multiple_workspaces(az_connect, monkeypatch): + """Test KqlDriverAZMon query when not connected.""" + az_connect.return_value = AzCredentials(legacy=None, modern=Credentials()) + + monkeypatch.setattr(azure_monitor_driver, "LogsQueryClient", LogsQueryClient) + + query_prov = QueryProvider("MSSentinel_New", debug=True) + with pytest.raises(MsticpyKqlConnectionError): + query_prov.connect( + workspace_ids=[ + "a927809c-8142-43e1-96b3-4ad87cfe95a3", + "a927809c-8142-43e1-96b3-4ad87cfe95a4", + ] + ) + + query_prov.connect( + tenant_id="72f988bf-86f1-41af-91ab-2d7cd011db49", + workspace_ids=[ + "a927809c-8142-43e1-96b3-4ad87cfe95a3", + "a927809c-8142-43e1-96b3-4ad87cfe95a4", + ], + ) + query = "Testtable | take 10" + results = query_prov.exec_query(query) + check.is_instance(results, pd.DataFrame) + check.equal(len(results), 3) + + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + query_prov.connect( + tenant_id="72f988bf-86f1-41af-91ab-2d7cd011db49", + workspaces=["MyTestWS", "MyTestWS2"], + ) + query = "Testtable | take 10" + results = query_prov.exec_query(query) + check.is_instance(results, pd.DataFrame) + check.equal(len(results), 3) diff --git a/tests/data/test_dataqueries.py b/tests/data/test_dataqueries.py index 847a16ce8..15223d736 100644 --- a/tests/data/test_dataqueries.py +++ b/tests/data/test_dataqueries.py @@ -20,13 +20,10 @@ from msticpy.common import pkg_config from msticpy.common.exceptions import MsticpyException -from msticpy.data.core.data_providers import ( - DriverBase, - QueryContainer, - QueryProvider, - _calc_split_ranges, -) +from msticpy.data.core.data_providers import QueryProvider, _calc_split_ranges +from msticpy.data.core.query_container import QueryContainer from msticpy.data.core.query_source import QuerySource +from msticpy.data.drivers.driver_base import DriverBase, DriverProps from ..unit_test_lib import get_test_data_path @@ -45,7 +42,7 @@ def __init__(self, **kwargs): self._kwargs = kwargs self._loaded = True self._connected = False - self.public_attribs = {"test": self._TEST_ATTRIB} + self.set_driver_property(DriverProps.PUBLIC_ATTRS, {"test": self._TEST_ATTRIB}) self.svc_queries = {} self.has_driver_queries = True diff --git a/tests/init/test_nbinit.py b/tests/init/test_nbinit.py index 8e6ae05a0..24c3d5a45 100644 --- a/tests/init/test_nbinit.py +++ b/tests/init/test_nbinit.py @@ -221,6 +221,7 @@ def test_check_config(conf_file, expected, tmp_path, monkeypatch): os.chdir(str(tmp_path)) with custom_mp_config(settings_file, path_check=False): + monkeypatch.setattr(nbinit, "current_config_path", lambda: None) monkeypatch.setattr(nbinit, "is_in_aml", lambda: True) monkeypatch.setattr(azure_ml_tools, "get_aml_user_folder", lambda: tmp_path) result = _get_or_create_config() diff --git a/tests/msticpyconfig-test.yaml b/tests/msticpyconfig-test.yaml index cf1339eed..32c8e4859 100644 --- a/tests/msticpyconfig-test.yaml +++ b/tests/msticpyconfig-test.yaml @@ -16,6 +16,12 @@ AzureSentinel: SubscriptionId: "cd928da3-dcde-42a3-aad7-d2a1268c2f48" ResourceGroup: ABC WorkspaceName: Workspace1 + MyTestWS2: + WorkspaceId: "a927809c-8142-43e1-96b3-4ad87cfe95a4" + TenantId: "69d28fd7-42a5-48bc-a619-af56397b9f28" + SubscriptionId: "cd928da3-dcde-42a3-aad7-d2a1268c2f48" + ResourceGroup: ABC + WorkspaceName: Workspace2 WSName: WorkspaceId: "a927809c-8142-43e1-96b3-4ad87cfe95a3" TenantId: "69d28fd7-42a5-48bc-a619-af56397b9f28" diff --git a/tests/testdata/azmondata/az_mon_raw_query.json b/tests/testdata/azmondata/az_mon_raw_query.json new file mode 100644 index 000000000..cfff9d5a5 --- /dev/null +++ b/tests/testdata/azmondata/az_mon_raw_query.json @@ -0,0 +1,255 @@ +{ + "tables": [ + { + "name": "PrimaryResult", + "columns": [ + { + "name": "TenantId", + "type": "string" + }, + { + "name": "TimeGenerated", + "type": "datetime" + }, + { + "name": "DisplayName", + "type": "string" + }, + { + "name": "AlertName", + "type": "string" + }, + { + "name": "AlertSeverity", + "type": "string" + }, + { + "name": "Description", + "type": "string" + }, + { + "name": "ProviderName", + "type": "string" + }, + { + "name": "VendorName", + "type": "string" + }, + { + "name": "VendorOriginalId", + "type": "string" + }, + { + "name": "SystemAlertId", + "type": "string" + }, + { + "name": "ResourceId", + "type": "string" + }, + { + "name": "SourceComputerId", + "type": "string" + }, + { + "name": "AlertType", + "type": "string" + }, + { + "name": "ConfidenceLevel", + "type": "string" + }, + { + "name": "ConfidenceScore", + "type": "real" + }, + { + "name": "IsIncident", + "type": "bool" + }, + { + "name": "StartTime", + "type": "datetime" + }, + { + "name": "EndTime", + "type": "datetime" + }, + { + "name": "ProcessingEndTime", + "type": "datetime" + }, + { + "name": "RemediationSteps", + "type": "string" + }, + { + "name": "ExtendedProperties", + "type": "string" + }, + { + "name": "Entities", + "type": "string" + }, + { + "name": "SourceSystem", + "type": "string" + }, + { + "name": "WorkspaceSubscriptionId", + "type": "string" + }, + { + "name": "WorkspaceResourceGroup", + "type": "string" + }, + { + "name": "ExtendedLinks", + "type": "string" + }, + { + "name": "ProductName", + "type": "string" + }, + { + "name": "ProductComponentName", + "type": "string" + }, + { + "name": "AlertLink", + "type": "string" + }, + { + "name": "Status", + "type": "string" + }, + { + "name": "CompromisedEntity", + "type": "string" + }, + { + "name": "Tactics", + "type": "string" + }, + { + "name": "Techniques", + "type": "string" + }, + { + "name": "Type", + "type": "string" + } + ], + "rows": [ + [ + "52b1ab41-869e-4138-9e40-2a4457f09bf0", + "2023-02-20T11:04:02.6371632Z", + "Linux security related process termination activity detected", + "Linux security related process termination activity detected", + "Medium", + "This query will alert on any attempts to terminate processes related to security monitoring on the host.", + "ASI Scheduled Alerts", + "Microsoft", + "a3ce3da5-93a4-42f7-ae0f-0fee36a16527", + "c19a6ccd-40a6-9ef9-5a8b-598bcc3c53b0", + "", + "", + "52b1ab41-869e-4138-9e40-2a4457f09bf0_1e65fb27-5c2a-4e44-80da-8b5f9d8fdb24", + "", + null, + false, + "2022-01-06T13:01:34Z", + "2022-01-06T13:04:53Z", + "2023-02-20T11:04:02.5649668Z", + "", + "{\"Query Period\":\"05:00:00\",\"Trigger Operator\":\"GreaterThan\",\"Trigger Threshold\":\"0\",\"Correlation Id\":\"7c1a062d-f817-41fa-87f0-c3a1c87a441f\",\"Search Query Results Overall Count\":\"3\",\"Data Sources\":\"[]\",\"Query\":\"// The query_now parameter represents the time (in UTC) at which the scheduled analytics rule ran to produce this alert.\\nset query_now = datetime(2023-02-20T10:58:58.1280000Z);\\nSummarizedLog4JEvents | where EventType == \\\"Linux security related process termination activity detected\\\"\\n| extend Account_0_Name = User\\n| extend Host_0_HostName = Host\",\"Query Start Time UTC\":\"2023-02-20 05:58:58Z\",\"Query End Time UTC\":\"2023-02-20 10:58:59Z\",\"Analytic Rule Ids\":\"[\\\"1e65fb27-5c2a-4e44-80da-8b5f9d8fdb24\\\"]\",\"Event Grouping\":\"SingleAlert\",\"Analytic Rule Name\":\"Linux security related process termination activity detected\",\"ProcessedBySentinel\":\"True\",\"Alert generation status\":\"Full alert created\"}", + "", + "Detection", + "40dcc8bf-0478-4f3b-b275-ed0a94f2c013", + "asihuntomsworkspacerg", + "", + "Azure Sentinel", + "Scheduled Alerts", + "", + "New", + "", + "Unknown", + "", + "SecurityAlert" + ], + [ + "52b1ab41-869e-4138-9e40-2a4457f09bf0", + "2023-02-20T15:45:37.4668968Z", + "Linux security related process termination activity detected", + "Linux security related process termination activity detected", + "Medium", + "This query will alert on any attempts to terminate processes related to security monitoring on the host.", + "ASI Scheduled Alerts", + "Microsoft", + "3d90a0ee-b031-425c-b720-cb296b927d2f", + "4422fdcc-ed33-8b4a-fec8-1cfbf799c74b", + "", + "", + "52b1ab41-869e-4138-9e40-2a4457f09bf0_2d31af09-e46f-4542-891a-f97dcadd0725", + "", + null, + false, + "2022-01-06T13:01:34Z", + "2022-01-06T13:04:53Z", + "2023-02-20T15:45:37.388088Z", + "", + "{\"Query Period\":\"05:00:00\",\"Trigger Operator\":\"GreaterThan\",\"Trigger Threshold\":\"0\",\"Correlation Id\":\"b0d7815a-b84b-467d-ba56-e5a1a9dfb9d1\",\"Search Query Results Overall Count\":\"3\",\"Data Sources\":\"[]\",\"Query\":\"// The query_now parameter represents the time (in UTC) at which the scheduled analytics rule ran to produce this alert.\\nset query_now = datetime(2023-02-20T15:40:34.6400000Z);\\nSummarizedLog4JEvents | where EventType == \\\"Linux security related process termination activity detected\\\"\\n| extend Account_0_Name = User\\n| extend Host_0_HostName = Host\",\"Query Start Time UTC\":\"2023-02-20 10:40:34Z\",\"Query End Time UTC\":\"2023-02-20 15:40:35Z\",\"Analytic Rule Ids\":\"[\\\"2d31af09-e46f-4542-891a-f97dcadd0725\\\"]\",\"Event Grouping\":\"SingleAlert\",\"Analytic Rule Name\":\"Linux security related process termination activity detected\",\"ProcessedBySentinel\":\"True\",\"Alert generation status\":\"Full alert created\"}", + "", + "Detection", + "40dcc8bf-0478-4f3b-b275-ed0a94f2c013", + "asihuntomsworkspacerg", + "", + "Azure Sentinel", + "Scheduled Alerts", + "", + "New", + "", + "Unknown", + "", + "SecurityAlert" + ], + [ + "52b1ab41-869e-4138-9e40-2a4457f09bf0", + "2023-02-20T21:11:02.2810341Z", + "Linux security related process termination activity detected", + "Linux security related process termination activity detected", + "Medium", + "This query will alert on any attempts to terminate processes related to security monitoring on the host.", + "ASI Scheduled Alerts", + "Microsoft", + "5261283a-bc81-462a-a69a-63809f2bec03", + "24242093-7022-7aaf-763c-4d651ffd94ff", + "", + "", + "52b1ab41-869e-4138-9e40-2a4457f09bf0_1e65fb27-5c2a-4e44-80da-8b5f9d8fdb24", + "", + null, + false, + "2022-01-06T13:01:34Z", + "2022-01-06T13:04:53Z", + "2023-02-20T21:11:02.241001Z", + "", + "{\"Query Period\":\"05:00:00\",\"Trigger Operator\":\"GreaterThan\",\"Trigger Threshold\":\"0\",\"Correlation Id\":\"1632aa6c-44f2-4d12-87c7-5968a7e9fd0b\",\"Search Query Results Overall Count\":\"3\",\"Data Sources\":\"[]\",\"Query\":\"// The query_now parameter represents the time (in UTC) at which the scheduled analytics rule ran to produce this alert.\\nset query_now = datetime(2023-02-20T20:58:58.1280000Z);\\nSummarizedLog4JEvents | where EventType == \\\"Linux security related process termination activity detected\\\"\\n| extend Account_0_Name = User\\n| extend Host_0_HostName = Host\",\"Query Start Time UTC\":\"2023-02-20 15:58:58Z\",\"Query End Time UTC\":\"2023-02-20 20:58:59Z\",\"Analytic Rule Ids\":\"[\\\"1e65fb27-5c2a-4e44-80da-8b5f9d8fdb24\\\"]\",\"Event Grouping\":\"SingleAlert\",\"Analytic Rule Name\":\"Linux security related process termination activity detected\",\"ProcessedBySentinel\":\"True\",\"Alert generation status\":\"Full alert created\"}", + "", + "Detection", + "40dcc8bf-0478-4f3b-b275-ed0a94f2c013", + "asihuntomsworkspacerg", + "", + "Azure Sentinel", + "Scheduled Alerts", + "", + "New", + "", + "Unknown", + "", + "SecurityAlert" + ] + ] + } + ] +} \ No newline at end of file diff --git a/tests/testdata/azmondata/az_mon_schema.json b/tests/testdata/azmondata/az_mon_schema.json new file mode 100644 index 000000000..ffb29c49b --- /dev/null +++ b/tests/testdata/azmondata/az_mon_schema.json @@ -0,0 +1,2464 @@ +{ + "value": [ + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Classic", + "name": "AuditLog_CL", + "tableType": "CustomLog", + "columns": [], + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "MG", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "ManagementGroupName", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "Computer", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "RawData", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AuditLog_CL", + "name": "AuditLog_CL" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AppServiceAntivirusScanAuditLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "Time when event is generated", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ScanStatus", + "type": "string", + "description": "Status of the scan", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalFilesScanned", + "type": "int", + "description": "Total number of scanned files", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "NumberOfInfectedFiles", + "type": "int", + "description": "Total number of files infected with virus", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ListOfInfectedFiles", + "type": "string", + "description": "List of each virus file path", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ErrorMessage", + "type": "string", + "description": "Error Message", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AppServiceAntivirusScanAuditLogs", + "name": "AppServiceAntivirusScanAuditLogs" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AirflowDagProcessingLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) of the log.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The name of the operation represented by this event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Category", + "type": "string", + "description": "The category of the log that belongs to Airflow application.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "The correlation id of the event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DataFactoryName", + "type": "string", + "description": "The name of the Data factory.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "IntegrationRuntimeName", + "type": "string", + "description": "The name of the Integration runtime.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Message", + "type": "string", + "description": "The application log of the Airflow event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AirflowDagProcessingLogs", + "name": "AirflowDagProcessingLogs" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "ADFAirflowTaskLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) of the log.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The name of the operation represented by this event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Category", + "type": "string", + "description": "The category of the log that belongs to Airflow application.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "The correlation id of the event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DataFactoryName", + "type": "string", + "description": "The name of the Data factory.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "IntegrationRuntimeName", + "type": "string", + "description": "The name of the Integration runtime.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DagId", + "type": "string", + "description": "The dag ID of the Airflow task run.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TaskId", + "type": "string", + "description": "The task ID of the Airflow task run.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Message", + "type": "string", + "description": "The application log of the Airflow event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/ADFAirflowTaskLogs", + "name": "ADFAirflowTaskLogs" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "WVDConnectionNetworkData", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) when the network event was generated on the VM", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "The correlation ID for the activity", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EstRoundTripTimeInMs", + "type": "int", + "description": "The average of estimated network round trip times (milliseconds) in the last evaluated connection time interval", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EstAvailableBandwidthKBps", + "type": "int", + "description": "The average of estimated network bandwidth (kilobyte per second) in the last evaluated connection time interval", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/WVDConnectionNetworkData", + "name": "WVDConnectionNetworkData" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AppPlatformSystemLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) when the log is collected by Azure Spring Cloud", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ServiceName", + "type": "string", + "description": "The service name that emitted the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "InstanceName", + "type": "string", + "description": "The instance name that emitted the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Level", + "type": "string", + "description": "The level of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Thread", + "type": "string", + "description": "The thread of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Logger", + "type": "string", + "description": "The logger of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Log", + "type": "string", + "description": "The log of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Stack", + "type": "string", + "description": "The stack of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "LogType", + "type": "string", + "description": "The type of the log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Category", + "type": "string", + "description": "Log Category", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The name of the operation represented by this event", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AppPlatformSystemLogs", + "name": "AppPlatformSystemLogs" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Classic", + "name": "MITREBrawlSysmon_CL", + "tableType": "CustomLog", + "columns": [ + { + "name": "data_model_fields_previous_creation_utc_time_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_file_name_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_creation_utc_time_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "game_id_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "_uuid_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "_version_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "path_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "type_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_dest_ipv6_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_exe_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_dest_fqdn_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_record_number_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_pid_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_keywords_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_fqdn_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_log_name_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_transport_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_event_code_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_dest_port_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_dest_port_name_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_log_type_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_severity_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_image_path_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_hostname_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_src_port_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_src_port_name_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_utc_time_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_uuid_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_op_code_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_user_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_process_guid_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_src_ipv6_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_dest_ip_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_src_ip_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_initiated_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_src_fqdn_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_action_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_object_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "_timestamp_t", + "type": "datetime", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "host_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_parent_process_guid_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_parent_exe_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_current_directory_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_logon_id_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_command_line_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_logon_guid_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_parent_image_path_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_parent_command_line_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_hash_SHA256_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_hash_MD5_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_hash_SHA1_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_hash_IMPHASH_g", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_terminal_session_id_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_integrity_level_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "data_model_fields_ppid_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + } + ], + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "MG", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "ManagementGroupName", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "Computer", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "RawData", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/MITREBrawlSysmon_CL", + "name": "MITREBrawlSysmon_CL" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "DatabricksDeltaPipelines", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp of the action (UTC).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The action, such as login, logout, read, write, etc.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationVersion", + "type": "string", + "description": "The Databricks schema version of the diagnostic log format.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Category", + "type": "string", + "description": "The service that logged the request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Identity", + "type": "string", + "description": "Information about the user that makes the requests.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceIPAddress", + "type": "string", + "description": "The IP address of the source request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "LogId", + "type": "string", + "description": "The unique identifier for the log messages.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ServiceName", + "type": "string", + "description": "The service of the source request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "UserAgent", + "type": "string", + "description": "The browser or API client used to make the request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SessionId", + "type": "string", + "description": "Session ID of the action.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ActionName", + "type": "string", + "description": "The action of the request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RequestId", + "type": "string", + "description": "Unique request ID.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Response", + "type": "string", + "description": "The HTTP response to the request, including error message (if applicable), result, and statusCode.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RequestParams", + "type": "string", + "description": "Parameter key-value pairs used in the event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/DatabricksDeltaPipelines", + "name": "DatabricksDeltaPipelines" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AgriFoodFarmOperationLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "Timestamp (in UTC) when the log was created.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Category", + "type": "string", + "description": "Logs generated as a result of operations executed using FarmBeats APIs are grouped into categories. Categories in FarmBeats are logical groupings based on either the data source the underlying APIs fetch data from or on the basis of hierarchy of entities in FarmBeats.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The operation name for which the log entry was created.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RequestUri", + "type": "string", + "description": "URI of the API request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "FarmerId", + "type": "string", + "description": "Farmer ID associated with the request, wherever applicable.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DataPlaneResourceId", + "type": "string", + "description": "ID that uniquely identifies a FarmBeats resource such as a Farm, Farmer, Boundary etc.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CallerIpAddress", + "type": "string", + "description": "IP address of the client that made the request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Level", + "type": "string", + "description": "The severity level of the event, will be one of Informational, Warning, Error, or Critical.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResultSignature", + "type": "int", + "description": "HTTP status of API request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResultType", + "type": "string", + "description": "Result of the REST API request.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResultDescription", + "type": "string", + "description": "Additional details about the result, when available.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Location", + "type": "string", + "description": "The region of the resource emitting the event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "Unique identifier to be used to correlate logs, when available.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DurationMs", + "type": "real", + "description": "Time it took to service the REST API request, in milliseconds.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ApplicationId", + "type": "string", + "description": "ApplicationId in identity claims.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ObjectId", + "type": "string", + "description": "ObjectId in identity claims.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ClientTenantId", + "type": "string", + "description": "TenantId in identity claims.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RequestBody", + "type": "dynamic", + "description": "Request body of the API calls.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AgriFoodFarmOperationLogs", + "name": "AgriFoodFarmOperationLogs" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "McasShadowItReporting", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "StreamName", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "MachineName", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "MachineId", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalEvents", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "BlockedEvents", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "UploadedBytes", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalBytes", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DownloadedBytes", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "IpAddress", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "UserName", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EnrichedUserName", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppName", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppId", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppInstance", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppCategory", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppTags", + "type": "dynamic", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AppScore", + "type": "int", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "SecurityInsights" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/McasShadowItReporting", + "name": "McasShadowItReporting" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Classic", + "name": "uploaddemo_CL", + "tableType": "CustomLog", + "columns": [ + { + "name": "Unnamed_0_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "TimeGenerated_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "FlowStartTime_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "FlowEndTime_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "FlowIntervalEndTime_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "FlowType_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "ResourceGroup", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "VMName_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "VMIPAddress_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "PublicIPs_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "SrcIP_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "DestIP_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "L4Protocol_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "L7Protocol_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "DestPort_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "FlowDirection_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "AllowedOutFlows_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "AllowedInFlows_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "DeniedInFlows_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "DeniedOutFlows_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "RemoteRegion_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "VMRegion_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "AllExtIPs_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "TotalAllowedFlows_s", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + } + ], + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "MG", + "type": "guid", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "ManagementGroupName", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "Computer", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + }, + { + "name": "RawData", + "type": "string", + "isDefaultDisplay": false, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/uploaddemo_CL", + "name": "uploaddemo_CL" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "DynamicSummary", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) of when the event was ingested to Azure Monitor.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AzureTenantId", + "type": "string", + "description": "The AAD tenant ID to which this DynamicSummary table belongs.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryId", + "type": "string", + "description": "Summary unique ID.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryItemId", + "type": "string", + "description": "Summary item unique ID.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryName", + "type": "string", + "description": "The Summary display name, unique within workspace.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RelationName", + "type": "string", + "description": "The original data source name.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RelationId", + "type": "string", + "description": "The original data source ID", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SearchKey", + "type": "string", + "description": "SearchKey is used to optimize query performance when using DynamicSummary for joins with other data. For example, enable a column with IP addresses to be the designated SearchKey field, then use this field to join in other event tables by IP address.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CreatedBy", + "type": "dynamic", + "description": "The JSON object with the user who created summary, including: object ID, email and name.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CreatedTimeUTC", + "type": "datetime", + "description": "The time (UTC) when the summary was created.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "UpdatedBy", + "type": "dynamic", + "description": "The JSON object with the user who updated summary, including: object ID, email and name.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "UpdatedTimeUTC", + "type": "datetime", + "description": "The time (UTC) when the summary was updated.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryDescription", + "type": "string", + "description": "The description provided by user.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Tactics", + "type": "dynamic", + "description": "MITRE ATT&CK tactics are what attackers are trying to achieve. For example, exfiltration.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Techniques", + "type": "dynamic", + "description": "MITRE ATT&CK techniques are how those tactics are accomplished.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryStatus", + "type": "string", + "description": "Active or deleted.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceInfo", + "type": "dynamic", + "description": "The JSON object with the data producer info, including source, name, version.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Query", + "type": "string", + "description": "This is the query that was used to generate the result.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "QueryStartDate", + "type": "datetime", + "description": "Events that occurred after this datetime will be included in the result.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "QueryEndDate", + "type": "datetime", + "description": "Events that occurred before this datetime will be included in the result.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EventTimeUTC", + "type": "datetime", + "description": "The time (UTC) when the summary item occurred originally.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ObservableType", + "type": "string", + "description": "Observables are stateful events ot properties that are related to the operation of computing system, which are helpful in identifying indicators of compromise. For example, login.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ObservableValue", + "type": "string", + "description": "Value for observable type, such as: anomalous RDP activity.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "PackedContent", + "type": "dynamic", + "description": "The JSON object has packed columns which can be generated by using KQL pack_all().", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SummaryDataType", + "type": "string", + "description": "This flag is used to tell if the record is either a summary level or a summary item level record.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "SecurityInsights" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/DynamicSummary", + "name": "DynamicSummary" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "ADPAudit", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The timestamp (UTC) of when the log record was generated.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The operation associated with the log record.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationVersion", + "type": "string", + "description": "The API version against which the operation was performed.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResultType", + "type": "string", + "description": "The result of the audit operation.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Identity", + "type": "dynamic", + "description": "Active Directory identity claims", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TraceContext", + "type": "dynamic", + "description": "W3C Trace Context information used for event correlation.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "Internal ADP correlation ID used in support scenarios.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Location", + "type": "string", + "description": "The location (region) of the resource.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Properties", + "type": "dynamic", + "description": "Additional properties related to the audit event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/ADPAudit", + "name": "ADPAudit" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "CDBPartitionKeyStatistics", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "Timestamp (in\u00a0UTC)\u00a0when\u00a0statistics for this logical partition key were generated.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "AccountName", + "type": "string", + "description": "The name of the Cosmos DB account\u00a0containing the dataset for which partition key stats were generated.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RegionName", + "type": "string", + "description": "The Azure region\u00a0from which statistics\u00a0for this partition\u00a0were\u00a0retrieved.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "PartitionKey", + "type": "string", + "description": "The logical partition key for which\u00a0storage\u00a0statistics were\u00a0retrieved.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SizeKb", + "type": "int", + "description": "The storage size (in KB) for the logical partition key within the physical partition.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DatabaseName", + "type": "string", + "description": "The\u00a0name of the\u00a0Cosmos DB\u00a0database,\u00a0which\u00a0contains\u00a0the partition.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CollectionName", + "type": "string", + "description": "The name of the Cosmos DB\u00a0collection,\u00a0which\u00a0contains\u00a0the partition.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/CDBPartitionKeyStatistics", + "name": "CDBPartitionKeyStatistics" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "ADXTableDetails", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "The time (UTC) at which this event was generated.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The name of this operation.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "The client request id.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "DatabaseName", + "type": "string", + "description": "The name of the database.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TableName", + "type": "string", + "description": "The name of the table.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EntityType", + "type": "string", + "description": "The type of the table. Can be Table or MaterializedView.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalExtentSize", + "type": "real", + "description": "The total size of extents (compressed size + index size) in the table (in bytes).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalOriginalSize", + "type": "real", + "description": "The total original size of data in the table (in bytes).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "HotExtentSize", + "type": "real", + "description": "The total size of extents (compressed size + index size) in the table, stored in the hot cache (in bytes).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RetentionPolicyOrigin", + "type": "string", + "description": "Retention policy origin entity (Table/Database/Cluster).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "RetentionPolicy", + "type": "dynamic", + "description": "The table's effective entity retention policy, serialized as JSON.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CachingPolicyOrigin", + "type": "string", + "description": "Caching policy origin entity (Table/Database/Cluster).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CachingPolicy", + "type": "dynamic", + "description": "The table's effective entity caching policy, serialized as JSON.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "MaxExtentsCreationTime", + "type": "datetime", + "description": "The maximum creation time of an extent in the table (or null, if there are no extents).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "MinExtentsCreationTime", + "type": "datetime", + "description": "The minimum creation time of an extent in the table (or null, if there are no extents).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalExtentCount", + "type": "long", + "description": "The total number of extents in the table.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TotalRowCount", + "type": "long", + "description": "The total number of rows in the table.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "HotExtentCount", + "type": "long", + "description": "The total number of extents in the table, stored in the hot cache.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "HotOriginalSize", + "type": "long", + "description": "The total original size of data in the table, stored in the hot cache (in bytes).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "HotRowCount", + "type": "long", + "description": "The total number of rows in the table, stored in the hot cache.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/ADXTableDetails", + "name": "ADXTableDetails" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AzureActivity", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "description": "ID of the worksapce that stores this record", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "description": "Azure is used always for AzureActivity", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CallerIpAddress", + "type": "string", + "description": "IP address of the user who has performed the operation UPN claim or SPN claim based on availability.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CategoryValue", + "type": "string", + "description": "Category of the activity log e.g. Administrative, Policy, Security.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CorrelationId", + "type": "string", + "description": "Usually a GUID in the string format. Events that share a correlationId belong to the same uber action.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Authorization", + "type": "string", + "description": "Blob of RBAC properties of the event. Usually includes the \u201caction\u201d, \u201crole\u201d and \u201cscope\u201d properties. Stored as string. The use of Authorization_d should be preferred going forward.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Authorization_d", + "type": "dynamic", + "description": "Blob of RBAC properties of the event. Usually includes the \u201caction\u201d, \u201crole\u201d and \u201cscope\u201d properties. Stored as dynamic column.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Claims", + "type": "string", + "description": "The JWT token used by Active Directory to authenticate the user or application to perform this operation in Resource Manager. The use of claims_d should be preferred going forward.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Claims_d", + "type": "dynamic", + "description": "The JWT token used by Active Directory to authenticate the user or application to perform this operation in Resource Manager.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Level", + "type": "string", + "description": "Level of the event. One of the following values: Critical, Error, Warning, Informational and Verbose.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationNameValue", + "type": "string", + "description": "Identifier of the operation e.g. Microsoft.Storage/storageAccounts/listAccountSas/action.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Properties", + "type": "string", + "description": "Set of pairs (i.e. Dictionary) describing the details of the event. Stored as string. Usage of Properties_d is recommended instead.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Properties_d", + "type": "dynamic", + "description": "Set of pairs (i.e. Dictionary) describing the details of the event. Stored as dynamic column.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Caller", + "type": "string", + "description": "GUID of the caller.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EventDataId", + "type": "string", + "description": "Unique identifier of an event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "EventSubmissionTimestamp", + "type": "datetime", + "description": "Timestamp when the event became available for querying.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "HTTPRequest", + "type": "string", + "description": "Blob describing the Http Request. Usually includes the \u201cclientRequestId\u201d, \u201cclientIpAddress\u201d and \u201cmethod\u201d (HTTP method. For example, PUT).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationId", + "type": "string", + "description": "GUID of the operation", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResourceGroup", + "type": "string", + "description": "Resource group name of the impacted resource.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResourceProviderValue", + "type": "string", + "description": "Id of the resource provider for the impacted resource - e.g. Microsoft.Storage.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ActivityStatusValue", + "type": "string", + "description": "Status of the operation in display-friendly format. Common values include Started, In Progress, Succeeded, Failed, Active, Resolved.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ActivitySubstatusValue", + "type": "string", + "description": "Substatus of the operation in display-friendly format. E.g. OK (HTTP Status Code: 200).", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Hierarchy", + "type": "string", + "description": "Management group hierarchy of the management group or subscription that event belongs to.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "Timestamp when the event was generated by the Azure service processing the request corresponding the event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SubscriptionId", + "type": "guid", + "description": "Subscription ID of the impacted resource.", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AzureActivity", + "name": "AzureActivity" + }, + { + "properties": { + "totalRetentionInDays": 90, + "archiveRetentionInDays": 0, + "plan": "Analytics", + "retentionInDaysAsDefault": true, + "totalRetentionInDaysAsDefault": true, + "schema": { + "tableSubType": "Any", + "name": "AppServiceAppLogs", + "tableType": "Microsoft", + "standardColumns": [ + { + "name": "TenantId", + "type": "guid", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "TimeGenerated", + "type": "datetime", + "description": "Time when event is generated", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Level", + "type": "string", + "description": "Verbosity level of log mapped to standard levels (Informational, Warning, Error, or Critical)", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Host", + "type": "string", + "description": "Host where the application is running", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ResultDescription", + "type": "string", + "description": "Log message description", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "CustomLevel", + "type": "string", + "description": "Verbosity level of log", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Source", + "type": "string", + "description": "Application source from where log message is emitted", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Method", + "type": "string", + "description": "Application Method from where log message is emitted", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Logger", + "type": "string", + "description": "Application logger used to emit log message", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "WebSiteInstanceId", + "type": "string", + "description": "Instance Id the application running", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ExceptionClass", + "type": "string", + "description": "Application class from where log message is emitted ", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "Message", + "type": "string", + "description": "Log message", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "StackTrace", + "type": "string", + "description": "Complete stack trace of the log message in case of exception", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "ContainerId", + "type": "string", + "description": "Application container id", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "OperationName", + "type": "string", + "description": "The name of the operation represented by this event.", + "isDefaultDisplay": true, + "isHidden": false + }, + { + "name": "SourceSystem", + "type": "string", + "isDefaultDisplay": true, + "isHidden": false + } + ], + "solutions": [ + "LogManagement" + ], + "isTroubleshootingAllowed": false + }, + "provisioningState": "Succeeded", + "retentionInDays": 90 + }, + "id": "/subscriptions/40dcc8bf-0478-4f3b-b275-ed0a94f2c013/resourceGroups/asihuntomsworkspacerg/providers/Microsoft.OperationalInsights/workspaces/ASIHuntOMSWorkspaceV4/tables/AppServiceAppLogs", + "name": "AppServiceAppLogs" + } + ] +} \ No newline at end of file diff --git a/tests/testdata/azmondata/query_response.pkl b/tests/testdata/azmondata/query_response.pkl new file mode 100644 index 0000000000000000000000000000000000000000..99f02cd1fe5f208bf6741a5615f08f3ed5c28701 GIT binary patch literal 15812 zcmeI3d3+nkb;l`*5yktGWcj8o$(HRb1rQ(zws8^|Bt@7ccz~4K#ICwn42T^8yYTLU zBHgAh^fc1@Zqxg|)62aNx9;84X`1v%6W@04Hof1Q-v9)u$YEkp{#d5|(0As| z?97{)H*e2P?wc7(Q2WbE4?is=c%Bs=vsx zCE-_!LG9t%9dE0>rFN=%Ffg-4;nyCh6Jn-y9evm|QO?gh^-nT_YGZI)ZC zlpKEWRQ1rZa7-si<*44WT@q)6BRn$@l<&&870(ix4L=a2TD2`p)Qmx3Db8SngreMgj+Ig%J=B}Hc+1V->sZ|FP#iDybJdwEt98APx7@}r31LtZ6FyamtY zwP6$NS9R(VDB8y|+gaW6U}B+?r70ncK;s=r>SxvUHn^2*n)+r7%bLE{p)FZ7(Nnuh zp@;5d(Wbeh<)Ibk=|w)Z(2#UMg6f@Q@Fmm9&Dsv{E;fG}679THjvj8GNKT|PsM$Xu z)@YI@gl~CvnHPpk95U9klNPXCG&gj!oO69;GBn*Vzr$T%R!BZJ5^ zTL(4U4r#`wblYTN0ZY<7Srlu$v6tQRf*NX{^gP!~l9!ZWY^W+1j9?VkL zLsOMvafA9Q@>a=ZDp@J4V9vA()GOX^E8zPhoqi+^~a# zsq%vBkqc=j%bOL>Dv!QWu4b@Hi`4V9`g1h%CKt_`S@!KDSYM!h zcER;>iB+_P)nJuoC^9yUy#ya&t z5Nin9NPv~=)G6(#(jp)=&tnTg!M3!xKyHKd?xYlrh0Ts8wu^F2zN%+4s|QAcB|#@G z9dq53&J@T5r;6q(FAC-Pctx8lrghdVRw#FfLsax@}GL0-IwMSoZlviLE zaF_(_?P!~>_;ltaadY9T&>H!wRe_s8n^uuFbkoXqZc)6WSq(1AYiCD$gK~jO*Qt!4 zTbskjVK0f+H=50i(+|YX_k)bOPLji=|EXiQcACIOiLhBM$@MSe4wC_ zPAx&MPv_`(@&avoTGYs?n;5G1DBEU#T!a?wN>EqN&S_($6N?YL3K?3DK8~7Hs$}^V z!?z2#+it2-m+<7Y=||}PqEpR>^kFR{Q{3XSv{*S&Ze>6Vww#0Y2DLDwu1&h)=4n{h znI+1vwo{e*fZEhGn0x4kAv8wThkUcL9T4HdI*1J2bIyxgL(g6<<%2}eN0Ws6Ifndv zCCJjY)1Di1espWm<`A?lyq5Jdz8!VdfV5s*pOy#(n-A-OHHP22EJ}2-3TWHYSPu?png8&|PLFAd3nx?AC*MW;C|a&Vch zRDQ`0Xp?ADB{LgfW%Ww1mY&N8Ckx4UQs_iGkZScSfoD6GU9KzF$FlqA%<^%QW}7{0 zdjVai8Y|m=macQIr!J(OO?7F5HRUkaKE!MFC{7uQ3drU?n=Txj-#>STt%uI9q)T^r z_5$P*=-4OE$|09>yJ)RdZ^LusFjV+R&x>le4mr;S6fz?4JjnjUS*0V8?hYbQXJbS5 zDJ2fryuuehaw+-6HD6u{^!%jSe$omr?k0cjpZYwbjxV!dNcWOE!olbL~G}fTlOFP&~0@8v3-|s5yQblwtCgtdTu4K zxYwoHJ8Mhy0%E@GL~=Tnp1fa$GZd0iG|Lf5E8nj8-S^_G>*!l| zt$KXUwmjE&^MQd^G`>*}XAJc&7B`u?tTxvYuI>-VYSot7T>n7zcr-RLI%EtboFNl0 z(TDQ%UQOhN0(YqBt`51*a7{k9uchU3|CdfzThq(4b4{(MFLxiGW_!@1dhq1Qkw`ew zq~NF7-sWS~u5f&GI36C3#fL{jwd%f5ZKkEV&)9Jv^japEEoJ=jP;KTwwKGCXr{5Fp zxQR_|Ug4>KCabI-2(5=|l9N|?$Y&I3e`o6e}VBgU2AD@~*(B~z7d$Q?n8cOVBUK;~?@yZnArP9V|WGoer$HSA! zv0C*+*2;&YmKZZaRz7M(V`HWfAJ4~){K!~tTtrQB#8Ijii-}0c42>I+acj(ohQorY zjgAU$(RFl=BHGu`n|iU%ex;p{ zv~xMz3izvRp;vy8Mp@{SA7%8*k1+=16O2LmB;&Arig83f%{VHbVO&FJM&X!zmebd& z1uk4iXGh_B`80j7Zz za4#?e%mQ;jI$fAYYC--34{s5 zE&%Jm2JjH@dBEobZvnml_(I^Vz}tW?0^SaMG4KxHoxqm>UkbbncsKB6Ko$6M;46Ty z1ilLRYT#>thXHuz!h4YWTHxz|uLr&X_(tHHfNuuA1^8Cr+ps|2F8_^Z_dAgLPLzEY z@ZG?Bf$ssn7x+Hl`+*+-eh~N};D>=90e%$tG2q96p8$Rm_$lD0fu8|>7Wlbz;eAN` zJn##^F9N>={IdLa9?c{2KN!CP;jaR}2K+ik`Wwjoru;b9{4L~u8~7bm`Ca*+T=uB^ zFUIeo?Dyq=bM7(uKa4+s@DG7M0{$5I6W~vQKLh?8ct7wL@_)IN$L0SqJ^ zd3lKOf@~*L=&SboNm2Z6`R{FZ7WQm8JNNfId3V#$9j6r#naadl%I3kZRj>>BpI*PfD)7L6v zsYfx_v*-qhZUk-uZU%;cTYy`EzLFkX;R!sUx%4z#_u<I=rJFN%2)H}T5&Dl;Clpp=ZpMIoJ!4S5fpJ*A zk#R&O7)Rwj3?0QJIlVQCnPAb&9K}qs4p(m!GsVNZf>F#F?(cF&G1J^2qL>sX_oA3x ziem2N*}ZC_m>C|;BXX9pqbO#MYhEIXNpsnwa-PwMViq{}m|SGM^Jw( literal 0 HcmV?d00001 diff --git a/tests/testdata/kusto/T1_MDE_Cluster.yaml b/tests/testdata/kusto/T1_MDE_Cluster.yaml new file mode 100644 index 000000000..124bd2cbc --- /dev/null +++ b/tests/testdata/kusto/T1_MDE_Cluster.yaml @@ -0,0 +1,132 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T1_MDE_Cluster] + database: "AppDB" + cluster: https://test.kusto.windows.net + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T1_MDE_Cluster_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T1_MDE_Cluster_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T1_MDE_Cluster_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/T2_MDE_Clusters.yaml b/tests/testdata/kusto/T2_MDE_Clusters.yaml new file mode 100644 index 000000000..fae04c7ce --- /dev/null +++ b/tests/testdata/kusto/T2_MDE_Clusters.yaml @@ -0,0 +1,134 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T2_MDE_Clusters] + database: "AppDB" + clusters: + - https://test.kusto.windows.net + - msticti + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T2_MDE_Clusters_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T2_MDE_Clusters_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T2_MDE_Clusters_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/T3_MDE_Cluster_Group1.yaml b/tests/testdata/kusto/T3_MDE_Cluster_Group1.yaml new file mode 100644 index 000000000..5bb149e1f --- /dev/null +++ b/tests/testdata/kusto/T3_MDE_Cluster_Group1.yaml @@ -0,0 +1,133 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T3_MDE_Cluster_Group1] + database: "AppDB" + cluster_groups: + - group1 + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T3_MDE_Cluster_Group1_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T3_MDE_Cluster_Group1_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T3_MDE_Cluster_Group1_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/T4_NoCluster_Info.yaml b/tests/testdata/kusto/T4_NoCluster_Info.yaml new file mode 100644 index 000000000..1595f8679 --- /dev/null +++ b/tests/testdata/kusto/T4_NoCluster_Info.yaml @@ -0,0 +1,131 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T4_NoCluster_Info] + database: "AppDB" + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T4_NoCluster_Info_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T4_NoCluster_Info_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T4_NoCluster_Info_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/T5_Help_Cluster_Group1.yaml b/tests/testdata/kusto/T5_Help_Cluster_Group1.yaml new file mode 100644 index 000000000..de126932b --- /dev/null +++ b/tests/testdata/kusto/T5_Help_Cluster_Group1.yaml @@ -0,0 +1,134 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T5_Help_Cluster_Group1] + database: "AppDB" + cluster: https://help.kusto.windows.net + cluster_groups: + - group3 + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T5_Help_Cluster_Group1_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T5_Help_Cluster_Group1_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T5_Help_Cluster_Group1_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/T6_Group2_Cluster.yaml b/tests/testdata/kusto/T6_Group2_Cluster.yaml new file mode 100644 index 000000000..300a6ce1f --- /dev/null +++ b/tests/testdata/kusto/T6_Group2_Cluster.yaml @@ -0,0 +1,132 @@ +metadata: + version: 1 + description: Kusto Queries + data_environments: [Kusto] + data_families: [T6_Group2_Cluster] + database: "AppDB" + cluster_groups: [group2] + tags: ["user"] +defaults: + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + default: -30 + end: + description: Query end time + type: datetime + default: 0 + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_host_processes: + description: Lists all process creations for a host + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where DeviceName has "{host_name}" + {add_query_items}' + uri: None + parameters: + host_name: + description: Name of host + type: str + process_creations: + description: Lists all processes created by name or hash + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{process_identifier}" or SHA1 has "{process_identifier}" or SHA256 has "{process_identifier}" or MD5 has "{process_identifier}" + {add_query_items}' + parameters: + process_identifier: + description: Identifier for the process, filename, or hash + type: str + process_paths: + description: Lists all processes created from a path + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName contains "{file_path}" + {add_query_items}' + parameters: + file_path: + description: full or partial path + type: str + process_cmd_line: + description: Lists all processes with a command line containing a string + metadata: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + query_new_alias: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T6_Group2_Cluster_item1] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_fam_no_dot: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T6_Group2_Cluster_item2] + cluster: https://msticapp.kusto.windows.net + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str + bad_query_no_cluster: + description: Lists all processes with a command line containing a string + metadata: + data_families: [T6_Group2_Cluster_item3] + cluster: + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where ProcessCommandLine contains "{cmd_line}" + {add_query_items}' + parameters: + cmd_line: + description: Command line artifact to search for + type: str diff --git a/tests/testdata/kusto/kusto_raw_resp.json b/tests/testdata/kusto/kusto_raw_resp.json new file mode 100644 index 000000000..88fe6b49f --- /dev/null +++ b/tests/testdata/kusto/kusto_raw_resp.json @@ -0,0 +1,2003 @@ +[ + { + "FrameType": "DataSetHeader", + "IsProgressive": false, + "Version": "v2.0" + }, + { + "FrameType": "DataTable", + "TableId": 0, + "TableKind": "QueryProperties", + "TableName": "@ExtendedProperties", + "Columns": [ + { + "ColumnName": "TableId", + "ColumnType": "int" + }, + { + "ColumnName": "Key", + "ColumnType": "string" + }, + { + "ColumnName": "Value", + "ColumnType": "dynamic" + } + ], + "Rows": [ + [ + 1, + "Visualization", + "{\"Visualization\":null,\"Title\":null,\"XColumn\":null,\"Series\":null,\"YColumns\":null,\"AnomalyColumns\":null,\"XTitle\":null,\"YTitle\":null,\"XAxis\":null,\"YAxis\":null,\"Legend\":null,\"YSplit\":null,\"Accumulate\":false,\"IsQuerySorted\":false,\"Kind\":null,\"Ymin\":\"NaN\",\"Ymax\":\"NaN\",\"Xmin\":null,\"Xmax\":null}" + ] + ] + }, + { + "FrameType": "DataTable", + "TableId": 1, + "TableKind": "PrimaryResult", + "TableName": "PrimaryResult", + "Columns": [ + { + "ColumnName": "CreatedProcessIsElevated", + "ColumnType": "bool" + }, + { + "ColumnName": "CreatedProcessIntegrityLevel", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessAccountSid", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessAccountName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessAccountDomainName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessTokenElevationType", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileMarkOfTheWeb", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessCreationTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "CreatedProcessId", + "ColumnType": "int" + }, + { + "ColumnName": "CreatedProcessName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessCommandLine", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileType", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileCreationTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "CreatedProcessFilePath", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileSize", + "ColumnType": "int" + }, + { + "ColumnName": "CreatedProcessFileMd5", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileSha256", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileSha1", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageMd5", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageSha256", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageSha1", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessAccountSid", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessAccountDomainName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessAccountName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessCreationTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessId", + "ColumnType": "int" + }, + { + "ColumnName": "InitiatingProcessName", + "ColumnType": "string" + }, + { + "ColumnName": "Process_CommandLine", + "ColumnType": "string" + }, + { + "ColumnName": "IsElevatedProcess", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessParentCreationTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessParentProcessId", + "ColumnType": "int" + }, + { + "ColumnName": "InitiatingProcessParentProcessName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessIntegrityLevel", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessTokenElevationType", + "ColumnType": "string" + }, + { + "ColumnName": "OrgId", + "ColumnType": "string" + }, + { + "ColumnName": "MachineId", + "ColumnType": "string" + }, + { + "ColumnName": "WcdMachineId", + "ColumnType": "string" + }, + { + "ColumnName": "SenseMachineGuid", + "ColumnType": "string" + }, + { + "ColumnName": "ReportTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "ReportArrivalTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "ReportGuid", + "ColumnType": "string" + }, + { + "ColumnName": "ComputerDnsName", + "ColumnType": "string" + }, + { + "ColumnName": "ReportIndex", + "ColumnType": "long" + }, + { + "ColumnName": "IsLastInQuota", + "ColumnType": "bool" + }, + { + "ColumnName": "ClientVersion", + "ColumnType": "string" + }, + { + "ColumnName": "IsTestOrg", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessStartKey", + "ColumnType": "string" + }, + { + "ColumnName": "ContainerId", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessStartKey", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessReparentingProcessCreationTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "CreatedProcessReparentingProcessId", + "ColumnType": "long" + }, + { + "ColumnName": "CreatedProcessParentCreationTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "CreatedProcessParentName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessParentId", + "ColumnType": "long" + }, + { + "ColumnName": "CreatedProcessAttributes", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessSource", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageFilePath", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageFileSizeInBytes", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessImagePeTimestampUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessImageLastWriteTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessImageLastAccessTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessImageCreationTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessAttributes", + "ColumnType": "long" + }, + { + "ColumnName": "TruncationPolicy", + "ColumnType": "string" + }, + { + "ColumnName": "RbacGroupId", + "ColumnType": "int" + }, + { + "ColumnName": "OsVersionK", + "ColumnType": "string" + }, + { + "ColumnName": "OsVersion", + "ColumnType": "string" + }, + { + "ColumnName": "IsMalformed", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessAccountUpn", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessAccountAzureADId", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessAccountUpn", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessAccountAzureADId", + "ColumnType": "string" + }, + { + "ColumnName": "IsMtpEnabled", + "ColumnType": "bool" + }, + { + "ColumnName": "TenantId", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessLogonId", + "ColumnType": "long" + }, + { + "ColumnName": "LogonId", + "ColumnType": "long" + }, + { + "ColumnName": "FirstSeen", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessShowWindow", + "ColumnType": "int" + }, + { + "ColumnName": "InitiatingProcessStartupFlags", + "ColumnType": "int" + }, + { + "ColumnName": "CreatedProcessShowWindow", + "ColumnType": "int" + }, + { + "ColumnName": "CreatedProcessStartupFlags", + "ColumnType": "int" + }, + { + "ColumnName": "InitiatingProcessCurrentWorkingDirectory", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessPosixProcessGroupId", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessPosixSessionId", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessPosixEffectiveUser", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessPosixEffectiveGroup", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessPosixAttachedTerminal", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessSignatureStatus", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessSignerType", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessSignatureStatus", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessSignerType", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoCompanyName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoProductName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoProductVersion", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoInternalFileName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoOriginalFileName", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessVersionInfoFileDescription", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoCompanyName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoProductName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoProductVersion", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoInternalFileName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoOriginalFileName", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessVersionInfoFileDescription", + "ColumnType": "string" + }, + { + "ColumnName": "IsMalformedBool", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessSessionId", + "ColumnType": "long" + }, + { + "ColumnName": "CreatedProcessSessionId", + "ColumnType": "long" + }, + { + "ColumnName": "SenseEpoch", + "ColumnType": "long" + }, + { + "ColumnName": "MsaDeviceId", + "ColumnType": "string" + }, + { + "ColumnName": "AsimovEpoch", + "ColumnType": "string" + }, + { + "ColumnName": "SenseSequenceNum", + "ColumnType": "long" + }, + { + "ColumnName": "ReportEventSource", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcess_PathNoEscaping", + "ColumnType": "string" + }, + { + "ColumnName": "IsParentProcessAbsent", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessReparentingProcessCreationTimeUtc", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessReparentingProcessId", + "ColumnType": "long" + }, + { + "ColumnName": "IsInitiatingProcessNameManipulated", + "ColumnType": "bool" + }, + { + "ColumnName": "InitiatingProcessImageMarkOfTheWeb", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageFileType", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageFileName", + "ColumnType": "string" + }, + { + "ColumnName": "InternalInfo", + "ColumnType": "string" + }, + { + "ColumnName": "CreatedProcessFileLsHash", + "ColumnType": "string" + }, + { + "ColumnName": "InitiatingProcessImageLsHash", + "ColumnType": "string" + }, + { + "ColumnName": "CloudBlobUri", + "ColumnType": "string" + }, + { + "ColumnName": "KnownPrevalence", + "ColumnType": "long" + }, + { + "ColumnName": "FirstSeenDateTime", + "ColumnType": "datetime" + }, + { + "ColumnName": "DistinctOrgCount", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessKnownPrevalence", + "ColumnType": "long" + }, + { + "ColumnName": "InitiatingProcessFirstSeen", + "ColumnType": "datetime" + }, + { + "ColumnName": "InitiatingProcessDistinctOrgCount", + "ColumnType": "long" + }, + { + "ColumnName": "AdditionalFields", + "ColumnType": "string" + } + ], + "Rows": [ + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:50.7862004Z", + 9784, + "omadmprc.exe", + "\"omadmprc.exe\"", + "PortableExecutable", + "2023-02-22T12:16:45.1721477Z", + "C:\\Windows\\System32\\omadmprc.exe", + 90112, + "6dbe9df3f4f09817a1f928380fecbd3d", + "96a65a2fc808551abcbbf8f147cac3239b10f206a328d1c2f970222ec2105ae9", + "220dc5b7ff4a8beef50a994b7881de5cf83e4459", + "b7f884c1b74a263f746ee12a5f7c9f6a", + "add683a6910abbbf0e28b557fad0ba998166394932ae2aca069d9aa19ea8fe88", + "1bc5066ddf693fc034d6514618854e26a84fd0d1", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:50.5948465Z", + 9504, + "svchost.exe", + "svchost.exe -k netsvcs -p -s dmwappushservice", + true, + "2023-04-11T16:19:42.0429875Z", + 1168, + "\\Device\\HarddiskVolume4\\Windows\\System32\\services.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1634079Z", + "2023-04-11T16:22:08.6091382Z", + "92e67d36-ae3a-40c4-b72c-e2d745b57b09", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 189, + false, + "10.8295.19041.2673", + false, + "77687093572141245", + "", + "77687093572141250", + "2023-04-11T16:19:50.5948465Z", + 9504, + "2023-04-11T16:19:50.5948465Z", + "svchost.exe", + 9504, + 256, + "ActiveProcessStartkey", + "C:\\Windows\\System32", + 55320, + "2036-11-16T10:28:07Z", + "2022-07-26T12:26:17.7862305Z", + "2023-04-11T16:19:50.681591Z", + "2022-07-26T12:26:17.7862305Z", + 128, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + 128, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1806", + "svchost.exe", + "svchost.exe", + "Host Process for Windows Services", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2546", + "omadmprc", + "omadmprc.exe", + "Host Process for Push Router Client of OMA-DM", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 186, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0429875Z", + 1168, + false, + "NotWeb", + "PortableExecutable", + "svchost.exe", + "", + "6feef67ea7ad9a6db7adfa6be759f55ade676bfa6eb575d96e67b7bebe96665ea77e5be6b9fb9e59a57ed755956edfe979df5f657def7fe7b95b5bd9b6ebbbab", + "bffde9af97ad995a77a9e56fe79af969de676ada7ff576e56f977abfad96755ab67e5775e97b9f79d76d6659d96eeab979ef9fe5beffbffae55b57d5ba9baba7", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 141135, + "2023-01-19T22:22:09.4487539Z", + 1, + 168660, + "2022-11-09T21:15:55.6445748Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:47.5790497Z", + 7744, + "conhost.exe", + "conhost.exe 0xffffffff -ForceV1", + "PortableExecutable", + "2023-02-22T12:16:19.1836847Z", + "C:\\Windows\\System32\\conhost.exe", + 867840, + "17a7131db3e37157c6d3c09a0bae95d3", + "a26a1ffb81a61281ffa55cb7778cc3fb0ff981704de49f75f51f18b283fba7a2", + "edd3f649f5f0556d2edc8b19265adb6b93465721", + "76cd6626dd8834bd4a42e6a565104dc2", + "013c013e0efd13c9380fad58418b7aca8356e591a5cceffdb910f7d8b0ad28ef", + "ccaa6ee9ef6d0f9c8157adc996ffd477f73b5d02", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:47.3460505Z", + 7576, + "schtasks.exe", + "schtasks /TN RtkAudUService64_BG /XML", + true, + "2023-04-11T16:19:46.1417252Z", + 6376, + "\\Device\\HarddiskVolume4\\Windows\\System32\\DriverStore\\FileRepository\\realtekservice.inf_amd64_bbb0597391852f64\\RtkAudUService64.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1511468Z", + "2023-04-11T16:22:08.6063341Z", + "14366102-d4be-426c-858c-1ef709d6b235", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 153, + false, + "10.8295.19041.2673", + false, + "77687093572141194", + "", + "77687093572141196", + "2023-04-11T16:19:47.3460505Z", + 7576, + "2023-04-11T16:19:47.3460505Z", + "\\Device\\HarddiskVolume4\\Windows\\System32\\schtasks.exe", + 7576, + 18560, + "TerminateProcessStartkey", + "C:\\Windows\\System32", + 235008, + "2003-04-01T16:49:17Z", + "2022-02-21T18:47:55.1893128Z", + "2023-04-11T16:19:47.6232114Z", + "2022-02-21T18:47:55.1847466Z", + 18560, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + null, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1503", + "schtasks.exe", + "schtasks.exe", + "Task Scheduler Configuration Tool", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2546", + "ConHost", + "CONHOST.EXE", + "Console Window Host", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 150, + "Asimov", + "", + false, + "2023-04-11T16:19:46.1417252Z", + 6376, + false, + "NotWeb", + "PortableExecutable", + "schtasks.exe", + "", + "bfd9e97eb79d966e77e5a66bdb99f966debb6fea6af675dd5db7799dbf76675eeb7b97adf5f7ea9ab56967d565af9da57ddf5fa67eed7ba6fa5b57d9bbebafba", + "6fd5ed6e6fdda9597feaa557e7eef956dea756ee6efd76997fa7b95daea6b55eb66f5adaf9bfee5af5bdde55a59e7a95f9dfdfa7aebe9bf6b95e56f9aadf77f6", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 152498, + "2023-01-19T22:19:53.2201757Z", + 1, + 180716, + "2022-10-06T08:15:11.2376201Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:50.8255981Z", + 9816, + "omadmclient.exe", + "\"omadmclient.exe\" /serverid \"911DE3AA-C65D-4895-B2CF-64FAA73E9287\" /lookuptype 1 /initiator 0", + "PortableExecutable", + "2023-03-22T14:21:59.502037Z", + "C:\\Windows\\System32\\omadmclient.exe", + 462848, + "19b55aa05dae59035cf8459b6a7efcc2", + "3c85b0e7f3ab8b83b5abbc23c54960b3eff811674ac36c6ffa43d2433428e4e4", + "6bef37ac3f4e8c5ee111354a8cb5bc00c467b1ca", + "b7f884c1b74a263f746ee12a5f7c9f6a", + "add683a6910abbbf0e28b557fad0ba998166394932ae2aca069d9aa19ea8fe88", + "1bc5066ddf693fc034d6514618854e26a84fd0d1", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:44.0895075Z", + 3140, + "svchost.exe", + "svchost.exe -k netsvcs -p -s Schedule", + true, + "2023-04-11T16:19:42.0429875Z", + 1168, + "services.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1641152Z", + "2023-04-11T16:22:08.6092094Z", + "86710269-3d9e-4436-8db7-48d9bfd77434", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 190, + false, + "10.8295.19041.2673", + false, + "77687093572141105", + "", + "77687093572141251", + "2023-04-11T16:19:44.0895075Z", + 3140, + "2023-04-11T16:19:44.0895075Z", + "svchost.exe", + 3140, + 256, + "ActiveProcessStartkey", + "C:\\Windows\\System32", + 55320, + "2036-11-16T10:28:07Z", + "2022-07-26T12:26:17.7862305Z", + "2023-04-11T16:19:53.97315Z", + "2022-07-26T12:26:17.7862305Z", + 256, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + null, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1806", + "svchost.exe", + "svchost.exe", + "Host Process for Windows Services", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2673", + "omadmclient", + "omadmclient.exe", + "Host Process for OMA-DM Client", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 187, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0429875Z", + 1168, + false, + "NotWeb", + "PortableExecutable", + "svchost.exe", + "", + "9fd9ed6fbfdd59597bdda567d7def955dfa777e96ef576edbf6b77edae65a65df76b5ae6bdfbee99b6bd9f95a5aeaa9569ee9f66bded6ba6f55b57e9e6bfbbea", + "bffde9af97ad995a77a9e56fe79af969de676ada7ff576e56f977abfad96755ab67e5775e97b9f79d76d6659d96eeab979ef9fe5beffbffae55b57d5ba9baba7", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 134490, + "2023-02-21T22:24:09.6321291Z", + 1, + 168660, + "2022-11-09T21:15:55.6445748Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:47.4760037Z", + 7688, + "acumbrellaagent.exe", + "\"acumbrellaagent.exe\"", + "PortableExecutable", + "2020-12-10T05:18:24Z", + "C:\\Program Files (x86)\\Cisco\\Cisco AnyConnect Secure Mobility Client\\acumbrellaagent.exe", + 530680, + "3126f1f5e32c42cc959cbe1da77c2c72", + "e163f9b6afd1fcbb512398a81cd5c6b801ad15e6a9b44225b8fb8bda9436cf78", + "11c9e80af359f7ca69cba13b60f75a20e991ea30", + "d8e577bf078c45954f4531885478d5a9", + "dfbea9e8c316d9bc118b454b0c722cd674c30d0a256340200e2c3a7480cba674", + "d7a213f3cfee2a8a191769eb33847953be51de54", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:42.0429875Z", + 1168, + "services.exe", + "services.exe", + true, + "2023-04-11T16:19:42.0085677Z", + 1096, + "\\Device\\HarddiskVolume4\\Windows\\System32\\wininit.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1509934Z", + "2023-04-11T16:22:08.6062575Z", + "426df9de-14ef-41e7-b1d4-433616431092", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 152, + false, + "10.8295.19041.2673", + false, + "77687093572141066", + "", + "77687093572141195", + "2023-04-11T16:19:42.0429875Z", + 1168, + "2023-04-11T16:19:42.0429875Z", + "services.exe", + 1168, + 256, + "ActiveProcessStartkey", + "C:\\Windows\\System32", + 714856, + "2060-05-15T06:03:53Z", + "2022-01-28T14:36:27.419896Z", + "2023-04-11T16:19:53.8122251Z", + "2022-01-28T14:36:27.4042761Z", + 128, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + 0, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Invalid", + "Unknown", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.928", + "services.exe", + "services.exe", + "Services and Controller app", + "Cisco Systems, Inc.", + "Cisco AnyConnect Secure Mobility Client", + "4.9.05042", + "acumbrellaagent", + "acumbrellaagent.exe", + "Cisco AnyConnect Roaming Security Agent", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 149, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0085677Z", + 1096, + false, + "NotWeb", + "PortableExecutable", + "services.exe", + "", + "bbda9e95d7abeaffa9bea97fe76fe55e9d5d6e676fe6b557bfd7f5695ed666fb6b5f7beddb7e7af5e57ee6a6967e6bbd65fbdeb9795bb6faa56b9ede795bda95", + "afdda9efabdd6da7bbd9a567e79af956dfebb6ed56f576ed5e6776adaf66565e766b5a95bdfbde5b759f6ba9799f5e95b5deafb7beeeaba7f66a57e5f5ffbbf6", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 41896, + "2020-12-18T19:13:11.2341727Z", + 1, + 376548, + "2021-04-09T11:56:53.3872126Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-19", + "LOCAL SERVICE", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:47.804151Z", + 7924, + "WUDFHost.exe", + "\"WUDFHost.exe\" -HostGUID:{193a1820-d9ac-4997-8c55-be817523f6aa} -IoEventPortName:\\UMDFCommunicationPorts\\WUDF\\HostProcess-4459c457-82d0-4485-917d-cef923b5f42e -SystemEventPortName:\\UMDFCommunicationPorts\\WUDF\\HostProcess-d8c85425-b271-4c03-b16a-2039ecf7003a -IoCancelEventPortName:\\UMDFCommunicationPorts\\WUDF\\HostProcess-f9dba288-d877-4b4e-bef4-2aa338b9316d -NonStateChangingEventPortName:\\UMDFCommunicationPorts\\WUDF\\HostProcess-dae81fc1-29c8-4dfe-ada4-636e5cc4594e -LifetimeId:f4747707-12fd-43e4-8f36-5f35eb07ed9c -DeviceGroupId:WudfDefaultDevicePool -HostArg:0", + "PortableExecutable", + "2022-08-21T02:16:03.2675851Z", + "C:\\Windows\\System32\\WUDFHost.exe", + 271872, + "00e2ef3d2c9309ca4135195a049cc79c", + "28cb4ebd59945b16e0e9e04154c7a1255b404a6e819c6c5fb4dbe4fa050c9b3b", + "7733b12b689baadff24afc5b0f0d3ab6488f0f70", + "d8e577bf078c45954f4531885478d5a9", + "dfbea9e8c316d9bc118b454b0c722cd674c30d0a256340200e2c3a7480cba674", + "d7a213f3cfee2a8a191769eb33847953be51de54", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:42.0429875Z", + 1168, + "services.exe", + "services.exe", + true, + "2023-04-11T16:19:42.0085677Z", + 1096, + "\\Device\\HarddiskVolume4\\Windows\\System32\\wininit.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1515157Z", + "2023-04-11T16:22:08.6064807Z", + "b9f098a7-1bcd-46f1-a5f7-2e75a79c1762", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 155, + false, + "10.8295.19041.2673", + false, + "77687093572141066", + "", + "77687093572141198", + "2023-04-11T16:19:42.0429875Z", + 1168, + "2023-04-11T16:19:42.0429875Z", + "\\Device\\HarddiskVolume4\\Windows\\System32\\services.exe", + 1168, + 128, + "ActiveProcessStartkey", + "C:\\Windows\\System32", + 714856, + "2060-05-15T06:03:53Z", + "2022-01-28T14:36:27.419896Z", + "2023-04-11T16:19:53.8122251Z", + "2022-01-28T14:36:27.4042761Z", + 128, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 997, + false, + null, + 0, + null, + 0, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.928", + "services.exe", + "services.exe", + "Services and Controller app", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1865", + "WUDFHost.exe", + "WUDFHost.exe", + "Windows Driver Foundation - User-mode Driver Framework Host Process", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 152, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0085677Z", + 1096, + false, + "NotWeb", + "PortableExecutable", + "services.exe", + "", + "9fdded6ebfdd596a7bed9b57d7ddf566eeab66d96eba75e95d67765d7ea7569dffaf9aa5bdfadd9ea5f59a99e96fad99b9dfaf67beae77afb65f67f6f6de67ea", + "afdda9efabdd6da7bbd9a567e79af956dfebb6ed56f576ed5e6776adaf66565e766b5a95bdfbde5b759f6ba9799f5e95b5deafb7beeeaba7f66a57e5f5ffbbf6", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 168433, + "2022-07-26T21:16:56.2881063Z", + 1, + 376548, + "2021-04-09T11:56:53.3872126Z", + 1, + "" + ], + [ + false, + "", + "S-1-5-18", + "UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785", + "DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13", + "None", + "NotWeb", + "2023-04-11T16:20:25.009678Z", + 54965, + "chmod", + "/bin/chmod 0664 \"/Library/Caches/com.microsoft.autoupdate.helper/Clones.noindex/Microsoft Word.app/Contents/Frameworks/mso40ui.framework/Versions/A/Resources/cs.lproj/tbbtn2.strings\"", + "MachOExecutable", + "2023-01-11T07:03:20Z", + "/bin/chmod", + 136960, + "20dace51adb335f72679219e1fd7e207", + "ca99dadc5d1771eadf732a915a08e7443d5fa688414b12554023966bd1727e2c", + "c7752022b9395338dd226ea0b02aa0b33b77a555", + "03c4befaffe8a85a772194289704557e", + "d82e2354b0c20f479770b490568c16a09144a274fb8511adb6a3a7fef9a013d1", + "426a4f1a6f1a3fdd7440ab9d3764a2df8fe363d4", + "S-1-5-18", + "DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13", + "UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785", + "2023-04-11T16:20:25.005691Z", + 54965, + "zsh", + "", + false, + "2023-04-11T16:18:50.962145Z", + 36594, + "zsh", + "", + "None", + "2686d60f-130f-4960-a5fb-237b76c3ce80", + "6bac8eee68741c6551e0ec5f9bd1f91f56c8af2c", + "", + "", + "2023-04-11T16:20:25.009678Z", + "2023-04-11T16:22:19.29349Z", + "743d9d03-14e5-4a06-803d-3f5082bef27f", + "computerpii_c3c1e3b1dad5b7b1332fa45e3903b48157599b13.domainpii_13d91d49f57221f07fa53a662a35b5ddbedb6a14.com", + 102864, + null, + "20.123012.19830.0", + false, + "0", + "", + "0", + null, + 0, + "2023-04-11T16:20:25.005691Z", + "zsh", + 54965, + 4, + "Missing", + "/bin/", + 1377872, + null, + "2023-01-11T07:03:20Z", + "2023-01-11T07:03:20Z", + "2023-01-11T07:03:20Z", + 4, + "NoTruncation", + 0, + "", + "12.6", + "False", + "", + "", + "", + "", + true, + "StrPII_5e8f90b3f1b8c3ae87a982b737e3256735308067", + 0, + 0, + true, + null, + null, + null, + null, + "", + 36594, + 1, + "{\"Sid\":\"S-1-5-18\",\"Name\":\"UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785\",\"DomainName\":\"DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13\",\"LogonId\":0,\"AadUserId\":null,\"AadUserUpn\":null,\"PosixUserId\":0,\"PrimaryPosixGroup\":{\"Name\":\"StrPII_3a7195d99bee6bb15aacf02a10d56b4cbd65b1fe\",\"PosixGroupId\":0}}", + "{\"Name\":\"StrPII_3a7195d99bee6bb15aacf02a10d56b4cbd65b1fe\",\"PosixGroupId\":0}", + "", + "Unknown", + "Unknown", + "Unknown", + "Unknown", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + false, + null, + null, + 20452320, + "StrPII_bc5699d9bfd152ecd5ecc8979872de278ffd5e58", + "f4cfd1ef-31e6-42cb-aef9-c3bf9346319e", + 26443, + "Asimov", + "", + false, + null, + null, + false, + "NotWeb", + "MachOExecutable", + "zsh", + "", + "", + "", + "https://cloudscrubbed5eusprd.blob.core.windows.net/2023041116/46371_ScrubbedCyberReport_20230411162233063_20230411162233063.cloud-scrubber-7bf5df7d8f-gqjg8_EastUs2_81_hourly.Bond.gz", + 60565, + "2022-09-07T21:22:14.4812196Z", + 1, + 60601, + "2022-09-08T19:06:13.3759065Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:47.5952302Z", + 7764, + "conhost.exe", + "conhost.exe 0xffffffff -ForceV1", + "PortableExecutable", + "2023-02-22T12:16:19.1836847Z", + "C:\\Windows\\System32\\conhost.exe", + 867840, + "17a7131db3e37157c6d3c09a0bae95d3", + "a26a1ffb81a61281ffa55cb7778cc3fb0ff981704de49f75f51f18b283fba7a2", + "edd3f649f5f0556d2edc8b19265adb6b93465721", + "e30e7a42a010bf95524514bdf2035695", + "20db4abf4539d2e054fbadde48078452a5a4adbca9eaeff66aba89f2c9164055", + "3f38989e61670025c2585a9e3cc8f1e1c9f229e9", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:47.2561347Z", + 7516, + "wevtutil.exe", + "wevtutil.exe uninstall-manifest \"C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\Microsoft-Antimalware-AMFilter.man\"", + true, + "2023-04-11T16:19:46.1453156Z", + 6404, + "\\Device\\HarddiskVolume4\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\MsMpEng.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1513105Z", + "2023-04-11T16:22:08.606409Z", + "9a8339a3-b200-4b67-969f-5ffe95b341c3", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 154, + false, + "10.8295.19041.2673", + false, + "77687093572141193", + "", + "77687093572141197", + "2023-04-11T16:19:47.2561347Z", + 7516, + "2023-04-11T16:19:47.2561347Z", + "\\Device\\HarddiskVolume4\\Windows\\System32\\wevtutil.exe", + 7516, + 18560, + "TerminateProcessStartkey", + "C:\\Windows\\System32", + 248320, + "1970-02-11T04:42:15Z", + "2022-11-15T00:48:35.8125454Z", + "2023-04-11T16:19:50.5663201Z", + "2022-11-15T00:48:35.7968049Z", + 18560, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + null, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2193", + "wevtutil.exe", + "wevtutil.exe", + "Eventing Command Line Utility", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2546", + "ConHost", + "CONHOST.EXE", + "Console Window Host", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 151, + "Asimov", + "", + false, + "2023-04-11T16:19:46.1453156Z", + 6404, + false, + "NotWeb", + "PortableExecutable", + "wevtutil.exe", + "", + "bfd9e97eb79d966e77e5a66bdb99f966debb6fea6af675dd5db7799dbf76675eeb7b97adf5f7ea9ab56967d565af9da57ddf5fa67eed7ba6fa5b57d9bbebafba", + "5fd6ed6ebbdd596abbddd66be799fd55dfb77aee5ef575d95a677a69ef7776aeabafb565f9bbfeaab5b99fe5656f699579de9fe7bde96be6f95b5bf5bbeaa7fa", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 152498, + "2023-01-19T22:19:53.2201757Z", + 1, + 164276, + "2022-10-24T21:35:41.1880025Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:48.0137075Z", + 8120, + "dasHost.exe", + "dashost.exe {c89dd910-3929-4aaf-8f47b9bd59be2ebf}", + "PortableExecutable", + "2022-07-26T12:26:14.6353688Z", + "C:\\Windows\\System32\\dasHost.exe", + 98816, + "2857a196985fc58a74c337b5e95b2174", + "bdbe0cab8e69c015d4c75036336da56c8a2d7d078b4626a1874dfe2cd8c0ec0a", + "ab5ae2d7ff61553e3472932370fcbd8457d475f0", + "b7f884c1b74a263f746ee12a5f7c9f6a", + "add683a6910abbbf0e28b557fad0ba998166394932ae2aca069d9aa19ea8fe88", + "1bc5066ddf693fc034d6514618854e26a84fd0d1", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:44.1653549Z", + 3560, + "svchost.exe", + "svchost.exe -k LocalSystemNetworkRestricted -p -s DeviceAssociationService", + true, + "2023-04-11T16:19:42.0429875Z", + 1168, + "\\Device\\HarddiskVolume4\\Windows\\System32\\services.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1524619Z", + "2023-04-11T16:22:08.6067032Z", + "0c3afb26-c4eb-4458-ac6e-0625c38c2bf3", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 158, + false, + "10.8295.19041.2673", + false, + "77687093572141118", + "", + "77687093572141202", + "2023-04-11T16:19:44.1653549Z", + 3560, + "2023-04-11T16:19:44.1653549Z", + "\\Device\\HarddiskVolume4\\Windows\\System32\\svchost.exe", + 3560, + 128, + "ActiveProcessStartkey", + "C:\\Windows\\System32", + 55320, + "2036-11-16T10:28:07Z", + "2022-07-26T12:26:17.7862305Z", + "2023-04-11T16:19:50.681591Z", + "2022-07-26T12:26:17.7862305Z", + 128, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + 128, + null, + 0, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1806", + "svchost.exe", + "svchost.exe", + "Host Process for Windows Services", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.1806", + "dasHost.exe", + "dasHost.exe", + "Device Association Framework Provider Host", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 155, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0429875Z", + 1168, + false, + "NotWeb", + "PortableExecutable", + "svchost.exe", + "", + "6fede97ebb6d9d6bbbf5a66bfbdef556dfa77a9b6db676ad7d677dba6e55559b777e6fd5bef7dfa9b6ba6f59a5af6e9975de9f966dabffa7b5a79ed575ab7bba", + "bffde9af97ad995a77a9e56fe79af969de676ada7ff576e56f977abfad96755ab67e5775e97b9f79d76d6659d96eeab979ef9fe5beffbffae55b57d5ba9baba7", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 159309, + "2022-11-22T15:55:38.1388669Z", + 1, + 168660, + "2022-11-09T21:15:55.6445748Z", + 1, + "" + ], + [ + true, + "System", + "S-1-5-18", + "SYSTEM", + "NT AUTHORITY", + "TokenElevationTypeDefault", + "NotWeb", + "2023-04-11T16:19:47.9610153Z", + 8068, + "wevtutil.exe", + "wevtutil.exe install-manifest \"C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\Microsoft-Antimalware-AMFilter.man\" \"/resourceFilePath:C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\Drivers\\WdFilter.sys\" \"/messageFilePath:C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\Drivers\\WdFilter.sys\" \"/parameterFilePath:C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\Drivers\\WdFilter.sys\"", + "PortableExecutable", + "2022-11-15T00:48:35.7968049Z", + "C:\\Windows\\System32\\wevtutil.exe", + 248320, + "e30e7a42a010bf95524514bdf2035695", + "20db4abf4539d2e054fbadde48078452a5a4adbca9eaeff66aba89f2c9164055", + "3f38989e61670025c2585a9e3cc8f1e1c9f229e9", + "05e6efaec9d8a034359a38d4fcc8a912", + "cf0902fe24ce006ccaa9675928bea6d0920b11d584012c298ff1e5b6b2ab972a", + "77b7ce8ed392986669a01eddb5e3a681033c5a9c", + "S-1-5-18", + "NT AUTHORITY", + "SYSTEM", + "2023-04-11T16:19:46.1453156Z", + 6404, + "MsMpEng.exe", + "\"MsMpEng.exe\"", + true, + "2023-04-11T16:19:42.0429875Z", + 1168, + "\\Device\\HarddiskVolume4\\Windows\\System32\\services.exe", + "System", + "TokenElevationTypeDefault", + "0335fe6b-dfdb-4312-a816-ef99d2d7eaee", + "10dbd4560e9b9288098b776a0ad3df8373b33444", + "", + "", + "2023-04-11T16:19:54.1518985Z", + "2023-04-11T16:22:08.6065521Z", + "a2021f1d-bbb5-4f28-8340-a805a97939b8", + "computerpii_0388bd87a76da2f542c3644a1fc4f6f8abd86c3c.na.domainpii_0d0de080a368dd713d459efd31113042be7f3e87.corp", + 156, + false, + "10.8295.19041.2673", + false, + "77687093572141183", + "", + "77687093572141200", + "2023-04-11T16:19:46.1453156Z", + 6404, + "2023-04-11T16:19:46.1453156Z", + "\\Device\\HarddiskVolume4\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0\\MsMpEng.exe", + 6404, + 18560, + "ActiveProcessStartkey", + "C:\\ProgramData\\Microsoft\\Windows Defender\\Platform\\4.18.2302.7-0", + 133544, + "2021-04-17T10:39:44Z", + "2023-03-28T11:57:59.906Z", + "2023-04-11T16:19:46.1977633Z", + "2023-03-28T11:58:03.4003743Z", + 128, + "NoTruncation", + 22815, + "", + "10.0", + "False", + "", + "", + "", + "", + true, + "StrPII_0b530e598af26ce0236fb3fcf7b34bc3b311ab82", + 999, + 999, + false, + null, + 128, + null, + null, + "", + null, + null, + "", + "", + "", + "Valid", + "OsVendor", + "Valid", + "OsVendor", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "4.18.2302.7", + "MsMpEng.exe", + "MsMpEng.exe", + "Antimalware Service Executable", + "Microsoft Corporation", + "Microsoft\u00ae Windows\u00ae Operating System", + "10.0.19041.2193", + "wevtutil.exe", + "wevtutil.exe", + "Eventing Command Line Utility", + false, + 0, + 0, + 27325456, + "StrPII_cdd4d90ff7c76f5bdabd9cde2dd51ad9c22c48a3", + "27124076", + 153, + "Asimov", + "", + false, + "2023-04-11T16:19:42.0429875Z", + 1168, + false, + "NotWeb", + "PortableExecutable", + "msmpeng.exe", + "", + "5fd6ed6ebbdd596abbddd66be799fd55dfb77aee5ef575d95a677a69ef7776aeabafb565f9bbfeaab5b99fe5656f699579de9fe7bde96be6f95b5bf5bbeaa7fa", + "bfd9f57fd7ad9a6eb7f9977bdfd9d969edf67ade7ee576dd6ea77a995ea665aea67a9ae5f7fed666e6faab95a97fae59799f5fa57efe7bf7f99a56d5f9ef675b", + "https://cloudscrubbed2eusprd.blob.core.windows.net/2023041116/2a4bc_ScrubbedCyberReport_20230411162233027_20230411162233027.cloud-scrubber-7bf5df7d8f-4rnm7_EastUs2_17_hourly.Bond.gz", + 164276, + "2022-10-24T21:35:41.1880025Z", + 1, + 195129, + "2023-03-20T22:00:37.1456262Z", + 1, + "" + ], + [ + false, + "", + "S-1-5-18", + "UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785", + "DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13", + "None", + "NotWeb", + "2023-04-11T16:20:25.009649Z", + 54964, + "mv", + "/bin/mv \"/Library/Caches/com.microsoft.autoupdate.helper/Clones.noindex/Microsoft Word.app/Contents/Frameworks/mso40ui.framework/Versions/A/Resources/cs.lproj/tbbtn2.strings.patched\" \"/Library/Caches/com.microsoft.autoupdate.helper/Clones.noindex/Microsoft Word.app/Contents/Frameworks/mso40ui.framework/Versions/A/Resources/cs.lproj/tbbtn2.strings\"", + "MachOExecutable", + "2023-01-11T07:03:20Z", + "/bin/mv", + 135552, + "9dca17e79b51583592a39a1f7e9c37ff", + "b673b36e048d105c0de07ff04198c2a009d0d747d1d20e21c8d3328a3bdb8fc5", + "1b530b960d9d0b9d1ff9feda005693e2ea5d4932", + "03c4befaffe8a85a772194289704557e", + "d82e2354b0c20f479770b490568c16a09144a274fb8511adb6a3a7fef9a013d1", + "426a4f1a6f1a3fdd7440ab9d3764a2df8fe363d4", + "S-1-5-18", + "DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13", + "UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785", + "2023-04-11T16:20:25.001841Z", + 54964, + "zsh", + "", + false, + "2023-04-11T16:18:50.962145Z", + 36594, + "zsh", + "", + "None", + "2686d60f-130f-4960-a5fb-237b76c3ce80", + "6bac8eee68741c6551e0ec5f9bd1f91f56c8af2c", + "", + "", + "2023-04-11T16:20:25.009649Z", + "2023-04-11T16:22:19.2935798Z", + "3927e8d2-82f1-4440-8433-7669a2f050c9", + "computerpii_c3c1e3b1dad5b7b1332fa45e3903b48157599b13.domainpii_13d91d49f57221f07fa53a662a35b5ddbedb6a14.com", + 102863, + null, + "20.123012.19830.0", + false, + "0", + "", + "0", + null, + 0, + "2023-04-11T16:20:25.001841Z", + "zsh", + 54964, + 4, + "Missing", + "/bin/", + 1377872, + null, + "2023-01-11T07:03:20Z", + "2023-01-11T07:03:20Z", + "2023-01-11T07:03:20Z", + 4, + "NoTruncation", + 0, + "", + "12.6", + "False", + "", + "", + "", + "", + true, + "StrPII_5e8f90b3f1b8c3ae87a982b737e3256735308067", + 0, + 0, + true, + null, + null, + null, + null, + "", + 36594, + 1, + "{\"Sid\":\"S-1-5-18\",\"Name\":\"UserPII_dc76e9f0c0006e8f919e0c515c66dbba3982f785\",\"DomainName\":\"DomainPII_c3c1e3b1dad5b7b1332fa45e3903b48157599b13\",\"LogonId\":0,\"AadUserId\":null,\"AadUserUpn\":null,\"PosixUserId\":0,\"PrimaryPosixGroup\":{\"Name\":\"StrPII_3a7195d99bee6bb15aacf02a10d56b4cbd65b1fe\",\"PosixGroupId\":0}}", + "{\"Name\":\"StrPII_3a7195d99bee6bb15aacf02a10d56b4cbd65b1fe\",\"PosixGroupId\":0}", + "", + "Unknown", + "Unknown", + "Unknown", + "Unknown", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + "", + false, + null, + null, + 20452320, + "StrPII_bc5699d9bfd152ecd5ecc8979872de278ffd5e58", + "f4cfd1ef-31e6-42cb-aef9-c3bf9346319e", + 26442, + "Asimov", + "", + false, + null, + null, + false, + "NotWeb", + "MachOExecutable", + "zsh", + "", + "", + "", + "https://cloudscrubbed5eusprd.blob.core.windows.net/2023041116/46371_ScrubbedCyberReport_20230411162233063_20230411162233063.cloud-scrubber-7bf5df7d8f-gqjg8_EastUs2_81_hourly.Bond.gz", + 59400, + "2022-09-08T20:42:21.6316953Z", + 1, + 60601, + "2022-09-08T19:06:13.3759065Z", + 1, + "" + ] + ] + }, + { + "FrameType": "DataTable", + "TableId": 2, + "TableKind": "QueryCompletionInformation", + "TableName": "QueryCompletionInformation", + "Columns": [ + { + "ColumnName": "Timestamp", + "ColumnType": "datetime" + }, + { + "ColumnName": "ClientRequestId", + "ColumnType": "string" + }, + { + "ColumnName": "ActivityId", + "ColumnType": "guid" + }, + { + "ColumnName": "SubActivityId", + "ColumnType": "guid" + }, + { + "ColumnName": "ParentActivityId", + "ColumnType": "guid" + }, + { + "ColumnName": "Level", + "ColumnType": "int" + }, + { + "ColumnName": "LevelName", + "ColumnType": "string" + }, + { + "ColumnName": "StatusCode", + "ColumnType": "int" + }, + { + "ColumnName": "StatusCodeName", + "ColumnType": "string" + }, + { + "ColumnName": "EventType", + "ColumnType": "int" + }, + { + "ColumnName": "EventTypeName", + "ColumnType": "string" + }, + { + "ColumnName": "Payload", + "ColumnType": "string" + } + ], + "Rows": [ + [ + "2023-04-11T16:25:15.1929946Z", + "KPC.execute;13f8c6dc-03f8-4498-b7bb-486a657a08a5", + "8777407f-caf3-4eed-ae73-90fba370bd93", + "16048e51-388e-4c83-85d3-ba5548d9ab34", + "5439d71c-10a2-4ce3-93ab-71b279fcfc4a", + 4, + "Info", + 0, + "S_OK (0)", + 4, + "QueryInfo", + "{\"Count\":1,\"Text\":\"Query completed successfully\"}" + ], + [ + "2023-04-11T16:25:15.1929946Z", + "KPC.execute;13f8c6dc-03f8-4498-b7bb-486a657a08a5", + "8777407f-caf3-4eed-ae73-90fba370bd93", + "16048e51-388e-4c83-85d3-ba5548d9ab34", + "5439d71c-10a2-4ce3-93ab-71b279fcfc4a", + 4, + "Info", + 0, + "S_OK (0)", + 6, + "EffectiveRequestOptions", + "{\"Count\":1,\"Text\":\"{\\\"DataScope\\\":\\\"All\\\",\\\"QueryConsistency\\\":\\\"strongconsistency\\\",\\\"MaxMemoryConsumptionPerIterator\\\":5368709120,\\\"MaxMemoryConsumptionPerQueryPerNode\\\":68719241216,\\\"QueryFanoutNodesPercent\\\":100,\\\"QueryFanoutThreadsPercent\\\":100}\"}" + ], + [ + "2023-04-11T16:25:15.2093468Z", + "KPC.execute;13f8c6dc-03f8-4498-b7bb-486a657a08a5", + "8777407f-caf3-4eed-ae73-90fba370bd93", + "16048e51-388e-4c83-85d3-ba5548d9ab34", + "5439d71c-10a2-4ce3-93ab-71b279fcfc4a", + 4, + "Info", + 0, + "S_OK (0)", + 5, + "WorkloadGroup", + "{\"Count\":1,\"Text\":\"default\"}" + ], + [ + "2023-04-11T16:25:15.2093468Z", + "KPC.execute;13f8c6dc-03f8-4498-b7bb-486a657a08a5", + "8777407f-caf3-4eed-ae73-90fba370bd93", + "16048e51-388e-4c83-85d3-ba5548d9ab34", + "5439d71c-10a2-4ce3-93ab-71b279fcfc4a", + 6, + "Stats", + 0, + "S_OK (0)", + 0, + "QueryResourceConsumption", + "{\"ExecutionTime\":0.625018,\"resource_usage\":{\"cache\":{\"memory\":{\"hits\":0,\"misses\":0,\"total\":0},\"disk\":{\"hits\":0,\"misses\":0,\"total\":0},\"shards\":{\"hot\":{\"hitbytes\":0,\"missbytes\":0,\"retrievebytes\":0},\"cold\":{\"hitbytes\":0,\"missbytes\":0,\"retrievebytes\":0},\"bypassbytes\":0}},\"cpu\":{\"user\":\"00:00:00.0156250\",\"kernel\":\"00:00:00.0156250\",\"total cpu\":\"00:00:00.0312500\"},\"memory\":{\"peak_per_node\":52075216},\"network\":{\"inter_cluster_total_bytes\":94609,\"cross_cluster_total_bytes\":0}},\"input_dataset_statistics\":{\"extents\":{\"total\":57426,\"scanned\":57426,\"scanned_min_datetime\":\"2023-02-10T03:33:04.4514216Z\",\"scanned_max_datetime\":\"2023-04-11T16:25:13.3655388Z\"},\"rows\":{\"total\":555679411525,\"scanned\":555679411525},\"rowstores\":{\"scanned_rows\":0,\"scanned_values_size\":0},\"shards\":{\"queries_generic\":0,\"queries_specialized\":0}},\"dataset_statistics\":[{\"table_row_count\":10,\"table_size\":23255}],\"cross_cluster_resource_usage\":{}}" + ] + ] + }, + { + "FrameType": "DataSetCompletion", + "HasErrors": false, + "Cancelled": false + } +] \ No newline at end of file diff --git a/tests/unit_test_lib.py b/tests/unit_test_lib.py index 48d841c66..a0d6207d2 100644 --- a/tests/unit_test_lib.py +++ b/tests/unit_test_lib.py @@ -6,12 +6,14 @@ """Unit test common utilities.""" import os +import sys from contextlib import contextmanager, suppress from os import chdir, getcwd from pathlib import Path -from typing import Any, Dict, Generator, Union +from typing import Any, Dict, Generator, Iterable, Optional, Union import nbformat +import yaml from filelock import FileLock from nbconvert.preprocessors import CellExecutionError, ExecutePreprocessor @@ -145,3 +147,88 @@ def exec_notebook( with open(nb_err, mode="w", encoding="utf-8") as file_handle: # type: ignore nbformat.write(nb_content, file_handle) raise + + +_DEFAULT_SENTINEL = object() + + +def create_get_config(settings: Dict[str, Any]): + """Return a get_config function with settings set to settings.""" + + def get_config( + setting_path: Optional[str] = None, default: Any = _DEFAULT_SENTINEL + ) -> Any: + """Get mocked setting item for path.""" + if setting_path is None: + return settings + try: + return _get_config(setting_path, settings) + except KeyError: + if default != _DEFAULT_SENTINEL: + return default + raise + + return get_config + + +def _get_config(setting_path: str, settings_dict: Dict[str, Any]) -> Any: + """Return value from setting_path.""" + path_elems = setting_path.split(".") + cur_node = settings_dict + for elem in path_elems: + cur_node = cur_node.get(elem, None) + if cur_node is None: + raise KeyError(f"{elem} value of {setting_path} is not a valid path") + return cur_node + + +@contextmanager +def custom_get_config( + monkeypatch: Any, + add_modules: Optional[Iterable[str]] = None, + settings: Optional[Dict[str, Any]] = None, + mp_path: Union[str, Path, None] = None, +) -> Generator[Dict[str, Any], None, None]: + """ + Context manager to temporarily set MSTICPYCONFIG path. + + Parameters + ---------- + monkeypatch : Any + Pytest monkeypatch fixture + module_name : str + The module to patch the get_config function for. + settings : Dict[str, Any] + The mocked settings to use. + mp_path : Union[str, Path] + Path to load msticpyconfig.yaml settings from. + + Yields + ------ + Dict[str, Any] + Custom settings. + + Raises + ------ + FileNotFoundError + If mp_path does not exist. + + """ + if mp_path: + if not Path(mp_path).is_file(): + raise FileNotFoundError( + f"Setting MSTICPYCONFIG to non-existent file {mp_path}" + ) + mp_text = Path(mp_path).read_text(encoding="utf-8") + settings = yaml.safe_load(mp_text) + + if settings: + core_modules = ["msticpy.common.pkg_config", "msticpy.common.settings"] + patched_get_config = create_get_config(settings=settings) + for module_name in core_modules + (list(add_modules or [])): + patched_module = sys.modules[module_name] + monkeypatch.setattr(patched_module, "get_config", patched_get_config) + print(f"using patched get_config for {module_name}") + yield settings + else: + raise ValueError("No settings specified") From 62cfaa543bc851628275ee4fb3d13b6d192bdbd8 Mon Sep 17 00:00:00 2001 From: hackeT <40039738+Tatsuya-hasegawa@users.noreply.github.com> Date: Thu, 11 May 2023 08:44:09 +0900 Subject: [PATCH 03/26] Fix a critical bug of Splunk results reader, lack of pagination (#657) * fix a critical bug of splunk result reader * typo pagenate -> paginate * Refactored code and reformatted long lines. Updated failing tests for new code. --------- Co-authored-by: Ian Hellen --- msticpy/data/drivers/splunk_driver.py | 151 +++++++++++++++++++---- tests/data/drivers/test_splunk_driver.py | 28 ++++- 2 files changed, 155 insertions(+), 24 deletions(-) diff --git a/msticpy/data/drivers/splunk_driver.py b/msticpy/data/drivers/splunk_driver.py index 68099da6a..b0c5694ef 100644 --- a/msticpy/data/drivers/splunk_driver.py +++ b/msticpy/data/drivers/splunk_driver.py @@ -4,7 +4,8 @@ # license information. # -------------------------------------------------------------------------- """Splunk Driver class.""" -from datetime import datetime +import logging +from datetime import datetime, timedelta from time import sleep from typing import Any, Dict, Iterable, Optional, Tuple, Union @@ -14,6 +15,7 @@ from ..._version import VERSION from ...common.exceptions import ( MsticpyConnectionError, + MsticpyDataQueryError, MsticpyImportExtraError, MsticpyUserConfigError, ) @@ -35,6 +37,8 @@ __version__ = VERSION __author__ = "Ashwin Patil" +logger = logging.getLogger(__name__) + SPLUNK_CONNECT_ARGS = { "host": "(string) The host name (the default is 'localhost').", @@ -73,7 +77,9 @@ def __init__(self, **kwargs): self.service = None self._loaded = True self._connected = False - self._debug = kwargs.get("debug", False) + if kwargs.get("debug", False): + logger.setLevel(logging.DEBUG) + self.set_driver_property( DriverProps.PUBLIC_ATTRS, { @@ -194,9 +200,17 @@ def query( Other Parameters ---------------- count : int, optional - Passed to Splunk oneshot method if `oneshot` is True, by default, 0 + Passed to Splunk job that indicates the maximum number + of entities to return. A value of 0 indicates no maximum, + by default, 0 oneshot : bool, optional Set to True for oneshot (blocking) mode, by default False + page_size = int, optional + Pass to Splunk results reader in terms of fetch speed, + which sets of result amount will be got at a time, + by default, 100 + timeout : int, optional + Amount of time to wait for results, by default 60 Returns ------- @@ -212,35 +226,38 @@ def query( # default to unlimited query unless count is specified count = kwargs.pop("count", 0) - # Normal, oneshot or blocking searches. Defaults to non-blocking - # Oneshot is blocking a blocking HTTP call which may cause time-outs - # https://dev.splunk.com/enterprise/docs/python/sdk-python/howtousesplunkpython/howtorunsearchespython + # Get sets of N results at a time, N=100 by default + page_size = kwargs.pop("page_size", 100) + + # Normal (non-blocking) searches or oneshot (blocking) searches. + # Defaults to Normal(non-blocking) + + # Oneshot is a blocking search that is scheduled to run immediately. + # Instead of returning a search job, this mode returns the results + # of the search once completed. + # Because this is a blocking search, the results are not available + # until the search has finished. + # https://dev.splunk.com/enterprise/docs/python/ + # sdk-python/howtousesplunkpython/howtorunsearchespython is_oneshot = kwargs.get("oneshot", False) if is_oneshot is True: + kwargs["output_mode"] = "json" query_results = self.service.jobs.oneshot(query, count=count, **kwargs) - reader = sp_results.ResultsReader(query_results) + reader = sp_results.JSONResultsReader( # pylint: disable=no-member + query_results + ) # due to DeprecationWarning of normal ResultsReader + resp_rows = [row for row in reader if isinstance(row, dict)] else: # Set mode and initialize async job kwargs_normalsearch = {"exec_mode": "normal"} - query_job = self.service.jobs.create(query, **kwargs_normalsearch) - - # Initiate progress bar and start while loop, waiting for async query to complete - progress_bar = tqdm(total=100, desc="Waiting Splunk job to complete") - while not query_job.is_done(): - current_state = query_job.state - progress = float(current_state["content"]["doneProgress"]) * 100 - progress_bar.update(progress) - sleep(1) - - # Update progress bar indicating completion and fetch results - progress_bar.update(100) - progress_bar.close() - reader = sp_results.ResultsReader(query_job.results()) + query_job = self.service.jobs.create( + query, count=count, **kwargs_normalsearch + ) + resp_rows, reader = self._exec_async_search(query_job, page_size, **kwargs) - resp_rows = [row for row in reader if isinstance(row, dict)] - if not resp_rows: + if len(resp_rows) == 0 or not resp_rows: print("Warning - query did not return any results.") return [row for row in reader if isinstance(row, sp_results.Message)] return pd.DataFrame(resp_rows) @@ -316,6 +333,94 @@ def driver_queries(self) -> Iterable[Dict[str, Any]]: ] return [] + def _exec_async_search(self, query_job, page_size, timeout=60): + """Execute an async search and return results.""" + # Initiate progress bar and start while loop, waiting for async query to complete + progress_bar = tqdm(total=100, desc="Waiting Splunk job to complete") + prev_progress = 0 + offset = 0 # Start at result 0 + start_time = datetime.now() + end_time = start_time + timedelta(seconds=timeout) + while True: + while not query_job.is_ready(): + sleep(1) + if self._retrieve_job_status(query_job, progress_bar, prev_progress): + break + if datetime.now() > end_time: + raise MsticpyDataQueryError( + "Timeout waiting for Splunk query to complete", + f"Job completion reported {query_job['doneProgress']}", + title="Splunk query timeout", + ) + sleep(1) + # Update progress bar indicating job completion + progress_bar.update(100) + progress_bar.close() + sleep(2) + + logger.info("Implicit parameter dump - 'page_size': %d", page_size) + return self._retrieve_results(query_job, offset, page_size) + + @staticmethod + def _retrieve_job_status(query_job, progress_bar, prev_progress): + """Poll the status of a job and update the progress bar.""" + stats = { + "is_done": query_job["isDone"], + "done_progress": float(query_job["doneProgress"]) * 100, + "scan_count": int(query_job["scanCount"]), + "event_count": int(query_job["eventCount"]), + "result_count": int(query_job["resultCount"]), + } + status = ( + "\r%(done_progress)03.1f%% %(scan_count)d scanned " + "%(event_count)d matched %(result_count)d results" + ) % stats + if prev_progress == 0: + progress = stats["done_progress"] + else: + progress = stats["done_progress"] - prev_progress + prev_progress = stats["done_progress"] + progress_bar.update(progress) + + if stats["is_done"] == "1": + logger.info(status) + logger.info("Splunk job completed.") + return True + return False + + @staticmethod + def _retrieve_results(query_job, offset, page_size): + """Retrieve the results of a job, decode and return them.""" + # Retrieving all the results by paginate + result_count = int( + query_job["resultCount"] + ) # Number of results this job returned + + resp_rows = [] + progress_bar_paginate = tqdm( + total=result_count, desc="Waiting Splunk result to retrieve" + ) + while offset < result_count: + kwargs_paginate = { + "count": page_size, + "offset": offset, + "output_mode": "json", + } + # Get the search results and display them + search_results = query_job.results(**kwargs_paginate) + # due to DeprecationWarning of normal ResultsReader + reader = sp_results.JSONResultsReader( # pylint: disable=no-member + search_results + ) + resp_rows.extend([row for row in reader if isinstance(row, dict)]) + progress_bar_paginate.update(page_size) + offset += page_size + # Update progress bar indicating fetch results + progress_bar_paginate.update(result_count) + progress_bar_paginate.close() + logger.info("Retrieved %d results.", len(resp_rows)) + return resp_rows, reader + @property def _saved_searches(self) -> Union[pd.DataFrame, Any]: """ diff --git a/tests/data/drivers/test_splunk_driver.py b/tests/data/drivers/test_splunk_driver.py index 15fc3a8a3..68c2d6f0f 100644 --- a/tests/data/drivers/test_splunk_driver.py +++ b/tests/data/drivers/test_splunk_driver.py @@ -13,6 +13,7 @@ from msticpy.common.exceptions import ( MsticpyConnectionError, + MsticpyDataQueryError, MsticpyNotConnectedError, MsticpyUserConfigError, ) @@ -69,15 +70,35 @@ def __init__(self, name, count): class _MockAsyncResponse: + stats = { + "isDone": "0", + "doneProgress": 0.0, + "scanCount": 1, + "eventCount": 100, + "resultCount": 100, + } + def __init__(self, query): self.query = query - def results(self): + def __getitem__(self, key): + """Mock method.""" + return self.stats[key] + + def results(self, **kwargs): return self.query def is_done(self): return True + def is_ready(self): + return True + + @classmethod + def set_done(cls): + cls.stats["isDone"] = "1" + cls.stats["doneProgress"] = 1 + class _MockSplunkCall: def create(query, **kwargs): @@ -260,6 +281,7 @@ def test_splunk_query_success(splunk_client, splunk_results): splunk_client.connect = cli_connect sp_driver = SplunkDriver() splunk_results.ResultsReader = _results_reader + splunk_results.JSONResultsReader = _results_reader # trying to get these before connecting should throw with pytest.raises(MsticpyNotConnectedError) as mp_ex: @@ -279,6 +301,10 @@ def test_splunk_query_success(splunk_client, splunk_results): check.is_not_instance(response, pd.DataFrame) check.equal(len(response), 0) + with pytest.raises(MsticpyDataQueryError): + df_result = sp_driver.query("some query", timeout=1) + + _MockAsyncResponse.set_done() df_result = sp_driver.query("some query") check.is_instance(df_result, pd.DataFrame) check.equal(len(df_result), 10) From eb08fdb1c64147c02d0408ec7c8c189001405ba4 Mon Sep 17 00:00:00 2001 From: FlorianBracq <97248273+FlorianBracq@users.noreply.github.com> Date: Fri, 12 May 2023 21:16:32 +0200 Subject: [PATCH 04/26] Update azure_kusto_driver.py (#664) Ensure that the driver is loaded so that queries can be executed. --- msticpy/data/drivers/azure_kusto_driver.py | 1 + 1 file changed, 1 insertion(+) diff --git a/msticpy/data/drivers/azure_kusto_driver.py b/msticpy/data/drivers/azure_kusto_driver.py index 81fe4b0e0..e67ddccda 100644 --- a/msticpy/data/drivers/azure_kusto_driver.py +++ b/msticpy/data/drivers/azure_kusto_driver.py @@ -174,6 +174,7 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): self.set_driver_property(DriverProps.PUBLIC_ATTRS, self._set_public_attribs()) self.set_driver_property(DriverProps.FILTER_ON_CONNECT, True) self.set_driver_property(DriverProps.EFFECTIVE_ENV, DataEnvironment.Kusto.name) + self._loaded = True def _set_public_attribs(self): """Expose subset of attributes via query_provider.""" From dec94ec37f8e828e51667c893af7b5680e1e519f Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Tue, 16 May 2023 07:08:34 -0700 Subject: [PATCH 05/26] Fix RTD Search - update conf.py Add sphinx jquery extension --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 34f5e924e..6783c3db6 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -59,7 +59,7 @@ "sphinx.ext.autodoc", "sphinx.ext.autosummary", "sphinx.ext.coverage", - # "sphinx.ext.githubpages", + "sphinxcontrib.jquery" "sphinx.ext.napoleon", "sphinx.ext.autosectionlabel", "sphinx.ext.intersphinx", From 0625d3e516326f1ceb553677feeb485612bea0bc Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Tue, 16 May 2023 15:19:44 -0700 Subject: [PATCH 06/26] Update requirements.txt Add sphinxcontrib-jquery to RTD requirements --- docs/requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/requirements.txt b/docs/requirements.txt index 32c3c5033..9a4e881c1 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -15,3 +15,4 @@ readthedocs-sphinx-ext==2.2.0 seed_intersphinx_mapping sphinx-rtd-theme==1.2.0 sphinx==6.1.3 +sphinxcontrib-jquery==4.1 From 91251b86b6b909fb0905ed3f17760291dc2792f5 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Tue, 16 May 2023 15:32:49 -0700 Subject: [PATCH 07/26] Update conf.py Fixing typo in RTD conf.py --- docs/source/conf.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/conf.py b/docs/source/conf.py index 6783c3db6..f32bd9c00 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -59,7 +59,7 @@ "sphinx.ext.autodoc", "sphinx.ext.autosummary", "sphinx.ext.coverage", - "sphinxcontrib.jquery" + "sphinxcontrib.jquery", "sphinx.ext.napoleon", "sphinx.ext.autosectionlabel", "sphinx.ext.intersphinx", From cb597471394d910b5ae30834fafda52d049e44b3 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Fri, 19 May 2023 14:37:19 -0700 Subject: [PATCH 08/26] Ianhelle/mp extensibility 2023 02 09 (#632) * second stage adding extension Co-authored-by: Ian Hellen * Working plugin code with tests. Still needs documentation * Fixing some mypy typing annotation errors * - Added documentation for PluginFramework - Added new document on createing TI providers - WritingTIAndContextProviders - Added docs for Development section in RTD - Adding ExtendingMsticpy section to RTD - moved sections for Queries, PivotFunctions, Creating data providers to this section - Have changed the internal _REQUIRED_PARAMS to use the same strings as in config and other places: - bulk edit of this http_provider, servicenow, alienvault_otx, greynoise, ibm_xforce, open_page_rank, virustotal Added ImportException trapping for mp_plugins.py Removing dev notebook - MSTICPyExtensions.ipynb * Moving some updates from Main into new extending/Queries.rst document * A couple of additions to docs - clarity and grammar * Test break due to merge * Addressing comments from Florian and Ryan's reviews. * Fixing issue with unit_test_lib not properly isolating temporary settings changes * Fixing bug in mp_plugins handling plugin paths * Adding locking around pivot data providers loader to fix config file for pivot tests. Changing test_nbinit.py to avoid using config locking and just use monkeypatch.setenv * Cleaned up ambiguity between DataEnvironment and environment_name in data_providers.py Removing unneeded comments in test_nbinit.py * Fixed error referencing "driver" variable in data_providers.py * Adding enviroment property for backward compatibility --------- Co-authored-by: Ian Hellen --- docs/source/DataAcquisition.rst | 2 +- docs/source/Development.rst | 94 +++ docs/source/ExtendingMsticpy.rst | 35 + docs/source/api/msticpy.init.rst | 1 + .../source/data_acquisition/DataProviders.rst | 617 +------------- docs/source/data_analysis/PivotFunctions.rst | 271 +------ docs/source/dev/CodingGuidelines.rst | 167 ++++ docs/source/extending/PivotFunctions.rst | 270 +++++++ docs/source/extending/PluginFramework.rst | 138 ++++ docs/source/extending/Queries.rst | 755 ++++++++++++++++++ .../WritingDataProviders.rst} | 12 +- .../WritingTIAndContextProviders.rst | 460 +++++++++++ docs/source/index.rst | 2 + msticpy/__init__.py | 25 +- msticpy/context/contextlookup.py | 5 +- .../context/contextproviders/servicenow.py | 6 +- msticpy/context/http_provider.py | 10 +- msticpy/context/lookup.py | 31 +- msticpy/context/tilookup.py | 5 +- msticpy/context/tiproviders/alienvault_otx.py | 4 +- msticpy/context/tiproviders/greynoise.py | 6 +- msticpy/context/tiproviders/ibm_xforce.py | 4 +- msticpy/context/tiproviders/intsights.py | 4 +- msticpy/context/tiproviders/open_page_rank.py | 4 +- msticpy/context/tiproviders/virustotal.py | 4 +- msticpy/data/core/data_providers.py | 54 +- msticpy/data/core/query_store.py | 14 +- msticpy/data/drivers/__init__.py | 30 +- msticpy/init/mp_plugins.py | 115 +++ tests/init/pivot/pivot_fixtures.py | 43 +- tests/init/test_mp_plugins.py | 126 +++ tests/init/test_nbinit.py | 30 +- tests/msticpyconfig-test.yaml | 2 + .../plugins/custom_prov_a_queries.yaml | 76 ++ tests/testdata/plugins/data_prov.py | 47 ++ tests/testdata/plugins/sql_test_queries.yaml | 509 ++++++++++++ tests/testdata/plugins/ti_prov.py | 113 +++ tests/unit_test_lib.py | 22 +- 38 files changed, 3121 insertions(+), 992 deletions(-) create mode 100644 docs/source/Development.rst create mode 100644 docs/source/ExtendingMsticpy.rst create mode 100644 docs/source/dev/CodingGuidelines.rst create mode 100644 docs/source/extending/PivotFunctions.rst create mode 100644 docs/source/extending/PluginFramework.rst create mode 100644 docs/source/extending/Queries.rst rename docs/source/{data_acquisition/WritingADataProvider.rst => extending/WritingDataProviders.rst} (98%) create mode 100644 docs/source/extending/WritingTIAndContextProviders.rst create mode 100644 msticpy/init/mp_plugins.py create mode 100644 tests/init/test_mp_plugins.py create mode 100644 tests/testdata/plugins/custom_prov_a_queries.yaml create mode 100644 tests/testdata/plugins/data_prov.py create mode 100644 tests/testdata/plugins/sql_test_queries.yaml create mode 100644 tests/testdata/plugins/ti_prov.py diff --git a/docs/source/DataAcquisition.rst b/docs/source/DataAcquisition.rst index 4013551e3..8837d0cfc 100644 --- a/docs/source/DataAcquisition.rst +++ b/docs/source/DataAcquisition.rst @@ -56,4 +56,4 @@ Contributing a Data Provider .. toctree:: :maxdepth: 2 - data_acquisition/WritingADataProvider + extending/WritingDataProviders diff --git a/docs/source/Development.rst b/docs/source/Development.rst new file mode 100644 index 000000000..385096c7d --- /dev/null +++ b/docs/source/Development.rst @@ -0,0 +1,94 @@ +MSTICPy Development Guidelines +============================== + +Contributions of improvements, fixes and new features are all +welcomed. Whether this is your first time contributing to a +project or you are a seasoned Open-Source contributor, +we welcome your contribution. In this guide you can find a few +pointers to help you create a great contribution. + +What to contribute +------------------ + +There are many things that can make a good contribution. +It might be a fix for a specific issue you have come across, +an improvement to an existing feature that you have thought +about such as a new data connector or threat intelligence provider, +or a completely new feature category. + +If you don’t have a specific idea in mind take a look at the +Issues page on GitHub: `https://github.com/microsoft/msticpy/issues`__ + +This page tracks a range of issues, enhancements, and features that +members of the community have thought of. The MSTICPy team uses these +issues as a way to track work and includes many things we have added ourselves. + +The issues are tagged with various descriptions that relate to the +type of issue. You may see some with the ‘good first issue’ tag. +These are issues that we think would make a good issue for someone +contributing to MSTICPy for the first time, however anyone is welcome +to work on any Issue. If you decide to start working on an Issue please +make a comment on the Issue so that we can assign it to you and other +members of the community know that it is being worked on and don’t +duplicate work. Also if you are unclear about what the Issue feel +free to comment on the Issue to get clarification from others. + + + +What makes a good contribution? +------------------------------- + +Whilst there is no one thing that makes a contribution good here are some guidelines: + +Scope +~~~~~ +Focus your contribution on a single thing per PR (Pull Request) raised, whether it +be a feature or a fix. If you have multiple things you want to contribute, +consider splitting them into multiple PRs. Keeping each PR to a single item +makes it easier for others to see what you are contributing and how it +fits with the rest of the project. + +Documentation +------------- +Make it clear what you are contributing, why its important, and how +it works. This provides much needed clarity for others when reviewing +contributions and helps to highlight the great value in your contribution. + +Unit test and test Coverage +--------------------------- +Write unit tests for your code. We use `pytest `__ +to run our tests. + +See the section :ref:`dev/CodingGuidelines:Unit Tests` for more information. + +Using Git +--------- +To contribute you will need to fork the MSTICPy repo. +**Create a branch** for your contribution, make the code changes +and then raise a PR to merge the changes back into +MSTICPy's main branch. Please *do not* make changes to `main` of your +fork and submit this as a PR. +You should also consider granting permission on your fork so that +we can push changes back to your forked branch. Sometimes, it's +quicker for us to make a quick change to fix something than to ask +you to make the change. If we cannot push any changes back +this is impossible to do. + +If you are unfamiliar with Git and GitHub you can find some +guidance here: https://docs.github.com/en/get-started/quickstart/set-up-git + + +Where to get help +----------------- +We are more than happy to help support your contributions, +if you need help you can comment on the Issue you are working on, +or email [msticpy@microsoft.com](mailto:msticpy@microsoft.com) + +You can also join our Discord +`#msticpy `. + + +.. toctree:: + :maxdepth: 2 + + dev/CodingGuidelines \ No newline at end of file diff --git a/docs/source/ExtendingMsticpy.rst b/docs/source/ExtendingMsticpy.rst new file mode 100644 index 000000000..6cc85d955 --- /dev/null +++ b/docs/source/ExtendingMsticpy.rst @@ -0,0 +1,35 @@ +Extending MSTICPy +================= + +Introduction to MSTICPy extensibility +------------------------------------- + +MSTICPy has several extensibility points. These range from adding +parameterized queries to writing your own data provider or +context provider. + +Some of these require coding, while others can be done +by creating YAML configuration files. For Data Providers and +Context/TI providers there is also a plugin model that allows +you to create private providers and load them from a local +path. + + +Contributing +------------ + +If you decide to extend MSTICPy in one of these ways and +think that this would be useful to other users of the +package, please consider contributing them into the package. + +Extension points documentation +------------------------------ + +.. toctree:: + :maxdepth: 2 + + extending/Queries + extending/PivotFunctions + extending/WritingDataProviders + extending/WritingTIAndContextProviders + extending/PluginFramework diff --git a/docs/source/api/msticpy.init.rst b/docs/source/api/msticpy.init.rst index 0d6867868..a193b678f 100644 --- a/docs/source/api/msticpy.init.rst +++ b/docs/source/api/msticpy.init.rst @@ -25,6 +25,7 @@ Submodules msticpy.init.azure_synapse_tools msticpy.init.logging msticpy.init.mp_pandas_accessors + msticpy.init.mp_plugins msticpy.init.nbinit msticpy.init.nbmagics msticpy.init.pivot diff --git a/docs/source/data_acquisition/DataProviders.rst b/docs/source/data_acquisition/DataProviders.rst index 6df6d2288..c1f07a903 100644 --- a/docs/source/data_acquisition/DataProviders.rst +++ b/docs/source/data_acquisition/DataProviders.rst @@ -546,617 +546,8 @@ get_host_events # query is now available as qry_prov.Custom.get_host_events(host_name="MyPC"....) +Adding queries to the MSTICPy query library +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Creating new queries --------------------- - -*msticpy* provides a number of -pre-defined queries to call with using the data package. You can also -add additional queries to be imported and used by your Query -Provider, these are defined in YAML format files and examples of these -files can be found at the msticpy GitHub site -https://github.com/microsoft/msticpy/tree/master/msticpy/data/queries. - -The required structure of these query definition files is as follows. - -At the top level the file has the following keys: -- **metadata** -- **defaults** -- **sources** - -These are described in the following sections. - -The metadata section -~~~~~~~~~~~~~~~~~~~~ - -- **version**: The version number of the definition file -- **description**: A description of the purpose of this collection of query - definitions -- **data_environments** []: A list of the Data Environments that - the defined queries can be run against (1 or more) -- **data_families** []: A list of Data Families the defined queries related - to, these families are defined as part of msticpy.data.query_defns but - you can add custom ones. -- **tags** []: A list of tags to help manage definition files (this is not - currently used - - -The defaults section -~~~~~~~~~~~~~~~~~~~~ - -A set of defaults that apply to all queries in the file. You -can use this section to define parameters that are common to all -of the queries in the file. Child keys of the ``defaults`` section -are inherited by the query definitions in the file. - -- **metadata**: Metadata regarding a query - - **data_source**: The data source to be used for the query -- **parameters**: parameter defaults for the queries (the format of - the parameters section is the same as described in - the sources section. - - -The sources section -~~~~~~~~~~~~~~~~~~~ - -Each key in the sources section defines a new query. The name of -the key is the query name and must be unique and a valid Python identifier. -Each query key has the following structure: - -- **description**: this is used to display help text for the query. -- **metadata**: (optional) - if you want to override the global metadata - for this query -- **args**: The primary item here is the query text. - - - **query**: usually a multi-line string that will be passed to the - data provider. The string is usually parameterized, the parameters - being denoted by surrounding them with single braces ({}). If - you need to include literal braces in the query, type two braces. - For example:: - "this {{literal_string}}" ->> "this {literal_string}" - Surround your query string with single quotes. - - **uri**: this is currently not used. -- **parameters**: The parameters section defines the name, data type and - optional default value for each parameter that will be substituted into - the query before being passed to the data provider. Each parameter - must have a unique name (for each query, not globally). All parameters - specified in the query text must have an entry here or in the file - defaults section. The parameter subsection has the following sub-keys: - - - **description**: A description of what the parameter is (used for generating - documentation strings. - - **type**: The data type of the parameter. Valid types include: "str", "int", - "float", "list" and "datetime". The list and datetime types cause additional - formatting to be applied (such as converting from a datestring) - - **default**: (optional) the default value for that parameter. Any parameter - that does not have a default value (here or in the file defaults section) - must be supplied at query time. - -Some common parameters used in the queries are: - -- **table**: making this a substitutable parameter allows you to use the same - query with different data sets. More commonly, you can add additional - filtering statements here, for example: - -.. code:: yaml - - parameters: - table: - description: The table name - type: str - default: SecurityEvent | where EventID == 4624 - -- **add_query_items**: This is a useful way of extending queries by adding - ad hoc statements to the end of the query (e.g. additional filtering order - summarization). - -Using known parameter names -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Try to use standard names for common entities and other parameter values. -This makes things easier for users of the queries and, in some cases, -enables functionality such as automatic insertion of times. - -Always use these names for common parameters - -================= ================================= ============= =============== -Query Parameter Description type default -================= ================================= ============= =============== -start The start datetime for the query datetime N/A -end The end datetime for the query datetime N/A -table The name of the main table (opt) str the table name -add_query_items Placeholder for additional query str "" -================= ================================= ============= =============== - -Entity names -For entities such as IP address, host name, account name, process, domain, etc., -always use one of the standard names - these are used by pivot functions to -map queries to the correct entity. - -For the current set of names see the following section in the Pivot Functions -documentation - :ref:`data_analysis/PivotFunctions:How are queries assigned to specific entities?` - - -Using yaml aliases and macros in your queries -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can use standard yaml aliasing to define substitutable strings in your -query definitions. E.g. you might have a parameter default that is a long -string expression. Define an alias in the ``aliases`` key of the file -metadata section. An alias is defined by prefixing the name with "&". -The alias is referenced (and inserted) by using the alias name prefixed -with "*" - -.. code:: yaml - - metadata: - ... - aliases: - - &azure_network_project '| project TenantId, TimeGenerated, - FlowStartTime = FlowStartTime_t, - FlowEndTime = FlowEndTime_t, - FlowIntervalEndTime = FlowIntervalEndTime_t, - FlowType = FlowType_s, - ResourceGroup = split(VM_s, "/")[0], - VMName = split(VM_s, "/")[1], - VMIPAddress = VMIP_s' - ... - sources: - list_azure_network_flows_by_host: - description: Retrieves Azure network analytics flow events. - ... - parameters: - ... - query_project: - description: Column project statement - type: str - default: *azure_network_project - - -You can also use *macros*, which work like parameters but are substituted -into the query before any parameter substitution is carried out. This -allows you to, for example, use a single base query but with different -filter and summarization clauses defined as macros. The macro text is -substituted into the main query. - -Macros are added to the ``query_macros`` subkey of a query. They have -two subkeys: description and value. value defines the text to be inserted. -The key name is the name of the macro. - -In the query, you denote the substitution point by surrounding the macro name -with "$<" and ">$". This is show in the example below. - -.. code:: yaml - - - query: ' - {table} - | where SubType_s == "FlowLog" - | where FlowStartTime_t >= datetime({start}) - | where FlowEndTime_t <= datetime({end}) - $$ - | where (AllowedOutFlows_d > 0 or AllowedInFlows_d > 0) - {query_project} - | extend AllExtIPs = iif(isempty(PublicIPs), pack_array(ExtIP), - iif(isempty(ExtIP), PublicIPs, array_concat(PublicIPs, pack_array(ExtIP))) - ) - | project-away ExtIP - | mvexpand AllExtIPs - {add_query_items}' - -Macros are particularly useful when combined with yaml aliases. You can, for -example, define a base query (using a yaml alias) with a macro reference in the -query body. Then in each query definition you can have different macro values -for the macro to be substituted. For example: - -.. code:: yaml - - metadata: - ... - aliases: - - &azure_network_base_query ' - {table} - | where SubType_s == "FlowLog" - | where FlowStartTime_t >= datetime({start}) - | where FlowEndTime_t <= datetime({end}) - $$ - | where (AllowedOutFlows_d > 0 or AllowedInFlows_d > 0) - {query_project} - | extend AllExtIPs = iif(isempty(PublicIPs), pack_array(ExtIP), - iif(isempty(ExtIP), PublicIPs, array_concat(PublicIPs, pack_array(ExtIP))) - ) - | project-away ExtIP - | mvexpand AllExtIPs - {add_query_items}' - ... - sources: - list_azure_network_flows_by_ip: - description: Retrieves Azure network analytics flow events. - args: - query: *azure_network_base_query - parameters: - query_project: - ... - end: - description: Query end time - type: datetime - query_macros: - query_condition: - description: Query-specific where clause - value: '| where (VMIP_s in ({ip_address_list}) - or SrcIP_s in ({ip_address_list}) - or DestIP_s in ({ip_address_list}) - )' - -This allows you define a series of related queries that have the -same basic logic but have different filter clauses. This is extremely useful -where the query is complex and allows you to keep a single copy. - -.. note:: Using aliases and macros complicates the logic for anyone - trying to read the query file, so use this sparingly. - - -Guidelines for creating and debugging queries -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It is often helpful to start with a working version of a query without -using any parameters. Just paste in a query that you know is working. Once -you have verified that this works and returns data as expected you can -start to parameterize it. - -As you add parameters you can expect to find escaping and quoting -issues with the parameter values. To see what the parameterized version -of the query (without submitting it to the data provider) run the query -with the first parameter "print". This will return the parameterized version -of the query as a string: - -.. code:: ipython3 - - qry_prov.SecurityEvents.my_new_query( - "print", - start=start_dt, - end=end_dt, - account="ian", - ) - - -There are also a number of tools within the package to assist in -validating new query definition files once created. - -:: - - data_query_reader.find_yaml_files - - Return iterable of yaml files found in `source_path`. - - Parameters - ---------- - source_path : str - The source path to search in. - recursive : bool, optional - Whether to recurse through subfolders. - By default False - - Returns - ------- - Iterable[str] - File paths of yaml files found. - - data_query_reader.validate_query_defs - - Validate content of query definition. - - Parameters - ---------- - query_def_dict : dict - Dictionary of query definition yaml file contents. - - Returns - ------- - bool - True if validation succeeds. - - Raises - ------ - ValueError - The validation failure reason is returned in the - exception message (arg[0]) - -validate_query_defs() does not perform comprehensive checks on the file -but does check key elements required in the file are present. - -.. code:: ipython3 - - for file in QueryReader.find_yaml_files(source_path="C:\\queries"): - with open(file) as f_handle: - yaml_file = yaml.safe_load(f_handle) - if QueryReader.validate_query_defs(query_def_dict = yaml_file) == True: - print(f' {file} is a valid query definition') - else: - print(f'There is an error with {file}') - - -.. parsed-literal:: - - C:\queries\example.yaml is a valid query definition - - -Adding a new set of queries and running them --------------------------------------------- - -Once you are happy with -a query definition file then you import it with - - *query_provider*.import_query_file(query_file= *path_to_query_file*) - -This will load the query file into the Query Provider's Query Store from -where it can be called. - -.. code:: ipython3 - - qry_prov.import_query_file(query_file='C:\\queries\\example.yaml') - -Once imported the queries in the files appear in the Query Provider's -Query Store alongside the others and can be called in the same manner as -pre-defined queries. - -If you have created a large number of query definition files and you -want to have the automatically imported into a Query Provider's query -store at initialization you can specify a directory containing these -queries in the msticpyconfig.yaml file under QueryDefinitions: Custom: - -For example if I have a folder at C:\\queries I will set the -config file to: - -.. code:: yaml - - QueryDefinitions: - Custom: - - C:\queries - - -Having the Custom field populated will mean the Query Provider will -automatically enumerate all the YAML files in the directory provided and -automatically import he relevant queries into the query store at -initialization alongside the default queries. Custom queries with the -same name as default queries will overwrite default queries. - -.. code:: ipython3 - - queries = qry_prov.list_queries() - for query in queries: - print(query) - - -.. parsed-literal:: - - LinuxSyslog.all_syslog - LinuxSyslog.cron_activity - LinuxSyslog.squid_activity - LinuxSyslog.sudo_activity - LinuxSyslog.syslog_example - LinuxSyslog.user_group_activity - LinuxSyslog.user_logon - SecurityAlert.get_alert - SecurityAlert.list_alerts - SecurityAlert.list_alerts_counts - SecurityAlert.list_alerts_for_ip - SecurityAlert.list_related_alerts - WindowsSecurity.get_host_logon - WindowsSecurity.get_parent_process - WindowsSecurity.get_process_tree - WindowsSecurity.list_host_logon_failures - WindowsSecurity.list_host_logons - WindowsSecurity.list_host_processes - WindowsSecurity.list_hosts_matching_commandline - WindowsSecurity.list_matching_processes - WindowsSecurity.list_processes_in_session - - -.. code:: ipython3 - - qry_prov.LinuxSyslog.syslog_example('?') - - -.. parsed-literal:: - - Query: syslog_example - Data source: LogAnalytics - Example query - - Parameters - ---------- - add_query_items: str (optional) - Additional query clauses - end: datetime - Query end time - host_name: str - Hostname to query for - query_project: str (optional) - Column project statement - (default value is: | project TenantId, Computer, Facility, TimeGener...) - start: datetime - Query start time - subscription_filter: str (optional) - Optional subscription/tenant filter expression - (default value is: true) - table: str (optional) - Table name - (default value is: Syslog) - Query: - {table} | where {subscription_filter} - | where TimeGenerated >= datetime({start}) - | where TimeGenerated <= datetime({end}) - | where Computer == "{host_name}" | take 5 - - -.. code:: ipython3 - - qry_prov.LinuxSyslog.syslog_example( - start='2019-07-21 23:43:18.274492', - end='2019-07-27 23:43:18.274492', - host_name='UbuntuDevEnv' - ) - - -.. raw:: html - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
TenantIdSourceSystemTimeGeneratedComputerEventTimeFacilityHostNameSeverityLevelSyslogMessageProcessIDHostIPProcessNameMGType_ResourceId
0b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.213UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvnoticeomsagent : TTY=unknown PWD=/opt/microsoft/om...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
1b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.313UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvinfopam_unix(sudo:session): session opened for use...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
2b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.917UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvinfopam_unix(sudo:session): session closed for use...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
3b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:50.793UbuntuDevEnv2019-07-25 15:15:50authprivUbuntuDevEnvinfopam_unix(cron:session): session closed for use...29486.010.0.1.4CRON00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
4b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:16:01.800UbuntuDevEnv2019-07-25 15:16:01authprivUbuntuDevEnvinfopam_unix(cron:session): session opened for use...29844.010.0.1.4CRON00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
-
- -| - -If you are having difficulties with a defined query and it is not -producing the expected results it can be useful to see the raw query -exactly as it is passed to the Data Environment. If you call a query -with "print" and the parameters required by that query it will construct -and print out the query string to be run. - -.. code:: ipython3 - - qry_prov.LinuxSyslog.syslog_example( - 'print', - start='2019-07-21 23:43:18.274492', - end='2019-07-27 23:43:18.274492', - host_name='UbuntuDevEnv' - ) - - - - -.. parsed-literal:: - - 'Syslog - | where true - | where TimeGenerated >= datetime(2019-07-21 23:43:18.274492) - | where TimeGenerated <= datetime(2019-07-27 23:43:18.274492) - | where Computer == "UbuntuDevEnv" - | take 5' - - +You can also add permanent parameterized queries to your data providers. +Read :doc:`../extending/Queries` for information on how to do this. diff --git a/docs/source/data_analysis/PivotFunctions.rst b/docs/source/data_analysis/PivotFunctions.rst index c396149c2..a2ec902c5 100644 --- a/docs/source/data_analysis/PivotFunctions.rst +++ b/docs/source/data_analysis/PivotFunctions.rst @@ -389,7 +389,8 @@ pivot functions with the ``pivots()`` function, the shortcut versions of the pivot functions appear without a "." in the name. You can create your own shortcut methods to existing or custom pivot functions -as described in `Creating and deleting shortcut pivot functions`_. +as described in +:ref:`../extending/PivotFunctions:Creating and deleting shortcut pivot functions`. Using the Pivot Browser @@ -2095,270 +2096,4 @@ and step details to be displayed as the pipeline is executed. Customizing and managing Pivots ------------------------------- -Adding custom functions to the pivot interface -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The pivot library supports adding functions as pivot functions from -any importable Python library. Not all functions will be wrappable. -Currently Pivot supports functions that take input parameters as -either scalar values (I'm including strings in this although that isn't -exactly correct), iterables or DataFrames with column specifications. - -You can create a persistent pivot function definition (that -will can be loaded from a yaml file) or an ad hoc definition -that you can create in code. The next two sections -describe creating a persistent function definition. Programmatically -creating pivots follows this. - -Information needed to define a pivot function -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you have a library function that you want to expose as a pivot function -you need to gather a bit of information about it. - -This table describes the configuration parameters needed to create a -pivot function (most are optional). - -+-------------------------+-------------------------------+------------+------------+ -| Item | Description | Required | Default | -+=========================+===============================+============+============+ -| src_module | The src_module to containing | Yes | - | -| | the class or function | | | -+-------------------------+-------------------------------+------------+------------+ -| class | The class containing function | No | - | -+-------------------------+-------------------------------+------------+------------+ -| src_func_name | The name of the function to | Yes | - | -| | wrap | | | -+-------------------------+-------------------------------+------------+------------+ -| func_new_name | Rename the function | No | - | -+-------------------------+-------------------------------+------------+------------+ -| input type | The input type that the | Yes | - | -| | wrapped function expects | | | -| | (DataFrame iterable value) | | | -+-------------------------+-------------------------------+------------+------------+ -| entity_map | Mapping of entity and | Yes | - | -| | attribute used for function | | | -+-------------------------+-------------------------------+------------+------------+ -| func_df_param_name | The param name that the | If DF | - | -| | function uses as input param | input | | -| | for DataFrame | | | -+-------------------------+-------------------------------+------------+------------+ -| func_df_col_param_name | The param name that function | If DF | - | -| | uses to identify the input | input | | -| | column name | | | -+-------------------------+-------------------------------+------------+------------+ -| func_out_column_name | Name of the column in the | If DF | - | -| | output DF to use as a key to | output | | -| | join | | | -+-------------------------+-------------------------------+------------+------------+ -| func_static_params | dict of static name/value | No | - | -| | params always sent to the | | | -| | function | | | -+-------------------------+-------------------------------+------------+------------+ -| func_input_value_arg | Name of the param that the | If not | - | -| | wrapped function uses for its | DF input | | -| | input value | | | -+-------------------------+-------------------------------+------------+------------+ -| can_iterate | True if the function supports | No | Yes | -| | being called multiple times | | | -+-------------------------+-------------------------------+------------+------------+ -| entity_container_name | The name of the container in | No | custom | -| | the entity where the func | | | -| | will appear | | | -+-------------------------+-------------------------------+------------+------------+ - -The ``entity_map`` item specifies which entity or entities the pivot function -will be added to. Each -entry requires an Entity name (see -:py:mod:`entities`) and an -entity attribute name. The attribute name is only used if you want to -use an instance of the entity as a parameter to the function. -If you don't care about this you can pick any attribute. - -For ``IpAddress`` in the (partial) example -below, the pivot function will try to extract the value of the -``Address`` attribute when an instance of IpAddress is used as a -function parameter. - -.. code:: yaml - - ... - entity_map: - IpAddress: Address - Host: HostName - Account: Name - ... - -This means that you can specify different attributes of the same entity -for different functions (or even for two instances of the same function) - -The ``func_df_param_name`` and ``func_df_col_param_name`` are needed -only if the source function (i.e. the function to be wrapped as a pivot -function) takes a DataFrame and column name as input -parameters. - -``func_out_column_name`` is needed if the source function returns a -DataFrame. In order to *join* input data with output data this needs to be -the column in the output that has the same value as the function input -For example, if your function processes IP addresses and returns the IP -in a column is named "ip_addr", put "ip_addr" as the value of ``func_out_column_name``. - -Adding ad hoc pivot functions in code -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can also add ad hoc functions as pivot functions in code. - -To do this use the Pivot method -:py:meth:`add_pivot_function` - -You can supply the -pivot registration parameters as keyword arguments to this function: - -.. code:: ipython3 - - def my_func(input: str): - return input.upper() - - Pivot.add_pivot_function( - func=my_func, - container="change_case", - input_type="value", - entity_map={"Host": "HostName"}, - func_input_value_arg="input", - func_new_name="upper_name", - ) - -Alternatively, you can create a -:py:class:`PivotRegistration` -object and supply that (along with the ``func`` parameter), to -``add_pivot_function``. - -.. code:: ipython3 - - from msticpy.init.pivot_core.pivot_register import PivotRegistration - - def my_func(input: str): - return input.upper() - - piv_reg = PivotRegistration( - input_type="value", - entity_map={"Host": "HostName"}, - func_input_value_arg="input", - func_new_name="upper_name" - ) - - Pivot.add_pivot_function(my_func, piv_reg, container="change_case") - -The function parameters and PivotRegistration attributes are -described in the previous section `Information needed to define a pivot function`_. - -Creating a persistent pivot function definition -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -You can also add pivot definitions to yaml files and load your -pivots from this. - -The top-level element of the file is ``pivot_providers``. - -Example from the *MSTICPy* ip_utils ``who_is`` function - -.. code:: yaml - - pivot_providers: - ... - who_is: - src_module: msticpy.context.ip_utils - src_func_name: get_whois_df - func_new_name: whois - input_type: dataframe - entity_map: - IpAddress: Address - func_df_param_name: data - func_df_col_param_name: ip_column - func_out_column_name: query - func_static_params: - all_columns: True - show_progress: False - func_input_value_arg: ip_address - -.. note:: the library also support creating pivots from ad hoc - functions created in the current notebook (see below). - -You can also put this function into a Python module. -If your module is in the current directory and is called -``my_new_module``, the value you specify for -``src_module`` will be "my_new_module". - -Once you have your yaml definition file you can call -:py:meth:`register_pivot_providers` - -.. code:: ipython3 - - Pivot.register_pivot_providers( - pivot_reg_path=path_to_your_yaml, - namespace=globals(), - def_container="my_container", - force_container=True - ) - -.. warning:: The pivot functions created will not persist - across notebook sessions. You will need to call - ``register_pivot_providers`` with your yaml files each - time you start a new session. Currently there is no option - to automate this via msticpyconfig.yaml but this would be - easy to add - please open an issue in - `the MSTICPy repo `_ - if you would like to see this. - - -Creating and deleting shortcut pivot functions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you are adding pivot functions of your own, you can add shortcuts -(i.e. direct methods of the entity, rather than methods in sub-containers) -to those functions. - -Every entity class has the class method -:py:meth:`make_pivot_shortcut`. -You can use this to add a shortcut to an existing pivot function on that -entity. Note that you must call this method on the entity *class* and not -on an instance of that Entity. - -The parameters that you must supply are ``func_name`` and ``target``. The former -is the relative path to the pivot function that you want to make the shortcut -to, e.g. for ``IpAddress.util.whois`` you would use the string "util.whois". -``target`` is the string name that you want the shortcut function to be called. -This should be a valid Python identifier - a string starting with a letter or -underscore, followed by any combination of letters, digits and underscores. If -you supply a string that is not a valid identifier, the function will try to -transform it into one. - -.. code:: ipython3 - - >>> IpAddress.make_pivot_shortcut(func_name="util.whois", target="my_whois") - >>> IpAddress.my_whois("157.53.1.1") - -.. parsed-literal:: - - ip_column AsnDescription whois_result - 157.53.1.1 NA {'nir': None, 'asn_registry': 'arin', ... - - -If the shortcut function already exists, you will get an error (AttributeError). -You can force overwriting of an existing shortcut by adding ``overwrite=True``. - -To delete a shortcut use -:py:meth:`del_pivot_shortcut`, -giving the single parameter ``func_name`` with the name of the shortcut function -you want to remove. - - -Removing pivot functions from an entity or all entities -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Although not a common operation you can remove *all* pivot functions -from an entity or from all entities. - -See -:py:meth:`remove_pivot_funcs` -for more details. +See :doc:`../extending/PivotFunctions` diff --git a/docs/source/dev/CodingGuidelines.rst b/docs/source/dev/CodingGuidelines.rst new file mode 100644 index 000000000..0e574e7b9 --- /dev/null +++ b/docs/source/dev/CodingGuidelines.rst @@ -0,0 +1,167 @@ +Code Guidelines +=============== + +Unit Tests +---------- + +We use pytest although some of the older tests are in +Python `unittest` format. + +Avoid making any calls to internet services in your unit tests. +If your feature connects to a live service, make sure that your unit tests mock +the sample responses expected. We use httpx for all our http requests - +you can use `respx `__ to mock +http responses (search for "@respx" in our tests to see examples +of how this works). + +Ensure your contribution has the highest possible of test coverage. +You should aim for a least 80% coverage and ideally reach 100%. +If you can't reach 80% for what ever reason let us know when you +raise a PR and we can work with you on this. + +You can map your test coverage using the following command: + +.. code:: bash + + pytest --cov=msticpy --cov-report=html + +This will create a folder called htmlcov in your current directory. Open +the index.html file in this folder to see the coverage report. + +You can also execute a subset of tests and check the coverage for +the areas of code that you are interesting in testing. + +.. code:: bash + + pytest --cov=msticpy.data.drivers --cov-report=html tests/data/drivers/test_xyz_driver.py + + +Type hints +---------- + +Use type annotations for parameters and return values in public methods, +properties and functions. + +.. code:: python + + from typing import Any, Dict, Optional, Union + + ... + + def build_process_tree( + procs: pd.DataFrame, + schema: Union[ProcSchema, Dict[str, Any]] = None, + show_summary: bool = False, + debug: bool = False, + **kwargs, + ) -> pd.DataFrame: + """ + Build process trees from the process events. + +`Python Type Hints +documentation `__ + +Docstrings +---------- + +Our documentation is automatically built for Readthedocs using Sphinx. +All public modules, functions, classes and methods should be documented +using the numpy documenation standard. + +.. code:: python + + def build_process_tree( + procs: pd.DataFrame, + schema: Union[ProcSchema, Dict[str, Any]] = None, + show_summary: bool = False, + debug: bool = False, + **kwargs, + ) -> pd.DataFrame: + """ + Build process trees from the process events. + + Parameters + ---------- + procs : pd.DataFrame + Process events (Windows 4688 or Linux Auditd) + schema : Union[ProcSchema, Dict[str, Any]], optional + The column schema to use, by default None. + If supplied as a dict it must include definitions for the + required fields in the ProcSchema class + If None, then the schema is inferred + show_summary : bool + Shows summary of the built tree, default is False. + debug : bool + If True produces extra debugging output, + by default False + + Returns + ------- + pd.DataFrame + Process tree dataframe. + + See Also + -------- + ProcSchema + + """ + +`numpy docstring +guide `__ + +Code Formatting +--------------- + +We use black everywhere and enforce this in the build. + +`Black - The Uncompromising Code +Formatter `__ + +Linters/Code Checkers +--------------------- + +We use the following code checkers: - pylint: ``pylint msticpy`` - mypy: +``mypy msticpy`` - bandit: +``bandit -r -lll -s B303,B404,B603,B607 msticpy`` - flake8: +``flake8 --max-line-length=90 --ignore=E501,W503 --exclude tests`` - +pydocstyle: ``pydocstyle --convention=numpy msticpy`` - isort: +``isort --profile black msticpy`` + +Pre-Commit +---------- + +We have a pre-commit configuration in the msticpy repo. This runs the +checks above (apart from mypy) when you commit. See `Pre-Commit +Script `__ +for more details. + +VSCode support +-------------- + +See this page for task definitions to run `linters/checkers in +VSCode `__ + +Create a branch +--------------- + +Before you submit a PR, create working branch in your fork and put your +changes in that. It's going to make it easier for you to re-sync the +main branch if this gets updated while you are working on your changes. + +See also +-------- + +- `Good coding + guidelines `__ +- `VS Code build + tasks `__ +- `Using + Pre-commit `__ + +A musical guide +--------------- + +`The PEP8 Song `__ + +Brilliantly written and performed by +[@lemonsaurus_rex](https://twitter.com/lemonsaurus_rex) diff --git a/docs/source/extending/PivotFunctions.rst b/docs/source/extending/PivotFunctions.rst new file mode 100644 index 000000000..2e7dc2ce6 --- /dev/null +++ b/docs/source/extending/PivotFunctions.rst @@ -0,0 +1,270 @@ +Adding Pivot functions +====================== + +See :doc:`../data_analysis/PivotFunctions` for more details on use +of pivot functions. + +The pivot library supports adding functions as pivot functions from +any importable Python library. Not all functions will be wrappable. +Currently Pivot supports functions that take input parameters as +either scalar values (I'm including strings in this although that isn't +exactly correct), iterables or DataFrames with column specifications. + +You can create a persistent pivot function definition (that +can be loaded from a yaml file) or an ad hoc definition +that you can create in code. The next two sections +describe creating a persistent function definition. Programmatically +creating pivots follows this. + +Information needed to define a pivot function +--------------------------------------------- + +If you have a library function that you want to expose as a pivot function +you need to gather a bit of information about it. + +This table describes the configuration parameters needed to create a +pivot function (most are optional). + ++-------------------------+-------------------------------+------------+------------+ +| Item | Description | Required | Default | ++=========================+===============================+============+============+ +| src_module | The src_module to containing | Yes | - | +| | the class or function | | | ++-------------------------+-------------------------------+------------+------------+ +| class | The class containing function | No | - | ++-------------------------+-------------------------------+------------+------------+ +| src_func_name | The name of the function to | Yes | - | +| | wrap | | | ++-------------------------+-------------------------------+------------+------------+ +| func_new_name | Rename the function | No | - | ++-------------------------+-------------------------------+------------+------------+ +| input type | The input type that the | Yes | - | +| | wrapped function expects | | | +| | (DataFrame iterable value) | | | ++-------------------------+-------------------------------+------------+------------+ +| entity_map | Mapping of entity and | Yes | - | +| | attribute used for function | | | ++-------------------------+-------------------------------+------------+------------+ +| func_df_param_name | The param name that the | If DF | - | +| | function uses as input param | input | | +| | for DataFrame | | | ++-------------------------+-------------------------------+------------+------------+ +| func_df_col_param_name | The param name that function | If DF | - | +| | uses to identify the input | input | | +| | column name | | | ++-------------------------+-------------------------------+------------+------------+ +| func_out_column_name | Name of the column in the | If DF | - | +| | output DF to use as a key to | output | | +| | join | | | ++-------------------------+-------------------------------+------------+------------+ +| func_static_params | dict of static name/value | No | - | +| | params always sent to the | | | +| | function | | | ++-------------------------+-------------------------------+------------+------------+ +| func_input_value_arg | Name of the param that the | If not | - | +| | wrapped function uses for its | DF input | | +| | input value | | | ++-------------------------+-------------------------------+------------+------------+ +| can_iterate | True if the function supports | No | Yes | +| | being called multiple times | | | ++-------------------------+-------------------------------+------------+------------+ +| entity_container_name | The name of the container in | No | custom | +| | the entity where the func | | | +| | will appear | | | ++-------------------------+-------------------------------+------------+------------+ + +The ``entity_map`` item specifies which entity or entities the pivot function +will be added to. Each +entry requires an Entity name (see +:py:mod:`entities`) and an +entity attribute name. The attribute name is only used if you want to +use an instance of the entity as a parameter to the function. +If you don't care about this you can pick any attribute. + +For ``IpAddress`` in the (partial) example +below, the pivot function will try to extract the value of the +``Address`` attribute when an instance of IpAddress is used as a +function parameter. + +.. code:: yaml + + ... + entity_map: + IpAddress: Address + Host: HostName + Account: Name + ... + +This means that you can specify different attributes of the same entity +for different functions (or even for two instances of the same function) + +The ``func_df_param_name`` and ``func_df_col_param_name`` are needed +only if the source function (i.e. the function to be wrapped as a pivot +function) takes a DataFrame and column name as input +parameters. + +``func_out_column_name`` is needed if the source function returns a +DataFrame. In order to *join* input data with output data this needs to be +the column in the output that has the same value as the function input +For example, if your function processes IP addresses and returns the IP +in a column named "ip_addr", put "ip_addr" as the value of ``func_out_column_name``. + +Adding ad hoc pivot functions in code +------------------------------------- + +You can also add ad hoc functions as pivot functions in code. + +To do this use the Pivot method +:py:meth:`add_pivot_function` + +You can supply the +pivot registration parameters as keyword arguments to this function: + +.. code:: python3 + + def my_func(input: str): + return input.upper() + + Pivot.add_pivot_function( + func=my_func, + container="change_case", + input_type="value", + entity_map={"Host": "HostName"}, + func_input_value_arg="input", + func_new_name="upper_name", + ) + +Alternatively, you can create a +:py:class:`PivotRegistration` +object and supply that (along with the ``func`` parameter), to +``add_pivot_function``. + +.. code:: python3 + + from msticpy.init.pivot_core.pivot_register import PivotRegistration + + def my_func(input: str): + return input.upper() + + piv_reg = PivotRegistration( + input_type="value", + entity_map={"Host": "HostName"}, + func_input_value_arg="input", + func_new_name="upper_name" + ) + + Pivot.add_pivot_function(my_func, piv_reg, container="change_case") + +The function parameters and PivotRegistration attributes are +described in the previous section `Information needed to define a pivot function`_. + +Creating a persistent pivot function definition +----------------------------------------------- + +You can also add pivot definitions to yaml files and load your +pivots from this. + +The top-level element of the file is ``pivot_providers``. + +Example from the *MSTICPy* ip_utils ``who_is`` function + +.. code:: yaml + + pivot_providers: + ... + who_is: + src_module: msticpy.context.ip_utils + src_func_name: get_whois_df + func_new_name: whois + input_type: dataframe + entity_map: + IpAddress: Address + func_df_param_name: data + func_df_col_param_name: ip_column + func_out_column_name: query + func_static_params: + all_columns: True + show_progress: False + func_input_value_arg: ip_address + +.. note:: the library also support creating pivots from ad hoc + functions created in the current notebook (see below). + +You can also put this function into a Python module. +If your module is in the current directory and is called +``my_new_module``, the value you specify for +``src_module`` will be "my_new_module". + +Once you have your yaml definition file you can call +:py:meth:`register_pivot_providers` + +.. code:: python3 + + Pivot.register_pivot_providers( + pivot_reg_path=path_to_your_yaml, + namespace=globals(), + def_container="my_container", + force_container=True + ) + +.. warning:: The pivot functions created will not persist + across notebook sessions. You will need to call + ``register_pivot_providers`` with your yaml files each + time you start a new session. Currently there is no option + to automate this via msticpyconfig.yaml but this would be + easy to add - please open an issue in + `the MSTICPy repo `_ + if you would like to see this. + + +Creating and deleting shortcut pivot functions +---------------------------------------------- + +If you are adding pivot functions of your own, you can add shortcuts +(i.e. direct methods of the entity, rather than methods in sub-containers) +to those functions. + +Every entity class has the class method +:py:meth:`make_pivot_shortcut`. +You can use this to add a shortcut to an existing pivot function on that +entity. Note that you must call this method on the entity *class* and not +on an instance of that Entity. + +The parameters that you must supply are ``func_name`` and ``target``. The former +is the relative path to the pivot function that you want to make the shortcut +to, e.g. for ``IpAddress.util.whois`` you would use the string "util.whois". +``target`` is the string name that you want the shortcut function to be called. +This should be a valid Python identifier - a string starting with a letter or +underscore, followed by any combination of letters, digits and underscores. If +you supply a string that is not a valid identifier, the function will try to +transform it into one. + +.. code:: python3 + + >>> IpAddress.make_pivot_shortcut(func_name="util.whois", target="my_whois") + >>> IpAddress.my_whois("157.53.1.1") + +.. parsed-literal:: + + ip_column AsnDescription whois_result + 157.53.1.1 NA {'nir': None, 'asn_registry': 'arin', ... + + +If the shortcut function already exists, you will get an error (AttributeError). +You can force overwriting of an existing shortcut by adding ``overwrite=True``. + +To delete a shortcut use +:py:meth:`del_pivot_shortcut`, +giving the single parameter ``func_name`` with the name of the shortcut function +you want to remove. + + +Removing pivot functions from an entity or all entities +------------------------------------------------------- + +Although not a common operation you can remove *all* pivot functions +from an entity or from all entities. + +See +:py:meth:`remove_pivot_funcs` +for more details. \ No newline at end of file diff --git a/docs/source/extending/PluginFramework.rst b/docs/source/extending/PluginFramework.rst new file mode 100644 index 000000000..ebb40573b --- /dev/null +++ b/docs/source/extending/PluginFramework.rst @@ -0,0 +1,138 @@ +MSTICPy Plugin Framework +======================== + +MSTICPy has several extensibility points, where you +can create your own modules to support data and +enrichment sources. + +These currently include: + +- Data Providers +- Threat Intelligence Providers +- Context Providers + +If you create one of these and think that it would be useful +to other users, please consider contributing it to MSTICPy. + +You can also create and keep local provider modules and +have them loaded into MSTICPy to work alongside the built-in +providers. This might be useful if you are creating something +that is very specific to your organization, for example. + + +For either of these cases, your classes must be derived +from the corresponding MSTICPy base classes. To read more +about building data, TI and context providers +see the following pages: + +- :doc:`WritingDataProviders` +- :doc:`WritingTIAndContextProviders` + +Specific Guidelines for plugin types +------------------------------------ + +TI and Context Providers +~~~~~~~~~~~~~~~~~~~~~~~~ + +Create a class attribute ``PROVIDER_NAME`` and assign +a friendly name to your provider. This is not mandatory - +if the class has no ``PROVIDER_NAME`` attribute, the +friendly name will default to the name of the class. + +.. code:: python3 + + class TIProviderTest(HttpTIProvider): + """Custom IT provider TI Base.""" + + PROVIDER_NAME = "MyProvider" + + +DataProviders +~~~~~~~~~~~~~ + +When you load a Data provider in MSTICPy you need to +pass the name of the ``DataEnvironment`` to the +:py:class:`QueryProvider ` +class. E.g. + +.. code:: python3 + + qry_prov = mp.QueryProvider("MyDataSource") + +By default, the name used to load your provider will be +name of your provider class. You can customize this by adding +a ``DATA_ENVIRONMENTS`` (list or tuple) attribute to your class. This should +be a list of strings. You can load your driver in the QueryProvider +by supplying any of the names in this list or tuple. +If you also want to use the name of the class, add it to the list. + +.. code:: python3 + + class CustomDataProvB(CustomDataProvA): + """Custom provider.""" + + DATA_ENVIRONMENTS = ["SQLTestProvider", "SQLProdProvider"] + +The provider will be registered to load when any of the strings +assigned here is passed as the QueryProvider identifier. + +Using multiple identifiers allows you to use aliases for +the provider. + +Additionally, because the Data Environment identifier is +passed to your provider class (as the parameter ``data_environment``) +when it is loaded, you can also +have alternative behavior coded into the ``__init__`` and other +methods of your class. For example, you might have a single provider class +that can work with two different versions of an API. + +Loading plugin classes +---------------------- + +Assuming that you have created one or more DataProvider +or Context/TI Provider classes, you should put these +modules in one or more folders accessible to your notebook +or python environment. + +You can load modules interactively or add these paths +to your ``msticpyconfig.yaml`` to have them loaded automatically +each time you import MSTICPy. + +Loading modules interactively +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +To load modules from a folder run the +:py:func:`load_plugins ` function. + +.. code:: python3 + + import msticpy as mp + + mp.load_plugins(plugin_paths="/my_modules") + + # or multiple paths + mp.load_plugins( + plugin_paths=["./my_modules", "./my_other_modules"] + ) + +Loading modules from configuration +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Add plugin module paths to ``msticpyconfig.yaml`` you can +tell MSTICPy to always try to load plugins from these paths. + +Add the following entry to ``msticpyconfig.yaml``: + +.. code-block:: yaml + :emphasize-lines: 4, 5 + + ... + Custom: + - "testdata" + PluginFolders: + - tests/testdata/plugins + Azure: + cloud: "global" + auth_methods: ["cli", "msi", "interactive"] + +You can include multiple paths under the ``PluginFolders`` key. diff --git a/docs/source/extending/Queries.rst b/docs/source/extending/Queries.rst new file mode 100644 index 000000000..3a067e44a --- /dev/null +++ b/docs/source/extending/Queries.rst @@ -0,0 +1,755 @@ +Adding Queries to MSTICPy +========================= + +See :doc:`../data_acquisition/DataProviders` for more details on use +of data queries. + +*msticpy* provides a number of +pre-defined queries to call with using the data package. You can also +add additional queries to be imported and used by your Query +Provider, these are defined in YAML format files and examples of these +files can be found at the msticpy GitHub site +https://github.com/microsoft/msticpy/tree/master/msticpy/data/queries. + +Here is an example of a query definition file: + +.. code:: yaml + + metadata: + version: 1 + description: Kql Sentinel Windows Logon Event Queries + data_environments: [MSSentinel] + data_families: [WindowsSecurity] + tags: ["process", "windows", "processtree", "session"] + defaults: + parameters: + start: + description: Query start time + type: datetime + end: + description: Query end time + type: datetime + table: + description: Table name + type: str + default: "SecurityEvent" + sources: + get_host_logon: + description: Retrieves the logon event for the session id on the host + metadata: + args: + query: ' + {table} + | where EventID == 4624 + | where Computer has "{host_name}" + | where TimeGenerated >= datetime({start}) + | where TimeGenerated <= datetime({end}) + | where TargetLogonId == "{logon_session_id}" + {add_query_items}' + parameters: + host_name: + description: Name of host + type: str + logon_session_id: + description: The logon session ID of the source process + type: str + list_host_logons: + description: Retrieves the logon events on the host + metadata: + args: + query: ' + {table} + | where EventID == 4624 + | where Computer has "{host_name}" + | where TimeGenerated >= datetime({start}) + | where TimeGenerated <= datetime({end}) + {add_query_items}' + parameters: + host_name: + description: Name of host + type: str + + +The required structure of these query definition files is as follows. + +At the top level the file has the following keys: +- **metadata** +- **defaults** +- **sources** + +These are described in the following sections. + +The metadata section +-------------------- + +.. code:: yaml + + metadata: + version: 1 + description: Kql Sentinel Windows Logon Event Queries + data_environments: [MSSentinel] + data_families: [WindowsSecurity] + tags: ["process", "windows", "processtree", "session"] + +- **version**: The version number of the definition file +- **description**: A description of the purpose of this collection of query + definitions +- **data_environments** []: A list of the Data Environments that + the defined queries can be run against (1 or more). This value defines + which QueryProvider instances the queries will be attached to. +- **data_families** []: A list of Data Families the defined queries related + to. These are just arbitrary strings that allow you to group related + queries in the same subcontainer (e.g. Logons, Processes). If you add + more than one `data_family` the query will be added to each group + (sub-container) +- **tags** []: A list of tags to help manage definition files (this is not + currently used) + +.. note:: You can override metadata items for individual queries. + +The defaults section +-------------------- + +.. code:: yaml + + defaults: + parameters: + start: + description: Query start time + type: datetime + end: + description: Query end time + type: datetime + +This section contains defaults that apply to all queries in the file. The most +common use for this section is to define parameters that are common to all +or many queries in the file. Child keys of the ``defaults`` section +are inherited by the individual query definitions in the file. + +.. note:: queries that do not use parameters defined in defaults + will just ignore them + +- **parameters**: parameter defaults for the queries (the format of + the parameters section is the same as described in + the sources section.) + + +The sources section +------------------- + +.. code:: yaml + + sources: + get_host_logon: + description: Retrieves the logon event for the session id on the host + metadata: + args: + query: ' + {table} + | where EventID == 4624 + | where Computer has "{host_name}" + | where TimeGenerated >= datetime({start}) + | where TimeGenerated <= datetime({end}) + | where TargetLogonId == "{logon_session_id}" + {add_query_items}' + parameters: + host_name: + description: Name of host + type: str + logon_session_id: + description: The logon session ID of the source process + type: str + +Each key in the sources section defines a new query (``get_host_logon`` in +the example above). The name of +the key is the query name and must be unique and a valid Python identifier. + +Each query key has the following structure: + +- **description**: this is used to display help text for the query. +- **metadata**: (optional) - if you want to override the global metadata + for this query +- **args**: The primary item here is the query text. + + - **query**: usually a multi-line string that will be passed to the + data provider. The string is usually parameterized, the parameters + being denoted by surrounding them with single braces ({}). If + you need to include literal braces in the query, type two braces. + For example:: + "this {{literal_string}}" ->> "this {literal_string}" + Surround your query string with single quotes. + - **uri**: this is currently not used. +- **parameters**: The parameters section defines the name, data type and + optional default value for each parameter that will be substituted into + the query before being passed to the data provider. Each parameter + must have a unique name (for each query, not globally). All parameters + specified in the query text must have an entry here or in the + **defaults** section. + Each parameter entry has the following sub-keys: + + - **description**: A description of what the parameter is (used for generating + documentation strings). + - **type**: The data type of the parameter. Valid types include: "str", "int", + "float", "list" and "datetime". The list and datetime types cause additional + formatting to be applied (such as converting from a date string) + - **default**: (optional) the default value for that parameter. Any parameter + that does not have a default value (here or in the file defaults section) + must be supplied at query time. + +Some common parameters used in the queries are: + +- **table**: making this a substitutable parameter allows you to use the same + query with different data sets. More commonly, you can add additional + filtering statements here, for example: + +.. code:: yaml + + table: + description: The table name + type: str + default: SecurityEvent | where EventID == 4624 + +- **add_query_items**: This is a useful way of extending queries by adding + ad hoc statements to the end of the query (e.g. additional filtering order + summarization). + +Using known parameter names +--------------------------- + +Try to use standard names for common entities and other parameter values. +This makes things easier for users of the queries and, in some cases, +enables functionality such as automatic insertion of times. + +Always use these names for common parameters + +================= ================================= ============= =============== +Query Parameter Description type default +================= ================================= ============= =============== +start The start datetime for the query datetime N/A +end The end datetime for the query datetime N/A +table The name of the main table (opt) str the table name +add_query_items Placeholder for additional query str "" +================= ================================= ============= =============== + +Entity names +For entities such as IP address, host name, account name, process, domain, etc., +always use one of the standard names - these are used by pivot functions to +map queries to the correct entity. + +For the current set of names see the following section in the Pivot Functions +documentation - :ref:`data_analysis/PivotFunctions:How are queries assigned to specific entities?` + +How parameter substitution works +-------------------------------- + +For simple text queries, such as KQL or SQL, parameters are substituted +into the query using string replace. Some parameter types may +have additional formatting applied (e.g. lists and datetimes) to +match the expected format of these types. + +You define a replacement parameter token by surrounding it with +single braces ("{" and "}"). + +.. note:: if you need to include a literal brace in the query + use a double brace - "{{" and "}}" + +For query providers that use JSON queries, the substitution process +is more complex but follows a similar pattern. + +Using yaml aliases and macros in your queries +--------------------------------------------- + +You can use standard yaml aliasing to define substitutable strings in your +query definitions. E.g. you might have a parameter default that is a long +string expression. Define an alias in the ``aliases`` key of the file +metadata section. An alias is defined by prefixing the name with "&". +The alias is referenced (and inserted) by using the alias name prefixed +with "*" + +.. code:: yaml + + metadata: + ... + aliases: + - &azure_network_project '| project TenantId, TimeGenerated, + FlowStartTime = FlowStartTime_t, + FlowEndTime = FlowEndTime_t, + FlowIntervalEndTime = FlowIntervalEndTime_t, + FlowType = FlowType_s, + ResourceGroup = split(VM_s, "/")[0], + VMName = split(VM_s, "/")[1], + VMIPAddress = VMIP_s' + ... + sources: + list_azure_network_flows_by_host: + description: Retrieves Azure network analytics flow events. + ... + parameters: + ... + query_project: + description: Column project statement + type: str + default: *azure_network_project + + +You can also use *macros*, which work like parameters but are substituted +into the query before any parameter substitution is carried out. This +allows you to, for example, use a single base query but with different +filter and summarization clauses defined as macros. The macro text is +substituted into the main query. + +Macros are added to the ``query_macros`` subkey of a query. They have +two subkeys: description and value. value defines the text to be inserted. +The key name is the name of the macro. + +In the query, you denote the substitution point by surrounding the macro name +with "$<" and ">$". This is show in the example below. + +.. code:: yaml + + - query: ' + {table} + | where SubType_s == "FlowLog" + | where FlowStartTime_t >= datetime({start}) + | where FlowEndTime_t <= datetime({end}) + $$ + | where (AllowedOutFlows_d > 0 or AllowedInFlows_d > 0) + {query_project} + | extend AllExtIPs = iif(isempty(PublicIPs), pack_array(ExtIP), + iif(isempty(ExtIP), PublicIPs, array_concat(PublicIPs, pack_array(ExtIP))) + ) + | project-away ExtIP + | mvexpand AllExtIPs + {add_query_items}' + +Macros are particularly useful when combined with yaml aliases. You can, for +example, define a base query (using a yaml alias) with a macro reference in the +query body. Then in each query definition you can have different macro values +for the macro to be substituted. For example: + +.. code:: yaml + + metadata: + ... + aliases: + - &azure_network_base_query ' + {table} + | where SubType_s == "FlowLog" + | where FlowStartTime_t >= datetime({start}) + | where FlowEndTime_t <= datetime({end}) + $$ + | where (AllowedOutFlows_d > 0 or AllowedInFlows_d > 0) + {query_project} + | extend AllExtIPs = iif(isempty(PublicIPs), pack_array(ExtIP), + iif(isempty(ExtIP), PublicIPs, array_concat(PublicIPs, pack_array(ExtIP))) + ) + | project-away ExtIP + | mvexpand AllExtIPs + {add_query_items}' + ... + sources: + list_azure_network_flows_by_ip: + description: Retrieves Azure network analytics flow events. + args: + query: *azure_network_base_query + parameters: + query_project: + ... + end: + description: Query end time + type: datetime + query_macros: + query_condition: + description: Query-specific where clause + value: '| where (VMIP_s in ({ip_address_list}) + or SrcIP_s in ({ip_address_list}) + or DestIP_s in ({ip_address_list}) + )' + +This allows you define a series of related queries that have the +same basic logic but have different filter clauses. This is extremely useful +where the query is complex and allows you to keep a single copy. + +.. note:: Using aliases and macros complicates the logic for anyone + trying to read the query file, so use this sparingly. + + +Guidelines for creating and debugging queries +--------------------------------------------- + +It is often helpful to start with a working version of a query without +using any parameters. Just paste in a query that you know is working. Once +you have verified that this works and returns data as expected you can +start to parameterize it. + +As you add parameters you can expect to find escaping and quoting +issues with the parameter values. To see what the parameterized version +of the query (without submitting it to the data provider) run the query +with the first parameter "print". This will return the parameterized version +of the query as a string: + +.. code:: python3 + + qry_prov.SecurityEvents.my_new_query( + "print", + start=start_dt, + end=end_dt, + account="ian", + ) + + +There are also a number of tools within the package to assist in +validating new query definition files once created. + +:: + + data_query_reader.find_yaml_files + + Return iterable of yaml files found in `source_path`. + + Parameters + ---------- + source_path : str + The source path to search in. + recursive : bool, optional + Whether to recurse through subfolders. + By default False + + Returns + ------- + Iterable[str] + File paths of yaml files found. + + data_query_reader.validate_query_defs + + Validate content of query definition. + + Parameters + ---------- + query_def_dict : dict + Dictionary of query definition yaml file contents. + + Returns + ------- + bool + True if validation succeeds. + + Raises + ------ + ValueError + The validation failure reason is returned in the + exception message (arg[0]) + +validate_query_defs() does not perform comprehensive checks on the file +but does check key elements required in the file are present. + +.. code:: python3 + + for file in QueryReader.find_yaml_files(source_path="C:\\queries"): + with open(file) as f_handle: + yaml_file = yaml.safe_load(f_handle) + if QueryReader.validate_query_defs(query_def_dict = yaml_file) == True: + print(f' {file} is a valid query definition') + else: + print(f'There is an error with {file}') + + +.. parsed-literal:: + + C:\queries\example.yaml is a valid query definition + + +Adding a new set of queries and running them +-------------------------------------------- + +Once you are happy with +a query definition file then you import it with: + + *query_provider*.import_query_file(query_file= *path_to_query_file*) + +Where *query_provider* is your QueryProvider instance. + +This will load the query file into the Query Provider's Query Store from +where it can be called. + +.. code:: python3 + + qry_prov.import_query_file(query_file='C:\\queries\\example.yaml') + +You can also put the file into a folder and load the queries +when you create your query provider: + +.. code::python3 + + qry_prov = mp.QueryProvider("Splunk", query_paths=["~/home/mp_queries"]) + +.. note:: ``query_paths``` is a list of strings, so make sure that you + surround a single path with Python list brackets. + +Once imported the queries in the files appear in the Query Provider's +Query Store alongside the others and can be called in the same manner as +pre-defined queries. + +If you have created a large number of query definition files and you +want to have the automatically imported into a Query Provider's query +store at initialization you can specify a directory containing these +queries in the msticpyconfig.yaml file under QueryDefinitions: Custom: + +For example, if you have two folders with queries in each that +you want to load, add entries for each to your msticpyconfig.yaml file. + +Example: + +.. code:: yaml + + QueryDefinitions: + Custom: + - /home/ian/mp_queries + - /home/ian/mp_queries_common + + +Having the ``Custom`` field populated will mean the Query Provider will +automatically enumerate all the YAML files in the directory provided and +automatically import he relevant queries into the query store at +initialization alongside the default queries. Custom queries with the +same name as default queries will overwrite default queries. + +.. code:: python3 + + queries = qry_prov.list_queries() + for query in queries: + print(query) + + +.. parsed-literal:: + + LinuxSyslog.all_syslog + LinuxSyslog.cron_activity + LinuxSyslog.squid_activity + LinuxSyslog.sudo_activity + LinuxSyslog.syslog_example + LinuxSyslog.user_group_activity + LinuxSyslog.user_logon + SecurityAlert.get_alert + SecurityAlert.list_alerts + SecurityAlert.list_alerts_counts + SecurityAlert.list_alerts_for_ip + SecurityAlert.list_related_alerts + WindowsSecurity.get_host_logon + WindowsSecurity.get_parent_process + WindowsSecurity.get_process_tree + WindowsSecurity.list_host_logon_failures + WindowsSecurity.list_host_logons + WindowsSecurity.list_host_processes + WindowsSecurity.list_hosts_matching_commandline + WindowsSecurity.list_matching_processes + WindowsSecurity.list_processes_in_session + + +.. code:: python3 + + qry_prov.LinuxSyslog.syslog_example('?') + + +.. parsed-literal:: + + Query: syslog_example + Data source: LogAnalytics + Example query + + Parameters + ---------- + add_query_items: str (optional) + Additional query clauses + end: datetime + Query end time + host_name: str + Hostname to query for + query_project: str (optional) + Column project statement + (default value is: | project TenantId, Computer, Facility, TimeGener...) + start: datetime + Query start time + subscription_filter: str (optional) + Optional subscription/tenant filter expression + (default value is: true) + table: str (optional) + Table name + (default value is: Syslog) + Query: + {table} | where {subscription_filter} + | where TimeGenerated >= datetime({start}) + | where TimeGenerated <= datetime({end}) + | where Computer == "{host_name}" | take 5 + + +.. code:: python3 + + qry_prov.LinuxSyslog.syslog_example( + start='2019-07-21 23:43:18.274492', + end='2019-07-27 23:43:18.274492', + host_name='UbuntuDevEnv' + ) + + +.. raw:: html + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TenantIdSourceSystemTimeGeneratedComputerEventTimeFacilityHostNameSeverityLevelSyslogMessageProcessIDHostIPProcessNameMGType_ResourceId
0b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.213UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvnoticeomsagent : TTY=unknown PWD=/opt/microsoft/om...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
1b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.313UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvinfopam_unix(sudo:session): session opened for use...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
2b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:37.917UbuntuDevEnv2019-07-25 15:15:37authprivUbuntuDevEnvinfopam_unix(sudo:session): session closed for use...NaN10.0.1.4sudo00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
3b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:15:50.793UbuntuDevEnv2019-07-25 15:15:50authprivUbuntuDevEnvinfopam_unix(cron:session): session closed for use...29486.010.0.1.4CRON00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
4b1315f05-4a7a-45b4-811f-73e715f7c122Linux2019-07-25 15:16:01.800UbuntuDevEnv2019-07-25 15:16:01authprivUbuntuDevEnvinfopam_unix(cron:session): session opened for use...29844.010.0.1.4CRON00000000-0000-0000-0000-000000000002Syslog/subscriptions/3b701f84-d04b-4479-89b1-fa8827e...
+
+ +| + +If you are having difficulties with a defined query and it is not +producing the expected results it can be useful to see the raw query +exactly as it is passed to the Data Environment. If you call a query +with "print" and the parameters required by that query it will construct +and print out the query string to be run. + +.. code:: python3 + + qry_prov.LinuxSyslog.syslog_example( + 'print', + start='2019-07-21 23:43:18.274492', + end='2019-07-27 23:43:18.274492', + host_name='UbuntuDevEnv' + ) + + + + +.. parsed-literal:: + + 'Syslog + | where true + | where TimeGenerated >= datetime(2019-07-21 23:43:18.274492) + | where TimeGenerated <= datetime(2019-07-27 23:43:18.274492) + | where Computer == "UbuntuDevEnv" + | take 5' + diff --git a/docs/source/data_acquisition/WritingADataProvider.rst b/docs/source/extending/WritingDataProviders.rst similarity index 98% rename from docs/source/data_acquisition/WritingADataProvider.rst rename to docs/source/extending/WritingDataProviders.rst index f4ee6393e..a3b77f645 100644 --- a/docs/source/data_acquisition/WritingADataProvider.rst +++ b/docs/source/extending/WritingDataProviders.rst @@ -1,6 +1,9 @@ Writing and Contributing a Data Provider ======================================== +See :doc:`../data_acquisition/DataProviders` for more details on use +of data providers. + A data provider lets you query data from a notebook in a standardized way. Before reading further you should familiarize yourself with how the data providers work from the :doc:`Querying and Importing Data <../DataAcquisition>` @@ -32,8 +35,9 @@ To implement a data provider you need to do the following: 2. Customize the driver (optional) 3. Register the driver 4. Add queries -5. Create documentation -6. Create unit tests +5. Add settings definition +6. Create documentation +7. Create unit tests 1. Write the driver class ------------------------- @@ -393,9 +397,9 @@ A data provider should have documentation describing its configuration and use. This should be in restructured text for generating document pages in Sphinx. -See the examples :doc:`./SplunkProvider` and :doc:`./DataProv-Sumologic` +See the examples :doc:`../data_acquisition/SplunkProvider` and :doc:`../data_acquisition/DataProv-Sumologic` -6. Create driver unit tests +7. Create driver unit tests --------------------------- Please add a unit test using mocks to simulate the service diff --git a/docs/source/extending/WritingTIAndContextProviders.rst b/docs/source/extending/WritingTIAndContextProviders.rst new file mode 100644 index 000000000..a2575ca61 --- /dev/null +++ b/docs/source/extending/WritingTIAndContextProviders.rst @@ -0,0 +1,460 @@ +Writing Threat Intelligence and Context Providers +================================================= + +See :doc:`../data_acquisition/TIProviders` for more details on use +of Threat Intelligence providers. + +You can write your own provider by extending one +of the MSTICPy base classes: + +- :py:class:`TIProvider ` +- :py:class:`HttpTIProvider ` +- :py:class:`ContextProvider ` + +The first two classes are for creating Threat Intelligence providers, +the third is for a Context provider. Most of the content of this +document is applicable to both so you should read the entire +thing. A short section on the differences for Context providers +follows this first section. + + +Threat Intelligence Providers +----------------------------- + +For most TI services you can use +:py:class:`HttpTIProvider `. + +You can derive a class from +:py:class:`TIProvider ` but +here, you have to implement all of the logic and plumbing to query +your data source by implementing a lookup_ioc and lookup_ioc_asycnc +methods. If you need to do this look at the +`OpenPageRank provider `__ +for an example. + +The HttpTIProvider base class +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This has built-in support for making and processing the requests +to the service API. You typically need to define: + +- The base API URL +- A mapping of IoC types to URL paths or query strings +- What authentication credentials the service needs and + how these will be passed in the requests +- A function that processes the response and extracts + severity and summary details. + +Here is a toy example: + +.. code:: python3 + + from typing import Any, Dict, Tuple + + from msticpy.context.http_provider import APILookupParams + from msticpy.context.tiproviders.ti_http_provider import HttpTIProvider, ResultSeverity + + class TIProviderHttpTest(HttpTIProvider): + """Custom IT provider TI HTTP.""" + + PROVIDER_NAME = "MyTIProvider" + _BASE_URL = "https://api.service.com" + _QUERIES = _QUERIES = { + "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"), + "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"), + "file_hash": APILookupParams(path="/api/v1/indicators/file/{observable}/general"), + "url": APILookupParams(path="/api/v1/indicators/url/{observable}/general"), + } + # aliases + _QUERIES["md5_hash"] = _QUERIES["file_hash"] + _QUERIES["sha1_hash"] = _QUERIES["file_hash"] + _QUERIES["sha256_hash"] = _QUERIES["file_hash"] + + _REQUIRED_PARAMS = ["AuthKey"] + + def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + if response["severity"] < 5: + severity = ResultSeverity.high + else: + severity = ResultSeverity.warning + details = { + "Source": response.get("source_domain"), + "RelatedIPs": response.get("source_ip_addrs") + } + return True, severity, details + + +We can see that the new provider class is derived from +:py:class:`HttpTIProvider `. + +This expects several items defined as class attributes: + +- _BASE_URL - the root URL for API calls +- _QUERIES - definitions for each IoC type to create the appropriate + http request. +- _REQUIRED_PARAMS - the mandatory items in the request parameters + (usually the Api Key) + +You also need to implement an instance method +:py:meth:`parse_results ` +(see below) + +The QUERIES Dictionary and APILookupParams +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``_QUERIES`` dictionary is the most complex part and requires +further explanation. + +Each entry has a key corresponding to an IoC type (e.g. "ipv4", "url", etc.). +The value of each item is an instance of +:py:class:`APILookupParams ` +which specifies the HTTP request configuration for each IoC type. + +You can re-use the same entry to create aliased items that map +multiple IoC types on the same request. +You can do this by adding existing values to the dictionary +with new keys, as shown below: + +.. code:: python3 + + # aliases + _QUERIES["md5_hash"] = _QUERIES["file_hash"] + _QUERIES["sha1_hash"] = _QUERIES["file_hash"] + _QUERIES["sha256_hash"] = _QUERIES["file_hash"] + +In this example, the service provider API accepts any type of hash using +the same request parameters. Creating multiple mappings like this +lets the user specify any of these types to perform a lookup. Also, +in cases where the user does not explicitly specify an ``ioc_type`` in +the call the ``lookup_ioc``, the TILookup class will try to infer the +type using regular expression matching and will pass the inferred type +to your provider class. By creating these aliases we can map all variants +of an IoC type (in this case a hash) to this one request definition. + +You can also create compound keys. This is useful where a given IoC +type has sub-queries for different classes of data related to the IoC. + +Here is an example from our Alienvault OTX provider, which has a +general "ipv4" path but also several types of more specialized +queries - passive DNS and geo-location. + +.. code:: python3 + + _QUERIES = { + "ipv4": _OTXParams(path="/api/v1/indicators/IPv4/{observable}/general"), + "ipv6": _OTXParams(path="/api/v1/indicators/IPv6/{observable}/general"), + "ipv4-passivedns": _OTXParams( + path="/api/v1/indicators/IPv4/{observable}/passive_dns" + ), + "ipv6-passivedns": _OTXParams( + path="/api/v1/indicators/IPv6/{observable}/passive_dns" + ), + "ipv4-geo": _OTXParams(path="/api/v1/indicators/IPv4/{observable}/geo"), + "ipv6-geo": _OTXParams(path="/api/v1/indicators/IPv6/{observable}/geo"), + ... + +This allows users to request the specific dataset for the IoC using +the ``query_type`` parameter: + +.. code:: python3 + + tilookup.lookup_ioc("123.4.56.78", query_type="passivedns") + + +Having decided on the keys needed, you can create the APILookupParams +instances to tell the TILookup module how to form the HTTP requests. + +The :py:class:`APILookupParams ` class +has the following attributes: + +.. code:: python3 + + class APILookupParams: + """HTTP Lookup Params definition.""" + + path: str = "" + verb: str = "GET" + full_url: bool = False + headers: Dict[str, str] = Factory(dict) + params: Dict[str, Union[str, int, float]] = Factory(dict) + data: Dict[str, str] = Factory(dict) + auth_type: str = "" + auth_str: List[str] = Factory(list) + sub_type: str = "" + +The value of each item in the queries dictionary should be an +instance of an ``APILookupParams`` class or one derived from it. + +.. note:: APILookupParams is an attrs class, if you create a subclass + from it you should also make that an attrs class. + +Several of the values of this class can have substitutable parameters +where runtime values (e.g. the observable value) +are inserted before making the request. + +**path** + +The sub-path for the query for this IoC type. This will be +appended to the ``_BASE_URL`` value. + +.. code:: python3 + + _QUERIES = _QUERIES = { + "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"), + +In this example you can see that "{observable}", the IoC value, is +a substitutable parameter. + +**verb** + +The provider framework currently only supports "GET" + +**full_url** +If True, the ``_BASE_URL`` value is ignored and the ``path`` value +is treated as a full URL and used in the request as-is. + +**headers** + +A dictionary of request headers. +This also supports parameter substitution of any value surrounded with +braces. + +Example: + +.. code:: python3 + + _QUERIES = _QUERIES = { + "ipv4": APILookupParams( + path="/api/v1/indicators/IPv4/{observable}/general" + headers = {"X-OTX-API-KEY": "{AuthKey}"} + ), + ... + +**params** +A dictionary of request parameters. This also supports +parameter substitution of values: + +.. code:: python3 + + _QUERIES = _QUERIES = { + "ipv4": APILookupParams( + path="/api/v1/indicators/IPv4" + params={"iocValue": "{observable}"}, + ), + ... + +**data** +This is currently not supported but we will implement if and +when required. This is a dictionary that will be supplied as request data. supports +parameter substitution for values. + +**auth_type** +Currently only "HTTPBasic" is supported. + +**auth_str** +This is an list of values to supply as the request ``auth`` property. +Supports substitution. + +.. code:: python3 + + _QUERIES = _QUERIES = { + "ipv4": APILookupParams( + path="/api/v1/indicators/IPv4" + auth_str = ["{ApiID}", "{AuthKey}"], + ), + ... + +**sub_type** +Not currently used. + +The parse_results method +~~~~~~~~~~~~~~~~~~~~~~~~ + +See :py:meth:`parse_results ` +for the method header. + +This method is responsible for taking the JSON response (as +a Python dictionary) and extracting and returning severity +and relevant details. + +The implementation in the example at the start of this document +(and below) shows a simple process, but it can be as complex as needed. + +.. note:: we would recommend creating child methods to handle + different response types if you need to do complex + processing. + +.. code:: python3 + + def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + if self._failed_response(response): + return False, ResultSeverity.information, "Not found." + + if response["severity"] < 5: + severity = ResultSeverity.high + else: + severity = ResultSeverity.warning + details = { + "Source": response.get("source_domain"), + "RelatedIPs": response.get("source_ip_addrs") + } + return True, severity, details + +The function returns a Tuple of: + +- parsing success (bool) - return False if the request produced no + useful data +- severity - using the :py:class:`ResultSeverity ` + enumeration: high, warning, information, unknown +- details - a dictionary of information from the response that + you want to highlight. The full raw response is always returned + to the user. + +In ``parse_results`` your responsibility is to check the response +data for an indication of severity - i.e. the level of threat posed +by the observable. + +The ``details`` dictionary can contain arbitrary data extracted +from the response. + +Provider configuration +~~~~~~~~~~~~~~~~~~~~~~ + +You can supply parameters (such as AuthKey and Api/User ID) to your +provider by creating an entry in ``msticpyconfig.yaml``. + +.. code:: yaml + + TIProviders: + MyProvider: + Args: + AuthKey: + EnvironmentVar: "MY_PROV_AUTH" + Primary: True + Provider: "MyProvider" + +Assuming that your provider is implemented in ``MyProvider``, +TILookup will read and pass the value for "AuthKey" to +the provider to include in the requests to the service API. +(In the above example the value of "AuthKey" will be read +from the environment variable "MY_PROV_AUTH".) + +For more details on MSTICPy configuration see :doc:`../getting_started/msticpyconfig` + +Using the TI Provider +~~~~~~~~~~~~~~~~~~~~~ + +You can use the TI provider in one of two ways: + +- You can use it as a MSTICPy plugin - see :doc:`PluginFramework` +- You can submit it as a pull request to the `MSTICPy repo `__ + - see :doc:`../Development` + +If you are going to do the second of these, please read the following +section. + +Integrating the TI Provider into MSTICPy +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Make sure that you follow the coding guidelines given in +:doc:`../Development`. + +To add your provider to the built-in providers, there are +some additional steps: + +- add your module to ``msticpy/context/tiproviders`` +- add an entry to msticpy.context.tiproviders.__init__ +- configure the msticpy settings UI to allow you to manage + the provider settings from MpConfigEdit. + + +In the file ``msticpy/context/tiproviders/__init__.py``, +add your provider to the ``TI_PROVIDERS`` dictionary. + +.. code-block:: Python3 + :emphasize-lines: 5 + + TI_PROVIDERS: Dict[str, Tuple[str, str]] = { + "OTX": ("alienvault_otx", "OTX"), + "AzSTI": ("azure_sent_byoti", "AzSTI"), + ... + "MyProvider": ("my_provider", "MyProviderClass"), + "" + } + +The highlighted example has the following syntax: + +*ProviderFriendlyName*: (*module_name*, *provider_class*) + +- ProviderFriendlyName: is the name used to refer to the provider + in ``msticpyconfig.yaml`` and when naming specific providers + in the call to ``lookup_ioc(s)`` +- module_name: is the name of the module holding your class. +- provider_class: is the name of the class implementing the provider. + +To enable settings in the MSTICPy settings editor, edit the file +``msticpy/resources/mpconfig_defaults.yaml``. + +Add an entry to the TIProviders section of this file. + +.. code:: yaml + + TIProviders: + # If a provider has Primary: True it will be run by default on IoC lookups + # Secondary providers can be run optionally + OTX: + Args: + AuthKey: *cred_key + Primary: bool(default=True) + Provider: "OTX" + MyProvider: + Args: + AuthKey: *cred_key + ApiID: *cred_key + Primary: bool(default=True) + Provider: "MyProvider" + +Add an item in the "Args:" subsection for each value that you +want to be editable in the UI. +The special value "\*cred_key" tells the settings editor that this +value can be treated as a string, an environment variable or a +Key Vault secret. + +If you have other configurable values you can add strings, booleans, etc. +- use the **Splunk** example in the ``DataProviders`` section to +help you. + +Context Providers +----------------- + +Context providers follow the same model as TI Providers. + +The key differences are as follows. + +parse_results method +~~~~~~~~~~~~~~~~~~~~ + +This method is used only to extract details - unlike the TI providers, there +is no severity scoring. It should be a tuple of (*Success*, *Details*) - where +*Success* is a boolean and *Details* is (usually) a Python dict. +*Details* can be any Python object but will ultimately be returned to the +user as a pandas DataFrame column, so should be something that +is easily extracted/viewed like a Python list or dict. + +Registering your Context Provider +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The ``msticpy.context.contextproviders.__init__.py`` module uses +the same syntax described in `Integrating the TI Provider into MSTICPy`_ +but adding an entry to the ``CONTEXT_PROVIDERS`` dictionary. + +.. code-block:: python3 + :emphasize-lines: 3 + + CONTEXT_PROVIDERS: Dict[str, Tuple[str, str]] = { + "ServiceNow": ("servicenow", "ServiceNow"), + "FriendlyName": ("module_name", "class_name"), + } + diff --git a/docs/source/index.rst b/docs/source/index.rst index e29667387..7e52cb90b 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -70,6 +70,8 @@ Contents DataAnalysis Visualization msticpyAPI + ExtendingMsticpy + Development notebooksamples blog_articles Releases diff --git a/msticpy/__init__.py b/msticpy/__init__.py index 03affeb6c..80ca3dae5 100644 --- a/msticpy/__init__.py +++ b/msticpy/__init__.py @@ -116,7 +116,7 @@ """ import importlib import os -from typing import Any +from typing import Any, Iterable, Union from . import nbwidgets @@ -188,3 +188,26 @@ def __getattr__(attrib: str) -> Any: def __dir__(): """Return attribute list.""" return sorted(set(_STATIC_ATTRIBS + list(_DEFAULT_IMPORTS))) + + +def load_plugins(plugin_paths: Union[str, Iterable[str]]): + """ + Load plugins from specified paths or configuration. + + Parameters + ---------- + plugin_paths : Union[str, Iterable[str]] + A path or collection of paths from which to + load plugins. If not supplied, msticpyconfig is checked for + a PluginFolders key and list of paths are read from there. + + Notes + ----- + No attempt to load plugins is made if both parameter and + configuration are empty. + + """ + # pylint: disable=import-outside-toplevel + from .init.mp_plugins import read_plugins + + read_plugins(plugin_paths) diff --git a/msticpy/context/contextlookup.py b/msticpy/context/contextlookup.py index b38f038e3..44fcbd7f6 100644 --- a/msticpy/context/contextlookup.py +++ b/msticpy/context/contextlookup.py @@ -12,7 +12,7 @@ requests per minute for the account type that you have. """ -from typing import Iterable, List, Mapping, Optional, Union +from typing import Dict, Iterable, List, Mapping, Optional, Union import pandas as pd @@ -22,7 +22,7 @@ # used in dynamic instantiation of providers from .contextproviders import CONTEXT_PROVIDERS from .lookup import Lookup -from .provider_base import _make_sync +from .provider_base import Provider, _make_sync __version__ = VERSION __author__ = "Ian Hellen" @@ -45,6 +45,7 @@ class ContextLookup(Lookup): PACKAGE = "contextproviders" PROVIDERS = CONTEXT_PROVIDERS + CUSTOM_PROVIDERS: Dict[str, Provider] = {} # pylint: disable=too-many-arguments def lookup_observable( diff --git a/msticpy/context/contextproviders/servicenow.py b/msticpy/context/contextproviders/servicenow.py index 50219698f..5b8636206 100644 --- a/msticpy/context/contextproviders/servicenow.py +++ b/msticpy/context/contextproviders/servicenow.py @@ -39,7 +39,7 @@ class _ServiceNowParams(APILookupParams): # override LookupParams to set common defaults def __attrs_post_init__(self): - self.auth_str = ["{API_ID}", "{API_KEY}"] + self.auth_str = ["{ApiID}", "{AuthKey}"] self.auth_type = "HTTPBasic" @@ -47,7 +47,7 @@ def __attrs_post_init__(self): class ServiceNow(HttpContextProvider): """ServiceNow Lookup.""" - _BASE_URL = "https://{INSTANCE}.service-now.com/api/now/table" + _BASE_URL = "https://{Instance}.service-now.com/api/now/table" _SERVICE_NOW_PARAMS = { "sysparm_display_value": True, @@ -89,7 +89,7 @@ class ServiceNow(HttpContextProvider): ), } - _REQUIRED_PARAMS = ["API_ID", "API_KEY", "INSTANCE"] + _REQUIRED_PARAMS = ["ApiID", "AuthKey", "Instance"] def parse_results(self, response: Dict[str, Any]) -> Tuple[bool, Any]: """ diff --git a/msticpy/context/http_provider.py b/msticpy/context/http_provider.py index b3b5f2958..3ede73114 100644 --- a/msticpy/context/http_provider.py +++ b/msticpy/context/http_provider.py @@ -74,7 +74,7 @@ class HttpProvider(Provider): # Community API "ipv4": APILookupParams( path="/v3/community/{observable}", - headers={"key": "{API_KEY}"}, + headers={"key": "{AuthKey}"}, ), # Enterprise API Quick Lookup "ipv4-quick": APILookupParams( @@ -91,7 +91,7 @@ class HttpProvider(Provider): .. code:: python - _REQUIRED_PARAMS = ["API_KEY"] + _REQUIRED_PARAMS = ["AuthKey"] In __init__ @@ -131,13 +131,13 @@ def __init__(self, **kwargs): self._request_params = {} if "ApiID" in kwargs: api_id = kwargs.pop("ApiID") - self._request_params["API_ID"] = api_id.strip() if api_id else None + self._request_params["ApiID"] = api_id.strip() if api_id else None if "AuthKey" in kwargs: auth_key = kwargs.pop("AuthKey") - self._request_params["API_KEY"] = auth_key.strip() if auth_key else None + self._request_params["AuthKey"] = auth_key.strip() if auth_key else None if "Instance" in kwargs: auth_key = kwargs.pop("Instance") - self._request_params["INSTANCE"] = auth_key.strip() if auth_key else None + self._request_params["Instance"] = auth_key.strip() if auth_key else None missing_params = [ param diff --git a/msticpy/context/lookup.py b/msticpy/context/lookup.py index f7dced4cd..df2a83083 100644 --- a/msticpy/context/lookup.py +++ b/msticpy/context/lookup.py @@ -16,7 +16,7 @@ import importlib import warnings from collections import ChainMap -from typing import Any, Dict, Iterable, List, Mapping, Optional, Tuple, Union +from typing import Any, Dict, Iterable, List, Mapping, Optional, Tuple, Union, cast import nest_asyncio import pandas as pd @@ -70,6 +70,7 @@ class Lookup: _HELP_URI = "https://msticpy.readthedocs.io/en/latest/DataEnrichment.html" PROVIDERS: Dict[str, Tuple[str, str]] = {} + CUSTOM_PROVIDERS: Dict[str, Provider] PACKAGE: str = "" @@ -607,7 +608,7 @@ async def _track_completion(prog_counter): @property def available_providers(self) -> List[str]: """ - Return a list of builtin providers. + Return a list of builtin and plugin providers. Returns ------- @@ -615,7 +616,7 @@ def available_providers(self) -> List[str]: List of TI Provider classes. """ - return list(self.PROVIDERS) + return list(self.PROVIDERS) + list(self.CUSTOM_PROVIDERS) @classmethod def list_available_providers( @@ -648,9 +649,7 @@ def list_available_providers( if show_query_types and provider_class: provider_class.usage() - if as_list: - return providers - return None + return providers if as_list else None @classmethod def import_provider(cls, provider: str) -> Provider: @@ -658,10 +657,12 @@ def import_provider(cls, provider: str) -> Provider: mod_name, cls_name = cls.PROVIDERS.get(provider, (None, None)) if not (mod_name and cls_name): + if hasattr(cls, "CUSTOM_PROVIDERS") and provider in cls.CUSTOM_PROVIDERS: + return cast(Provider, cls.CUSTOM_PROVIDERS[provider]) raise LookupError( - f"No driver available for environment {provider}.", + f"No provider named '{provider}'.", "Possible values are:", - ", ".join(list(cls.PROVIDERS)), + ", ".join(list(cls.PROVIDERS) + list(cls.CUSTOM_PROVIDERS)), ) imp_module = importlib.import_module( @@ -673,21 +674,21 @@ def _load_providers(self, **kwargs): """Load provider classes based on config.""" prov_type = kwargs.get("providers", "Providers") prov_settings = get_provider_settings(prov_type) - - for provider_entry, settings in prov_settings.items(): - # Allow overriding provider name to use another class - provider_name = settings.provider or provider_entry + provider_class: Provider + for provider_name, settings in prov_settings.items(): if ( self._providers_to_load is not None and provider_name not in self._providers_to_load - ) or provider_name == "--no-load--": + ) or settings.provider == "--no-load--": continue try: - provider_class: Provider = self.import_provider(provider_name) + provider_name = settings.provider or provider_name + provider_class = self.import_provider(provider_name) except LookupError: warnings.warn( f"Could not find provider class for {provider_name} " - f"in config section {provider_entry}" + f"in config section '{provider_name}'. " + f"Provider class name in config is '{settings.provider}'" ) continue diff --git a/msticpy/context/tilookup.py b/msticpy/context/tilookup.py index 37c3c79d1..d2034cfa1 100644 --- a/msticpy/context/tilookup.py +++ b/msticpy/context/tilookup.py @@ -13,16 +13,16 @@ """ -from typing import Iterable, List, Mapping, Optional, Union +from typing import Dict, Iterable, List, Mapping, Optional, Union import pandas as pd from .._version import VERSION from ..common.utility import export from .lookup import Lookup -from .provider_base import _make_sync # used in dynamic instantiation of providers +from .provider_base import Provider, _make_sync from .tiproviders import TI_PROVIDERS __version__ = VERSION @@ -44,6 +44,7 @@ class TILookup(Lookup): PROVIDERS = TI_PROVIDERS PACKAGE = "tiproviders" + CUSTOM_PROVIDERS: Dict[str, Provider] = {} # pylint: disable=too-many-arguments def lookup_ioc( diff --git a/msticpy/context/tiproviders/alienvault_otx.py b/msticpy/context/tiproviders/alienvault_otx.py index 92926cac2..aca935f67 100644 --- a/msticpy/context/tiproviders/alienvault_otx.py +++ b/msticpy/context/tiproviders/alienvault_otx.py @@ -32,7 +32,7 @@ class _OTXParams(APILookupParams): # override APILookupParams to set common defaults def __attrs_post_init__(self): # pylint: disable= - self.headers = {"X-OTX-API-KEY": "{API_KEY}"} + self.headers = {"X-OTX-API-KEY": "{AuthKey}"} @export @@ -67,7 +67,7 @@ class OTX(HttpTIProvider): _QUERIES["sha1_hash"] = _QUERIES["file_hash"] _QUERIES["sha256_hash"] = _QUERIES["file_hash"] - _REQUIRED_PARAMS = ["API_KEY"] + _REQUIRED_PARAMS = ["AuthKey"] def __init__(self, **kwargs): """Set OTX specific settings.""" diff --git a/msticpy/context/tiproviders/greynoise.py b/msticpy/context/tiproviders/greynoise.py index ad6a3ede2..6d89c7c29 100644 --- a/msticpy/context/tiproviders/greynoise.py +++ b/msticpy/context/tiproviders/greynoise.py @@ -32,17 +32,17 @@ class GreyNoise(HttpTIProvider): # Community API "ipv4": APILookupParams( path="/v3/community/{observable}", - headers={"key": "{API_KEY}"}, + headers={"key": "{AuthKey}"}, ), # Enterprise API Quick Lookup "ipv4-quick": APILookupParams( path="/v2/noise/quick/{observable}", - headers={"key": "{API_KEY}"}, + headers={"key": "{AuthKey}"}, ), # Enterprise API Full Lookup "ipv4-full": APILookupParams( path="/v2/noise/context/{observable}", - headers={"key": "{API_KEY}"}, + headers={"key": "{AuthKey}"}, ), } diff --git a/msticpy/context/tiproviders/ibm_xforce.py b/msticpy/context/tiproviders/ibm_xforce.py index 4441d3787..943a3c45a 100644 --- a/msticpy/context/tiproviders/ibm_xforce.py +++ b/msticpy/context/tiproviders/ibm_xforce.py @@ -31,7 +31,7 @@ class _XForceParams(APILookupParams): # override APILookupParams to set common defaults def __attrs_post_init__(self): - self.auth_str = ["{API_ID}", "{API_KEY}"] + self.auth_str = ["{ApiID}", "{AuthKey}"] self.auth_type = "HTTPBasic" @@ -68,7 +68,7 @@ class XForce(HttpTIProvider): _QUERIES["hostname-whois"] = _QUERIES["ipv4-whois"] _QUERIES["dns-whois"] = _QUERIES["ipv4-whois"] - _REQUIRED_PARAMS = ["API_ID", "API_KEY"] + _REQUIRED_PARAMS = ["ApiID", "AuthKey"] def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: """ diff --git a/msticpy/context/tiproviders/intsights.py b/msticpy/context/tiproviders/intsights.py index 77aa48941..b1c094e7c 100644 --- a/msticpy/context/tiproviders/intsights.py +++ b/msticpy/context/tiproviders/intsights.py @@ -34,7 +34,7 @@ class _IntSightsParams(APILookupParams): # override APILookupParams to set common defaults def __attrs_post_init__(self): - self.auth_str = ["{API_ID}", "{API_KEY}"] + self.auth_str = ["{ApiID}", "{AuthKey}"] self.auth_type = "HTTPBasic" @@ -87,7 +87,7 @@ class IntSights(HttpTIProvider): ), } - _REQUIRED_PARAMS = ["API_ID", "API_KEY"] + _REQUIRED_PARAMS = ["ApiID", "AuthKey"] def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: """ diff --git a/msticpy/context/tiproviders/open_page_rank.py b/msticpy/context/tiproviders/open_page_rank.py index 4bb950e88..d265ce665 100644 --- a/msticpy/context/tiproviders/open_page_rank.py +++ b/msticpy/context/tiproviders/open_page_rank.py @@ -39,11 +39,11 @@ class OPR(HttpTIProvider): "dns": APILookupParams( path="/api/v1.0/getPageRank", params={"domains[0]": "{observable}"}, - headers={"API-OPR": "{API_KEY}"}, + headers={"API-OPR": "{AuthKey}"}, ) } - _REQUIRED_PARAMS = ["API_KEY"] + _REQUIRED_PARAMS = ["AuthKey"] def __init__(self, **kwargs): """Initialize a new instance of the class.""" diff --git a/msticpy/context/tiproviders/virustotal.py b/msticpy/context/tiproviders/virustotal.py index 7fc28f70d..a9d6cc7b7 100644 --- a/msticpy/context/tiproviders/virustotal.py +++ b/msticpy/context/tiproviders/virustotal.py @@ -35,7 +35,7 @@ class VirusTotal(HttpTIProvider): _BASE_URL = "https://www.virustotal.com/" - _PARAMS = {"apikey": "{API_KEY}"} + _PARAMS = {"apikey": "{AuthKey}"} _QUERIES = { "ipv4": APILookupParams( path="vtapi/v2/ip-address/report", @@ -64,7 +64,7 @@ class VirusTotal(HttpTIProvider): _QUERIES["sha1_hash"] = _QUERIES["file_hash"] _QUERIES["sha256_hash"] = _QUERIES["file_hash"] - _REQUIRED_PARAMS = ["API_KEY"] + _REQUIRED_PARAMS = ["AuthKey"] _VT_DETECT_RESULTS = { "detected_urls": ("url", "scan_date"), diff --git a/msticpy/data/core/data_providers.py b/msticpy/data/core/data_providers.py index 661993540..cbd681821 100644 --- a/msticpy/data/core/data_providers.py +++ b/msticpy/data/core/data_providers.py @@ -8,7 +8,7 @@ from functools import partial from itertools import tee from pathlib import Path -from typing import Any, Dict, Iterable, List, Optional, Union +from typing import Any, Dict, Iterable, List, Optional, Tuple, Union import pandas as pd from tqdm.auto import tqdm @@ -17,7 +17,7 @@ from ...common.pkg_config import get_config from ...common.utility import export, valid_pyname from ...nbwidgets import QueryTime -from ..drivers import import_driver +from .. import drivers from ..drivers.driver_base import DriverBase, DriverProps from .param_extractor import extract_query_params from .query_container import QueryContainer @@ -94,16 +94,13 @@ def __init__( # noqa: MC0001 # pylint: enable=import-outside-toplevel setattr(self.__class__, "_add_pivots", add_data_queries_to_entities) - if isinstance(data_environment, str): - data_env = DataEnvironment.parse(data_environment) - if data_env != DataEnvironment.Unknown: - data_environment = data_env - else: - raise TypeError(f"Unknown data environment {data_environment}") + data_environment, self.environment_name = self._check_environment( + data_environment + ) self._driver_kwargs = kwargs.copy() if driver is None: - self.driver_class = import_driver(data_environment) + self.driver_class = drivers.import_driver(data_environment) if issubclass(self.driver_class, DriverBase): driver = self.driver_class(data_environment=data_environment, **kwargs) else: @@ -112,11 +109,10 @@ def __init__( # noqa: MC0001 ) else: self.driver_class = driver.__class__ - # allow the driver to override the data environment used - # for selecting queries - self.environment = ( + # allow the driver to override the data environment used for selecting queries + self.environment_name = ( driver.get_driver_property(DriverProps.EFFECTIVE_ENV) - or data_environment.name + or self.environment_name ) self._additional_connections: Dict[str, DriverBase] = {} @@ -133,11 +129,32 @@ def __init__( # noqa: MC0001 self._read_queries_from_paths(query_paths=query_paths) ) self.query_store = data_env_queries.get( - self.environment, QueryStore(self.environment) + self.environment_name, QueryStore(self.environment_name) ) self._add_query_functions() self._query_time = QueryTime(units="day") + def _check_environment( + self, data_environment + ) -> Tuple[Union[str, DataEnvironment], str]: + """Check environment against known names.""" + if isinstance(data_environment, str): + data_env = DataEnvironment.parse(data_environment) + if data_env != DataEnvironment.Unknown: + data_environment = data_env + environment_name = data_environment.name + elif data_environment in drivers.CUSTOM_PROVIDERS: + environment_name = data_environment + else: + raise TypeError(f"Unknown data environment {data_environment}") + elif isinstance(data_environment, DataEnvironment): + environment_name = data_environment.name + else: + raise TypeError( + f"Unknown data environment type {data_environment} ({type(data_environment)})" + ) + return data_environment, environment_name + def __getattr__(self, name): """Return the value of the named property 'name'.""" if "." in name: @@ -147,6 +164,11 @@ def __getattr__(self, name): return getattr(parent, child_name) raise AttributeError(f"{name} is not a valid attribute.") + @property + def environment(self) -> str: + """Return the environment name.""" + return self.environment_name + def connect(self, connection_str: Optional[str] = None, **kwargs): """ Connect to data source. @@ -303,7 +325,7 @@ def _read_queries_from_paths(self, query_paths) -> Dict[str, QueryStore]: for def_qry_path in settings.get("Default"): # type: ignore # only read queries from environment folder builtin_qry_paths = self._get_query_folder_for_env( - def_qry_path, self.environment + def_qry_path, self.environment_name ) all_query_paths.extend( str(qry_path) for qry_path in builtin_qry_paths if qry_path.is_dir() @@ -326,7 +348,7 @@ def _read_queries_from_paths(self, query_paths) -> Dict[str, QueryStore]: driver_query_filter=self._query_provider.query_attach_spec, ) # if no queries - just return an empty store - return {self.environment: QueryStore(self.environment)} + return {self.environment_name: QueryStore(self.environment_name)} def _add_query_functions(self): """Add queries to the module as callable methods.""" diff --git a/msticpy/data/core/query_store.py b/msticpy/data/core/query_store.py index 2075c15d9..a299280a6 100644 --- a/msticpy/data/core/query_store.py +++ b/msticpy/data/core/query_store.py @@ -235,6 +235,7 @@ def apply_query_filter(self, query_filter: Callable[[QuerySource], bool]): for source in self._all_sources: source.show = query_filter(source) + # pylint: disable=too-many-locals @classmethod # noqa: MC0001 def import_files( # noqa: MC0001 cls, @@ -287,11 +288,14 @@ def import_files( # noqa: MC0001 if "." in env_value: env_value = env_value.split(".")[1] environment = DataEnvironment.parse(env_value) - if environment == DataEnvironment.Unknown: - raise ValueError(f"Unknown environment {env_value}") + environment_name = ( + environment.name + if environment != DataEnvironment.Unknown + else env_value + ) - if environment.name not in env_stores: - env_stores[environment.name] = cls(environment=environment.name) + if environment_name not in env_stores: + env_stores[environment_name] = cls(environment=environment_name) for source_name, source in sources.items(): new_source = QuerySource( source_name, source, defaults, metadata @@ -300,7 +304,7 @@ def import_files( # noqa: MC0001 driver_query_filter and _matches_driver_filter(new_source, driver_query_filter) ): - env_stores[environment.name].add_data_source(new_source) + env_stores[environment_name].add_data_source(new_source) return env_stores def get_query( diff --git a/msticpy/data/drivers/__init__.py b/msticpy/data/drivers/__init__.py index 976896d1c..3115ab0a5 100644 --- a/msticpy/data/drivers/__init__.py +++ b/msticpy/data/drivers/__init__.py @@ -5,7 +5,8 @@ # -------------------------------------------------------------------------- """Data provider sub-package.""" import importlib -from typing import Union +from functools import singledispatch +from typing import Dict, Union from ..._version import VERSION from ..core.query_defns import DataEnvironment @@ -35,8 +36,20 @@ DataEnvironment.Kusto_New: ("azure_kusto_driver", "AzureKustoDriver"), } +CUSTOM_PROVIDERS: Dict[str, type] = {} -def import_driver(data_environment: DataEnvironment) -> type: + +@singledispatch +def import_driver(data_environment) -> type: + """Unsupported type for environment.""" + raise TypeError( + "'data_environment' must be a str or DataEnvironment type.", + f"Called with type: {type(data_environment)}", + ) + + +@import_driver.register +def _(data_environment: DataEnvironment) -> type: """Import driver class for a data environment.""" mod_name, cls_name = _ENVIRONMENT_DRIVERS.get(data_environment, (None, None)) @@ -51,3 +64,16 @@ def import_driver(data_environment: DataEnvironment) -> type: f"msticpy.data.drivers.{mod_name}", package="msticpy" ) return getattr(imp_module, cls_name) + + +@import_driver.register +def _(data_environment: str) -> type: + """Import custom driver class for a data environment.""" + if plugin_cls := CUSTOM_PROVIDERS.get(data_environment): + return plugin_cls + + raise ValueError( + f"No driver available for environment {data_environment}.", + "Possible values are:", + ", ".join(CUSTOM_PROVIDERS), + ) diff --git a/msticpy/init/mp_plugins.py b/msticpy/init/mp_plugins.py new file mode 100644 index 000000000..a478b3365 --- /dev/null +++ b/msticpy/init/mp_plugins.py @@ -0,0 +1,115 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +""" +MSTICPy plugin loader. + +To load MSTICPy plugins you must have modules with classes +defined from one of the supported plugin types. + +To specify the locations to look for plugins specify +the folder(s) as absolute paths or relative to the current +directory + +msticpyconfig.yaml +------------------ +PluginFolders: + - tests/testdata/dp_plugins + - tests/testdata/ti_plugins + - tests/testdata/mixed_plugins + +""" + + +import contextlib +import sys +from importlib import import_module +from inspect import getmembers, isabstract, isclass +from pathlib import Path +from types import ModuleType +from typing import Iterable, NamedTuple, Optional, Union +from warnings import warn + +from .._version import VERSION +from ..common.pkg_config import get_config +from ..context.contextlookup import ContextLookup +from ..context.contextproviders.context_provider_base import ContextProvider +from ..context.tilookup import TILookup +from ..context.tiproviders.ti_provider_base import TIProvider +from ..data import drivers +from ..data.drivers import DriverBase + +__version__ = VERSION +__author__ = "Ian Hellen" + +_PLUGIN_KEY = "PluginFolders" + + +class PluginReg(NamedTuple): + """Plugin registration tuple.""" + + reg_dest: Union[type, ModuleType] # class or module containing CUSTOM_PROVIDERS + name_property: Optional[str] # Custom name(s) for provider + + +# This dictionary maps the class of the plugin to +# the module or class where the CUSTOM_PROVIDERS dictionary +# for that provider type is defined. +# This dictionary maps the class of the plugin to +# the module or class where the CUSTOM_PROVIDERS dictionary +# for that provider type is defined. +_PLUGIN_TYPES = { + DriverBase: PluginReg(drivers, "DATA_ENVIRONMENTS"), + TIProvider: PluginReg(TILookup, "PROVIDER_NAME"), + ContextProvider: PluginReg(ContextLookup, "PROVIDER_NAME"), +} + + +def read_plugins(plugin_paths: Union[str, Iterable[str]]): + """Load plugins from folders specified in msticpyconfig.yaml.""" + plugin_config = [plugin_paths] if isinstance(plugin_paths, str) else plugin_paths + if not plugin_config: + with contextlib.suppress(KeyError): + plugin_config = get_config(_PLUGIN_KEY) + if not plugin_config: + return + for plugin_path in plugin_config: + load_plugins_from_path(plugin_path=plugin_path) + + +def load_plugins_from_path(plugin_path: Union[str, Path]): + """Load all compatible plugins found in plugin_path.""" + sys.path.append(str(plugin_path)) + for module_file in Path(plugin_path).glob("*.py"): + try: + module = import_module(module_file.stem) + except ImportError: + warn(f"Unable to import plugin {module_file} from {plugin_path}") + for name, obj in getmembers(module, isclass): + if not isinstance(obj, type): + continue + if isabstract(obj): + continue + plugin_type, reg_object = _get_plugin_type(obj) + if plugin_type: + # if no specified registration, use the root class + reg_dest = reg_object.reg_dest or plugin_type + plugin_names = getattr(obj, reg_object.name_property, name) + if not isinstance(plugin_names, (list, tuple)): + plugin_names = (plugin_names,) + for plugin_name in plugin_names: + reg_dest.CUSTOM_PROVIDERS[plugin_name] = obj + + +def _get_plugin_type(plugin_class): + """Return matching parent class for plugin_class.""" + return next( + ( + (parent_class, plugin_reg) + for parent_class, plugin_reg in _PLUGIN_TYPES.items() + if issubclass(plugin_class, parent_class) and plugin_class != parent_class + ), + (None, None), + ) diff --git a/tests/init/pivot/pivot_fixtures.py b/tests/init/pivot/pivot_fixtures.py index 8d670b256..6378d85c5 100644 --- a/tests/init/pivot/pivot_fixtures.py +++ b/tests/init/pivot/pivot_fixtures.py @@ -16,6 +16,8 @@ from msticpy.data import QueryProvider from msticpy.init.pivot import Pivot +from ...unit_test_lib import custom_mp_config, get_test_data_path + __author__ = "Ian Hellen" # pylint: disable=redefined-outer-name, protected-access @@ -33,6 +35,7 @@ del splunk_driver _SPLUNK_IMP_OK = True + _IPSTACK_IMP_OK = False ip_stack_cls: Optional[type] try: @@ -55,18 +58,21 @@ def exec_connect(provider): def create_data_providers(): """Return dict of providers.""" prov_dict = {} - with warnings.catch_warnings(): - warnings.simplefilter("ignore", category=UserWarning) - if _KQL_IMP_OK: - prov_dict["az_sent_prov"] = QueryProvider("MSSentinel") - prov_dict["mdatp_prov"] = QueryProvider("MDE") - if _SPLUNK_IMP_OK: - prov_dict["splunk_prov"] = QueryProvider("Splunk") - prov_dict["ti_lookup"] = TILookup() - prov_dict["geolite"] = GeoLiteLookup() - - if _IPSTACK_IMP_OK: - prov_dict["ip_stack"] = ip_stack_cls() + with custom_mp_config( + get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + ): + with warnings.catch_warnings(): + warnings.simplefilter("ignore", category=UserWarning) + if _KQL_IMP_OK: + prov_dict["az_sent_prov"] = QueryProvider("MSSentinel") + prov_dict["mdatp_prov"] = QueryProvider("MDE") + if _SPLUNK_IMP_OK: + prov_dict["splunk_prov"] = QueryProvider("Splunk") + prov_dict["ti_lookup"] = TILookup() + prov_dict["geolite"] = GeoLiteLookup() + + if _IPSTACK_IMP_OK: + prov_dict["ip_stack"] = ip_stack_cls() return prov_dict @@ -96,3 +102,16 @@ def create_pivot(data_providers): if isinstance(provider, QueryProvider): exec_connect(provider) return pivot + + +# @pytest.fixture +# def create_pivot(data_providers): +# """Return Pivot instance with initialized data providers.""" +# with warnings.catch_warnings(): +# warnings.simplefilter("ignore", category=UserWarning) +# pivot = Pivot(namespace=data_providers) +# pivot.reload_pivots(namespace=data_providers, clear_existing=True) +# for provider in data_providers.values(): +# if isinstance(provider, QueryProvider): +# exec_connect(provider) +# return pivot diff --git a/tests/init/test_mp_plugins.py b/tests/init/test_mp_plugins.py new file mode 100644 index 000000000..08d5ed0ce --- /dev/null +++ b/tests/init/test_mp_plugins.py @@ -0,0 +1,126 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Tests for mp_plugins module.""" +import re + +import pandas as pd +import pytest +import pytest_check as check +import respx + +from msticpy.common import pkg_config +from msticpy.context.contextlookup import ContextLookup +from msticpy.context.contextproviders.context_provider_base import ContextProvider +from msticpy.context.tilookup import TILookup +from msticpy.context.tiproviders.ti_provider_base import TIProvider +from msticpy.data import QueryProvider, drivers +from msticpy.data.drivers import DriverBase +from msticpy.init.mp_plugins import read_plugins + +from ..unit_test_lib import custom_mp_config, get_test_data_path + +__author__ = "Ian Hellen" + + +_TI_PROV_SETTINGS = { + "Provider1": { + "Args": {"AuthKey": "12345"}, + "Primary": True, + "Provider": "Provider1", + }, + "Provider2": { + "Args": {"AuthKey": "12345"}, + "Primary": True, + "Provider": "Provider2", + }, +} + +# pylint: disable=redefined-outer-name + + +@pytest.fixture(scope="module") +def load_plugins(): + """Fixture_docstring.""" + plugin_path = get_test_data_path().joinpath("plugins") + read_plugins(plugin_paths=str(plugin_path)) + + +def test_load_plugins(load_plugins): + """Load plugins from test data.""" + + for provider in ("Provider1", "Provider2"): + check.is_in(provider, TILookup.CUSTOM_PROVIDERS) + check.is_true(issubclass(TILookup.CUSTOM_PROVIDERS[provider], TIProvider)) + + check.is_true(issubclass(TILookup.CUSTOM_PROVIDERS["Provider2"], TIProvider)) + check.is_in("ContextProviderTest", ContextLookup.CUSTOM_PROVIDERS) + check.is_true( + issubclass( + ContextLookup.CUSTOM_PROVIDERS["ContextProviderTest"], ContextProvider + ) + ) + for provider in ("CustomDataProvA", "SQLTestProvider", "SQLProdProvider"): + check.is_in(provider, drivers.CUSTOM_PROVIDERS) + check.is_true(issubclass(drivers.CUSTOM_PROVIDERS[provider], DriverBase)) + + +def test_custom_data_provider(load_plugins): + """Test data provider operations.""" + qry_prov = QueryProvider( + "SQLProdProvider", query_paths=[str(get_test_data_path().joinpath("plugins"))] + ) + qry_prov.connect() + results = qry_prov.exec_query("Get test query data") + check.is_true(results.shape, (1, 4)) + check.is_true(hasattr(qry_prov, "CustomSQL")) + check.is_true(hasattr(qry_prov.CustomSQL, "accessibility_persistence")) + + results = qry_prov.CustomSQL.accessibility_persistence() + check.is_true(results.shape, (1, 4)) + + qry_prov2 = QueryProvider( + "CustomDataProvA", query_paths=["tests/testdata/plugins/"] + ) + qry_prov2.connect() + results = qry_prov2.Custom.file_path(path="foo") + + +# pylint: disable=protected-access +@respx.mock +def test_custom_ti_provider(load_plugins): + """Test TI plugin.""" + config_path = get_test_data_path().parent.joinpath("msticpyconfig-test.yaml") + with custom_mp_config(config_path): + pkg_config._settings["TIProviders"].update(_TI_PROV_SETTINGS) + print("Settings", pkg_config._settings["TIProviders"]) + ti = TILookup() + + check.is_in("Provider1", ti.available_providers) + print("Loaded provs", ti.loaded_providers.keys()) + check.is_in("Provider1", ti.loaded_providers) + check.is_in("Provider2", ti.loaded_providers) + + results = ti.lookup_ioc(ioc="20.1.2.3", providers=["Provider1"]) + check.equal(results.shape, (1, 6)) + + _df_results = pd.DataFrame( + [ + { + "ID": "tests results", + "Status": 0, + **{name: str(idx) for idx, name in enumerate("ABC")}, + } + ] + ) + respx.get(re.compile("https://api\.service\.com/.*")).respond( + 200, json=_df_results.iloc[0].to_dict() + ) + + results = ti.lookup_ioc(ioc="20.1.2.3", providers=["Provider2"]) + check.equal(results.shape, (1, 11)) + check.is_in( + "https://api.service.com/api/v1/indicators/IPv4", results.iloc[0].Reference + ) diff --git a/tests/init/test_nbinit.py b/tests/init/test_nbinit.py index 24c3d5a45..594d4a90c 100644 --- a/tests/init/test_nbinit.py +++ b/tests/init/test_nbinit.py @@ -220,26 +220,16 @@ def test_check_config(conf_file, expected, tmp_path, monkeypatch): else: os.chdir(str(tmp_path)) - with custom_mp_config(settings_file, path_check=False): - monkeypatch.setattr(nbinit, "current_config_path", lambda: None) - monkeypatch.setattr(nbinit, "is_in_aml", lambda: True) - monkeypatch.setattr(azure_ml_tools, "get_aml_user_folder", lambda: tmp_path) - result = _get_or_create_config() - - print("result=", result) - # print("errs=", "\n".join(errs) if errs else "no errors") - # print("warnings=", "\n".join(warns) if warns else "no warnings") - check.equal(result, expected, "Result") - # reported_errs = 0 if not errs else len(errs) - # reported_warns = 0 if not warns else len(warns) - # if isinstance(expected.errs, tuple): - # check.is_in(reported_errs, expected.errs, "Num errors") - # else: - # check.equal(reported_errs, expected.errs, "Num errors") - # if isinstance(expected.wrns, tuple): - # check.is_in(reported_warns, expected.wrns, "Num errors") - # else: - # check.equal(reported_warns, expected.wrns, "Num warnings") + # with custom_mp_config(settings_file, path_check=False): + monkeypatch.setenv("MSTICPYCONFIG", str(settings_file)) + monkeypatch.setattr(nbinit, "current_config_path", lambda: None) + monkeypatch.setattr(nbinit, "is_in_aml", lambda: True) + monkeypatch.setattr(azure_ml_tools, "get_aml_user_folder", lambda: tmp_path) + result = _get_or_create_config() + + print("result=", result) + check.equal(result, expected, "Result") + finally: os.chdir(init_cwd) diff --git a/tests/msticpyconfig-test.yaml b/tests/msticpyconfig-test.yaml index 32c8e4859..a6bd04c78 100644 --- a/tests/msticpyconfig-test.yaml +++ b/tests/msticpyconfig-test.yaml @@ -33,6 +33,8 @@ QueryDefinitions: - "queries" Custom: - "testdata" +PluginFolders: + - tests/testdata/plugins Azure: cloud: "global" auth_methods: ["cli", "msi", "interactive"] diff --git a/tests/testdata/plugins/custom_prov_a_queries.yaml b/tests/testdata/plugins/custom_prov_a_queries.yaml new file mode 100644 index 000000000..fc44fec07 --- /dev/null +++ b/tests/testdata/plugins/custom_prov_a_queries.yaml @@ -0,0 +1,76 @@ +metadata: + version: 1 + description: Custom plugin test Queries + data_environments: [CustomDataProvA] + data_families: [Custom] + tags: ["file"] +defaults: + metadata: + data_source: "file_events" + parameters: + table: + description: Table name + type: str + default: "DeviceProcessEvents" + start: + description: Query start time + type: datetime + end: + description: Query end time + type: datetime + add_query_items: + description: Additional query clauses + type: str + default: "" +sources: + list_files: + description: Lists all file events by filename + metadata: + pivot: + short_name: file_events_filename + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FileName has "{file_name}" + {add_query_items}' + uri: None + parameters: + file_name: + description: Name of file + type: str + file_path: + description: Lists all file events from files in a certain path + metadata: + pivot: + short_name: file_events_path + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where FolderPath contains "{path}" + {add_query_items}' + parameters: + path: + description: Full or partial path to search in + type: str + list_filehash: + description: Lists all file events by hash + metadata: + pivot: + short_name: file_events_hash + args: + query: ' + {table} + | where Timestamp >= datetime({start}) + | where Timestamp <= datetime({end}) + | where SHA1 == "{file_hash}" or SHA256 == "{file_hash}" or MD5 == "{file_hash}" + {add_query_items}' + uri: None + parameters: + file_hash: + description: Hash of file + type: str + aliases: hash diff --git a/tests/testdata/plugins/data_prov.py b/tests/testdata/plugins/data_prov.py new file mode 100644 index 000000000..4b7589f30 --- /dev/null +++ b/tests/testdata/plugins/data_prov.py @@ -0,0 +1,47 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Test module for plugin import tests.""" +from typing import Any, Optional, Tuple, Union + +import pandas as pd + +from msticpy.data.core.query_source import QuerySource +from msticpy.data.drivers import DriverBase + +__author__ = "Ian Hellen" + +results_df = pd.DataFrame( + [{"ID": "tests results", **{name: str(idx) for idx, name in enumerate("ABC")}}] +) + + +class CustomDataProvA(DriverBase): + """Custom provider 1.""" + + def __init__(self, **kwargs): + """Initialize the class.""" + super().__init__(**kwargs) + self._loaded = True + + def connect(self, connection_str: Optional[str] = None, **kwargs): + """Connect to data source.""" + self._connected = True + + def query( + self, query: str, query_source: QuerySource = None, **kwargs + ) -> Union[pd.DataFrame, Any]: + """Execute query string and return DataFrame of results.""" + return self.query_with_results(query, **kwargs)[0] + + def query_with_results(self, query: str, **kwargs) -> Tuple[pd.DataFrame, Any]: + """Return results with optional status object.""" + return results_df, True + + +class CustomDataProvB(CustomDataProvA): + """Custom provider 2.""" + + DATA_ENVIRONMENTS = ["SQLTestProvider", "SQLProdProvider"] diff --git a/tests/testdata/plugins/sql_test_queries.yaml b/tests/testdata/plugins/sql_test_queries.yaml new file mode 100644 index 000000000..f50ff0741 --- /dev/null +++ b/tests/testdata/plugins/sql_test_queries.yaml @@ -0,0 +1,509 @@ +metadata: + version: 1 + description: MDATP Queries + data_environments: [SQLTestProvider, SQLProdProvider] + data_families: [CustomSQL] + tags: ['user'] +defaults: + metadata: + data_source: 'hunting_queries' + parameters: + start: + description: Query start time + type: datetime + end: + description: Query end time + type: datetime + add_query_items: + description: Additional query clauses + type: str + default: '' +sources: + doc_with_link: + description: Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. + metadata: + args: + query: ' + let minTimeRange = ago(7d); + let wordLinks = + DeviceEvents + // Filter on click on links from WinWord + | where Timestamp > minTimeRange and ActionType == "BrowserLaunchedToOpenUrl" and isnotempty(RemoteUrl) and InitiatingProcessFileName =~ "winword.exe" + | project ClickTime=Timestamp, DeviceId, DeviceName, ClickUrl=RemoteUrl; + let docAttachments = + DeviceFileEvents + | where Timestamp > minTimeRange + // Query for common document file extensions + and (FileName endswith ".docx" or FileName endswith ".docm" or FileName endswith ".doc") + // Query for files saved from email clients such as the Office Outlook app or the Windows Mail app + and InitiatingProcessFileName in~ ("outlook.exe", "hxoutlook.exe") + | summarize AttachmentSaveTime=min(Timestamp) by AttachmentName=FileName, DeviceId; + let browserDownloads = + DeviceFileEvents + | where Timestamp > minTimeRange + // Query for files created by common browsers + and InitiatingProcessFileName in~ ("browser_broker.exe", "chrome.exe", "iexplore.exe", "firefox.exe") + // Exclude JS files that are used for loading sites (but still query for JS files that are known to be downloaded) + and not (FileName endswith ".js" and isempty(FileOriginUrl)) + // Further filter to exclude file extensions that are less indicative of an attack (when there were already previously a doc attachment that included a link) + | where FileName !endswith ".partial" and FileName !endswith ".docx" + | summarize (Timestamp, SHA1) = argmax(Timestamp, SHA1) by FileName, DeviceId, FileOriginUrl; + // Perf tip: start the joins from the smallest table (put it on the left-most side of the joins) + wordLinks + | join kind= inner (docAttachments) on DeviceId | where ClickTime - AttachmentSaveTime between (0min..3min) + | join kind= inner (browserDownloads) on DeviceId | where Timestamp - ClickTime between (0min..3min) + // Aggregating multiple "attachments" together - because oftentimes the same file is stored multiple times under different names + | summarize Attachments=makeset(AttachmentName), AttachmentSaveTime=min(AttachmentSaveTime), ClickTime=min(ClickTime) + by bin(Timestamp, 1tick), FileName, FileOriginUrl, ClickUrl, SHA1, DeviceName, DeviceId + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Delivery/Doc%20attachment%20with%20link%20to%20download.txt" + parameters: + dropbox_link: + description: Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. + metadata: + args: + query: ' + DeviceFileEvents + | where + Timestamp > ago(7d) + and FileOriginUrl startswith "https://dl.dropboxusercontent.com/" + and isnotempty(FileOriginReferrerUrl) + and FileOriginReferrerUrl !startswith "https://www.dropbox.com/" + | project FileOriginReferrerUrl, FileName + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Delivery/Dropbox%20downloads%20linked%20from%20other%20site.txt" + parameters: + email_smartscreen: + description: Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning + metadata: + args: + query: ' + let smartscreenAppWarnings = + // Query for SmartScreen warnings of unknown executed applications + DeviceEvents + | where ActionType == "SmartScreenAppWarning" + | project WarnTime=Timestamp, DeviceName, WarnedFileName=FileName, WarnedSHA1=SHA1, ActivityId=extractjson("$.ActivityId", AdditionalFields, typeof(string)) + // Select only warnings that the user has decided to ignore and has executed the app. + | join kind=leftsemi ( + DeviceEvents + | where ActionType == "SmartScreenUserOverride" + | project DeviceName, ActivityId=extractjson("$.ActivityId", AdditionalFields, typeof(string))) + on DeviceName, ActivityId + | project-away ActivityId; + // Query for links opened from outlook, that are close in time to a SmartScreen warning + let emailLinksNearSmartScreenWarnings = + DeviceEvents + | where ActionType == "BrowserLaunchedToOpenUrl" and isnotempty(RemoteUrl) and InitiatingProcessFileName =~ "outlook.exe" + | extend WasOutlookSafeLink=(tostring(parse_url(RemoteUrl).Host) endswith "safelinks.protection.outlook.com") + | project DeviceName, MailLinkTime=Timestamp, + MailLink=iff(WasOutlookSafeLink, url_decode(tostring(parse_url(RemoteUrl)["Query Parameters"]["url"])), RemoteUrl) + | join kind=inner smartscreenAppWarnings on DeviceName | where (WarnTime-MailLinkTime) between (0min..4min); + // Add the browser download event to tie in all the dots + DeviceFileEvents + | where isnotempty(FileOriginUrl) and InitiatingProcessFileName in~ ("chrome.exe", "browser_broker.exe") + | project FileName, FileOriginUrl, FileOriginReferrerUrl, DeviceName, Timestamp, SHA1 + | join kind=inner emailLinksNearSmartScreenWarnings on DeviceName + | where (Timestamp-MailLinkTime) between (0min..3min) and (WarnTime-Timestamp) between (0min..1min) + | project FileName, MailLink, FileOriginUrl, FileOriginReferrerUrl, WarnedFileName, DeviceName, SHA1, WarnedSHA1, Timestamp + | distinct * + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Delivery/Email%20link%20%2B%20download%20%2B%20SmartScreen%20warning.txt" + parameters: + email_link: + description: Look for links opened from mail apps – if a detection occurred right afterwards + metadata: + args: + query: ' + let minTimeRange = ago(7d); + let outlookLinks = + DeviceEvents + // Filter on click on links from outlook + | where Timestamp > minTimeRange and ActionType == "BrowserLaunchedToOpenUrl" and isnotempty(RemoteUrl) + | where + // outlook.exe is the Office Outlook app + InitiatingProcessFileName =~ "outlook.exe" + // RuntimeBroker.exe opens links for all apps from the Windows store, including the Windows Mail app (HxOutlook.exe). + // However, it will also include some links opened from other apps. + or InitiatingProcessFileName =~ "runtimebroker.exe" + | project Timestamp, DeviceId, DeviceName, RemoteUrl, InitiatingProcessFileName, ParsedUrl=parse_url(RemoteUrl) + // When applicable, parse the link sent via email from the clicked O365 ATP SafeLink + | extend WasOutlookSafeLink=(tostring(ParsedUrl.Host) endswith "safelinks.protection.outlook.com") + | project Timestamp, DeviceId, DeviceName, WasOutlookSafeLink, InitiatingProcessFileName, + OpenedLink=iff(WasOutlookSafeLink, url_decode(tostring(ParsedUrl["Query Parameters"]["url"])), RemoteUrl); + let alerts = + AlertInfo | join AlertEvidence on AlertId + | summarize (FirstDetectedActivity, Title)=argmin(Timestamp, Title) by AlertId, DeviceId + // Filter alerts that include events from before the queried time period + | where FirstDetectedActivity > minTimeRange; + // Join the two together - looking for alerts that are right after an abnormal network logon + alerts | join kind=inner (outlookLinks) on DeviceId | where FirstDetectedActivity - Timestamp between (0min..3min) + // If there are multiple alerts close to a single click-on-link, aggregate them together to a single row + // Note: bin(Timestamp, 1tick) is used because when summarizing by a datetime field, the default "bin" used is 1-hour. + | summarize FirstDetectedActivity=min(FirstDetectedActivity), AlertTitles=makeset(Title) by OpenedLink, InitiatingProcessFileName, Timestamp=bin(Timestamp, 1tick), DeviceName, DeviceId, WasOutlookSafeLink + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Delivery/Open%20email%20link.txt" + parameters: + av_sites: + description: Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites + metadata: + args: + query: ' + let detectedDownloads = + DeviceEvents + | where ActionType == "AntivirusDetection" and isnotempty(FileOriginUrl) + | project Timestamp, FileOriginUrl, FileName, DeviceId, + ThreatName=tostring(parse_json(AdditionalFields).ThreatName) + // Filter out less severe threat categories on which we do not want to pivot + | where ThreatName !startswith "PUA" + and ThreatName !startswith "SoftwareBundler:" + and FileOriginUrl != "about:internet"; + let detectedDownloadsSummary = + detectedDownloads + // Get a few examples for each detected Host: + // up to 4 filenames, up to 4 threat names, one full URL) + | summarize DetectedUrl=any(FileOriginUrl), + DetectedFiles=makeset(FileName, 4), + ThreatNames=makeset(ThreatName, 4) + by Host=tostring(parse_url(FileOriginUrl).Host); + // Query for downloads from sites from which other downloads were detected by Windows Defender Antivirus + DeviceFileEvents + | where isnotempty(FileOriginUrl) + | project FileName, FileOriginUrl, DeviceId, Timestamp, + Host=tostring(parse_url(FileOriginUrl).Host), SHA1 + // Filter downloads from hosts serving detected files + | join kind=inner(detectedDownloadsSummary) on Host + // Filter out download file create events that were also detected. + // This is needed because sometimes both of these events will be reported, + // and sometimes only the AntivirusDetection event - depending on timing. + | join kind=leftanti(detectedDownloads) on DeviceId, FileOriginUrl + // Summarize a single row per host - with the machines count + // and an example event for a missed download (select the last event) + | summarize MachineCount=dcount(DeviceId), arg_max(Timestamp, *) by Host + // Filter out common hosts, as they probably ones that also serve benign files + | where MachineCount < 20 + | project Host, MachineCount, DeviceId, FileName, DetectedFiles, + FileOriginUrl, DetectedUrl, ThreatNames, Timestamp, SHA1 + | order by MachineCount desc + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Delivery/Pivot%20from%20detections%20to%20related%20downloads.txt" + parameters: + tor: + description: Looks for Tor client, or for a common Tor plugin called Meek. + metadata: + args: + query: ' + DeviceNetworkEvents + | where Timestamp < ago(3d) and InitiatingProcessFileName in~ ("tor.exe", "meek-client.exe") + // Returns MD5 hashes of files used by Tor, to enable you to block them. + // We count how prevalent each file is (by machines) and show examples for some of them (up to 5 machine names per hash). + | summarize MachineCount=dcount(DeviceName), MachineNames=makeset(DeviceName, 5) by InitiatingProcessMD5 + | order by MachineCount desc + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Command%20and%20Control/Tor.txt" + parameters: + network_scans: + description: Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process + metadata: + args: + query: ' + let remotePortCountThreshold = 10; // Please change the min value, for a host reaching out to remote ports on a remote IP, that you consider to be threshold for a suspicious behavior + DeviceNetworkEvents + | where Timestamp > ago(1d) and RemoteIP startswith "172.16" or RemoteIP startswith "192.168" + | summarize + by DeviceName, RemoteIP, RemotePort, InitiatingProcessFileName + | summarize RemotePortCount=dcount(RemotePort) by DeviceName, RemoteIP, InitiatingProcessFileName + | where RemotePortCount > remotePortCountThreshold + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Command%20and%20Control/Tor.txt" + parameters: + user_enumeration: + description: The query finds attempts to list users or groups using Net commands + metadata: + args: + query: ' + DeviceProcessEvents + | where Timestamp > ago(14d) + | where FileName == "net.exe" and AccountName != "" and ProcessCommandLine !contains "\\" and ProcessCommandLine !contains "/add" + | where (ProcessCommandLine contains " user " or ProcessCommandLine contains " group ") and (ProcessCommandLine contains " /do" or ProcessCommandLine contains " /domain") + | extend Target = extract("(?i)[user|group] (\"*[a-zA-Z0-9-_ ]+\"*)", 1, ProcessCommandLine) | filter Target != "" + | project AccountName, Target, ProcessCommandLine, DeviceName, Timestamp + | sort by AccountName, Target + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Discovery/Enumeration%20of%20users%20%26%20groups%20for%20lateral%20movement.txt" + parameters: + smb_discovery: + description: Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. + metadata: + args: + query: ' + DeviceNetworkEvents + | where RemotePort == 445 and Timestamp > ago(7d) + // Exclude Kernel processes, as they are too noisy in this query + and InitiatingProcessId !in (0, 4) + | summarize RemoteIPCount=dcount(RemoteIP) by DeviceName, InitiatingProcessFileName, InitiatingProcessId, InitiatingProcessCreationTime + | where RemoteIPCount > 10 + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Discovery/SMB%20shares%20discovery.txt" + parameters: + b64_pe: + description: Finding base64 encoded PE files header seen in the command line parameters + metadata: + args: + query: ' + DeviceProcessEvents + | where Timestamp > ago(7d) + | where ProcessCommandLine contains "TVqQAAMAAAAEAAA" + | top 1000 by Timestamp + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Execution/Base64encodePEFile.txt" + parameters: + malware_recycle: + description: Finding attackers hiding malware in the recycle bin. + metadata: + args: + query: ' + DeviceProcessEvents + | where Timestamp > ago(7d) + | where FileName in~("cmd.exe","ftp.exe","schtasks.exe","powershell.exe","rundll32.exe","regsvr32.exe","msiexec.exe") + | where ProcessCommandLine contains ":\\recycler" + | project Timestamp, DeviceName, ProcessCommandLine, InitiatingProcessFileName + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Execution/Malware_In_recyclebin.txt" + parameters: + powershell_downloads: + description: Finds PowerShell execution events that could involve a download. + metadata: + args: + query: ' + DeviceProcessEvents + | where Timestamp > ago(7d) + | where FileName in~ ("powershell.exe", "powershell_ise.exe") + | where ProcessCommandLine has "Net.WebClient" + or ProcessCommandLine has "DownloadFile" + or ProcessCommandLine has "Invoke-WebRequest" + or ProcessCommandLine has "Invoke-Shellcode" + or ProcessCommandLine has "http" + or ProcessCommandLine has "IEX" + or ProcessCommandLine has "Start-BitsTransfer" + or ProcessCommandLine has "mpcmdrun.exe" + | project Timestamp, DeviceName, InitiatingProcessFileName, FileName, ProcessCommandLine + | top 100 by Timestamp + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Execution/PowerShell%20downloads.txt" + parameters: + uncommon_powershell: + description: Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. + metadata: + args: + query: ' + let DeviceId = "{host_name}"; + let timestamp = datetime({timestamp}); + let powershellCommands = + DeviceEvents + | where ActionType == "PowerShellCommand" + // Extract the powershell command name from the Command field in the AdditionalFields JSON column + | project PowershellCommand=extractjson("$.Command", AdditionalFields, typeof(string)), InitiatingProcessCommandLine, InitiatingProcessParentFileName, Timestamp, DeviceId + | where PowershellCommand !endswith ".ps1" and PowershellCommand !endswith ".exe"; + // Filter Powershell cmdlets executed on relevant machine and time period + powershellCommands | where DeviceId == DeviceId and Timestamp between ((timestamp-5min) .. 10min) + // Filter out common powershell cmdlets + | join kind=leftanti (powershellCommands | summarize MachineCount=dcount(DeviceId) by PowershellCommand | where MachineCount > 20) on PowershellCommand + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Execution/PowershellCommand%20-%20uncommon%20commands%20on%20machine.txt" + parameters: + host_name: + description: hostname of computer to focus query on + type: str + aliases: + - hostname + timestamp: + description: timestamp to base investigation scope on + type: str + cve_2018_1000006l: + description: Looks for CVE-2018-1000006 exploitation + metadata: + args: + query: ' + DeviceProcessEvents + | where Timestamp > ago(14d) + | where FileName in~ ("code.exe", "skype.exe", "slack.exe", "teams.exe") + | where InitiatingProcessFileName in~ ("iexplore.exe", "runtimebroker.exe", "chrome.exe") + | where ProcessCommandLine has "--gpu-launcher" + | summarize FirstEvent=min(Timestamp), LastEvent=max(Timestamp) by DeviceName, ProcessCommandLine, FileName, InitiatingProcessFileName + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Exploits/Electron-CVE-2018-1000006.txt" + parameters: + cve_2018_4878: + description: This query checks for specific processes and domain TLD used in the CVE-2018-4878 + metadata: + args: + query: ' + DeviceNetworkEvents + | where Timestamp > ago(14d) + | where InitiatingProcessFileName =~ "cmd.exe" and InitiatingProcessParentFileName =~ "excel.exe" + | where RemoteUrl endswith ".kr" + | project Timestamp, DeviceName, RemoteIP, RemoteUrl + | top 100 by Timestamp + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Exploits/Flash-CVE-2018-4848.txt" + parameters: + cve_2018_1111: + description: Looks for CVE-2018-1111 exploitation + metadata: + args: + query: ' + DeviceProcessEvents + | where InitiatingProcessCommandLine contains "/etc/NetworkManager/dispatcher.d/" + and InitiatingProcessCommandLine contains "-dhclient" + and isnotempty(ProcessCommandLine) + and FileName !endswith ".exe" + | project Timestamp, DeviceName , FileName, ProcessCommandLine, InitiatingProcessCommandLine + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Exploits/Linux-DynoRoot-CVE-2018-1111.txt" + parameters: + brute_force: + description: Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. + metadata: + args: + query: ' + DeviceLogonEvents + | where isnotempty(RemoteIP) + and AccountName !endswith "$" + and RemoteIPType == "Public" + | extend Account=strcat(AccountDomain, "\\", AccountName) + | summarize + Successful=countif(ActionType == "LogonSuccess"), + Failed = countif(ActionType == "LogonFailed"), + FailedAccountsCount = dcountif(Account, ActionType == "LogonFailed"), + SuccessfulAccountsCount = dcountif(Account, ActionType == "LogonSuccess"), + FailedAccounts = makeset(iff(ActionType == "LogonFailed", Account, ""), 5), + SuccessfulAccounts = makeset(iff(ActionType == "LogonSuccess", Account, ""), 5) + by DeviceName, RemoteIP, RemoteIPType + | where Failed > 10 and Successful > 0 and FailedAccountsCount > 2 and SuccessfulAccountsCount == 1 + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Lateral%20Movement/Account%20brute%20force.txt" + parameters: + service_account_powershell: + description: Service Accounts Performing Remote PowerShell + metadata: + args: + query: ' + let InteractiveTypes = pack_array( + "Interactive", + "CachedInteractive", + "Unlock", + "RemoteInteractive", + "CachedRemoteInteractive", + "CachedUnlock" + ); + let WhitelistedCmdlets = pack_array( + "Out-Default", + "out-lineoutput", + "format-default", + "Set-StrictMode", + "TabExpansion2" + ); + let WhitelistedAccounts = pack_array("FakeWhitelistedAccount"); + DeviceLogonEvents + | where AccountName !in~ (WhitelistedAccounts) + | where ActionType == "LogonSuccess" + | where AccountName !contains "$" + | where AccountName !has "winrm va_" + | extend IsInteractive=(LogonType in (InteractiveTypes)) + | summarize HasInteractiveLogon=max(IsInteractive) + by AccountName + | where HasInteractiveLogon == 0 + | join kind=rightsemi ( + DeviceEvents + | where ActionType == "PowerShellCommand" + | where InitiatingProcessFileName =~ "wsmprovhost.exe" + | extend AccountName = InitiatingProcessAccountName + ) on AccountName + | extend Command = tostring(extractjson("$.Command", AdditionalFields)) + | where Command !in (WhitelistedCmdlets) + | summarize (Timestamp, ReportId)=argmax(Timestamp, ReportId), + makeset(Command), count(), min(Timestamp) by + AccountName, DeviceName, DeviceId + | order by AccountName asc + | where min_Timestamp > ago(1d) + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Lateral%20Movement/ServiceAccountsPerformingRemotePS.txt" + parameters: + accessibility_persistence: + description: This query looks for persistence or privilege escalation done using Windows Accessibility features. + metadata: + args: + query: ' + let minTime = ago(7d); + let accessibilityProcessNames = dynamic(["utilman.exe","osk.exe","magnify.exe","narrator.exe","displayswitch.exe","atbroker.exe","sethc.exe", "helppane.exe"]); + // Query for debuggers attached using a Registry setting to the accessibility processes + let attachedDebugger = + DeviceRegistryEvents + | where Timestamp > minTime + and RegistryKey startswith @"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\" + and RegistryValueName =~ "debugger" + // Parse the debugged process name from the registry key + | parse RegistryKey with @"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\" FileName + | where FileName in~ (accessibilityProcessNames) and isnotempty(RegistryValueData) + | project Technique="AttachedDebugger", FileName, AttachedDebuggerCommandline=RegistryValueData, InitiatingProcessCommandLine, Timestamp, DeviceName; + // Query for overwrites of the accessibility files + let fileOverwiteOfAccessibilityFiles = + DeviceFileEvents + | where Timestamp > minTime + and FileName in~ (accessibilityProcessNames) + and FolderPath contains @"Windows\System32" + | project Technique="OverwriteFile", Timestamp, DeviceName, FileName, SHA1, InitiatingProcessCommandLine; + // Query for unexpected hashes of processes with names matching the accessibility processes. + // Specifically, query for hashes matching cmd.exe and powershell.exe, as these MS-signed general-purpose consoles are often used with this technique. + let executedProcessIsPowershellOrCmd = + DeviceProcessEvents + | project Technique="PreviousOverwriteFile", Timestamp, DeviceName, FileName, SHA1 + | where Timestamp > minTime + | where FileName in~ (accessibilityProcessNames) + | join kind=leftsemi( + DeviceProcessEvents + | where Timestamp > ago(14d) and (FileName =~ "cmd.exe" or FileName =~ "powershell.exe") + | summarize MachinesCount = dcount(DeviceName) by SHA1 + | where MachinesCount > 5 + | project SHA1 + ) on SHA1; + // Union all results together. + // An outer union is used because the schemas are a bit different between the tables - and we want to get the superset of all tables combined. + attachedDebugger + | union kind=outer fileOverwiteOfAccessibilityFiles + | union kind=outer executedProcessIsPowershellOrCmd + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Persistence/Accessibility%20Features.txt" + parameters: + smartscreen_ignored: + description: Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. + metadata: + args: + query: ' + let minTimeRange = ago(7d); + let smartscreenUrlBlocks = + DeviceEvents + | where ActionType == "SmartScreenUrlWarning" and Timestamp > minTimeRange + // Filter out SmartScreen test URLs under https://demo.smartscreen.msft.net/ + and RemoteUrl !startswith "https://demo.smartscreen.msft.net/" + | extend ParsedFields=parse_json(AdditionalFields) + | project Timestamp, DeviceName, BlockedUrl=RemoteUrl, Recommendation=tostring(ParsedFields.Recommendation), Experience=tostring(ParsedFields.Experience), ActivityId=tostring(ParsedFields.ActivityId); + // Query for UserDecision events - each one means the user has decided to ignore the warning and run the app. + let userIgnoredWarning= + DeviceEvents + | where ActionType == "SmartScreenUserOverride" and Timestamp > minTimeRange + | project DeviceName, ActivityId=extractjson("$.ActivityId", AdditionalFields, typeof(string)); + // Join the block and user decision event using an ActivityId + let ignoredBlocks = smartscreenUrlBlocks | join kind=leftsemi (userIgnoredWarning) on DeviceName, ActivityId | project-away ActivityId; + // Optional additional filter - look only for cases where a file was downloaded from Microsoft Edge following the URL block being ignored + let edgeDownloads = + DeviceFileEvents + | where Timestamp > minTimeRange and InitiatingProcessFileName =~ "browser_broker.exe" + | summarize (DownloadTime, SHA1) = argmax(Timestamp, SHA1) by FileName, DeviceName, FileOriginUrl, FileOriginReferrerUrl; + ignoredBlocks + | join kind=inner (edgeDownloads) on DeviceName + | where DownloadTime - Timestamp between (0min .. 2min) + | project-away DeviceName1 + {add_query_items}' + uri: "https://github.com/microsoft/WindowsDefenderATP-Hunting-Queries/blob/master/Protection%20events/SmartScreen%20URL%20block%20ignored%20by%20user.txt" + parameters: \ No newline at end of file diff --git a/tests/testdata/plugins/ti_prov.py b/tests/testdata/plugins/ti_prov.py new file mode 100644 index 000000000..c41cdec81 --- /dev/null +++ b/tests/testdata/plugins/ti_prov.py @@ -0,0 +1,113 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Test module for plugin import tests.""" +from typing import Any, Dict, Tuple + +import pandas as pd + +from msticpy.context.contextproviders.context_provider_base import ContextProvider +from msticpy.context.http_provider import APILookupParams +from msticpy.context.tiproviders import TIProvider +from msticpy.context.tiproviders.ti_http_provider import HttpTIProvider, ResultSeverity + +__author__ = "Ian Hellen" + + +def _parse_results(response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + return ( + ( + True, + ResultSeverity.information, + {"Detail1": "something", "Detail2": "something_else"}, + ) + if response + else (False, ResultSeverity.information, "Not found.") + ) + + +_QUERIES = { + "ipv4": APILookupParams(path="/api/v1/indicators/IPv4/{observable}/general"), + "ipv6": APILookupParams(path="/api/v1/indicators/IPv6/{observable}/general"), + "file_hash": APILookupParams(path="/api/v1/indicators/file/{observable}/general"), + "url": APILookupParams(path="/api/v1/indicators/url/{observable}/general"), +} +# aliases +_QUERIES["md5_hash"] = _QUERIES["file_hash"] +_QUERIES["sha1_hash"] = _QUERIES["file_hash"] +_QUERIES["sha256_hash"] = _QUERIES["file_hash"] + + +_df_results = pd.DataFrame( + [ + { + "ID": "tests results", + "Status": 0, + **{name: str(idx) for idx, name in enumerate("ABC")}, + } + ] +) + + +class TIProviderTest(TIProvider): + """Custom IT provider TI Base.""" + + PROVIDER_NAME = "Provider1" + _BASE_URL = "https://api.service.com" + _QUERIES = _QUERIES + _REQUIRED_PARAMS = ["AuthKey"] + + def lookup_ioc( + self, + ioc: str, + ioc_type: str = None, + query_type: str = None, + **kwargs, + ) -> pd.DataFrame: + """Lookup a single IoC observable.""" + return _df_results + + def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + return _parse_results(response) + + +class TIProviderHttpTest(HttpTIProvider): + """Custom IT provider TI HTTP.""" + + PROVIDER_NAME = "Provider2" + _BASE_URL = "https://api.service.com" + _QUERIES = _QUERIES + _REQUIRED_PARAMS = ["AuthKey"] + + def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + return _parse_results(response) + + +class ContextProviderTest(ContextProvider): + """Custom context provider TI HTTP.""" + + name = "ContextTest" + + _BASE_URL = "https://api.service.com" + _QUERIES = _QUERIES + _REQUIRED_PARAMS = ["AuthKey"] + + def lookup_observable( # type: ignore + self, + observable: str, + observable_type: str = None, + query_type: str = None, + **kwargs, + ) -> pd.DataFrame: + """Lookup a single observable.""" + return _df_results + + def parse_results(self, response: Dict) -> Tuple[bool, Any]: + """Return the details of the response.""" + results = _parse_results(response) + return results[0], results[2] diff --git a/tests/unit_test_lib.py b/tests/unit_test_lib.py index a0d6207d2..9d8fb15c3 100644 --- a/tests/unit_test_lib.py +++ b/tests/unit_test_lib.py @@ -62,21 +62,23 @@ def custom_mp_config( raise FileNotFoundError(f"Setting MSTICPYCONFIG to non-existent file {mp_path}") _lock_file_path = "./.mp_settings.lock" try: - # We need to lock the settings since these are global - # Otherwise the tests interfere with each other. with FileLock(_lock_file_path): - os.environ[pkg_config._CONFIG_ENV_VAR] = str(mp_path) - pkg_config.refresh_config() - yield pkg_config._settings + try: + # We need to lock the settings since these are global + # Otherwise the tests interfere with each other. + os.environ[pkg_config._CONFIG_ENV_VAR] = str(mp_path) + pkg_config.refresh_config() + yield pkg_config._settings + finally: + if not current_path: + del os.environ[pkg_config._CONFIG_ENV_VAR] + else: + os.environ[pkg_config._CONFIG_ENV_VAR] = current_path + pkg_config.refresh_config() finally: - if not current_path: - del os.environ[pkg_config._CONFIG_ENV_VAR] - else: - os.environ[pkg_config._CONFIG_ENV_VAR] = current_path if Path(_lock_file_path).is_file(): with suppress(Exception): Path(_lock_file_path).unlink() - pkg_config.refresh_config() @contextmanager From bbfbc987396592f7ff40f34e7d873109b72847b9 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Fri, 19 May 2023 15:47:51 -0700 Subject: [PATCH 09/26] Format of cluster name has changed in new KustoClient. Fixing test cases to allow for old and new format. (#667) Build-breaking in test cases - no change to production code. --- tests/data/drivers/test_azure_kusto_driver.py | 27 +++++++++---------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/tests/data/drivers/test_azure_kusto_driver.py b/tests/data/drivers/test_azure_kusto_driver.py index 2a405a4b8..db8e8f0e1 100644 --- a/tests/data/drivers/test_azure_kusto_driver.py +++ b/tests/data/drivers/test_azure_kusto_driver.py @@ -204,8 +204,9 @@ class ConnectTest(NamedTuple): init_args={"connection_str": "https://test1.kusto.windows.net"}, connect_args={"connection_str": "https://test.kusto.windows.net"}, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://test.kusto.windows.net" + lambda driver: driver.client._kusto_cluster.startswith( + "https://test.kusto.windows.net" + ) ], ), ConnectTest( @@ -213,8 +214,6 @@ class ConnectTest(NamedTuple): init_args={}, connect_args={"cluster": "https://test.kusto.windows.net"}, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://test.kusto.windows.net", lambda driver: driver._current_config.cluster == "https://test.kusto.windows.net", ], @@ -230,8 +229,6 @@ class ConnectTest(NamedTuple): "tenant_id": "test_tenant_id", }, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://help.kusto.windows.net", lambda driver: driver._current_config.cluster == "https://help.kusto.windows.net", lambda driver: driver._az_tenant_id == "test_tenant_id", @@ -250,8 +247,6 @@ class ConnectTest(NamedTuple): "database": "test_db", }, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://help.kusto.windows.net", lambda driver: driver._current_config.cluster == "https://help.kusto.windows.net", lambda driver: driver._az_tenant_id == "test_tenant_id", @@ -324,8 +319,9 @@ class ConnectTest(NamedTuple): init_args={}, connect_args={"connection_str": "https://test.kusto.windows.net"}, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://test.kusto.windows.net", + lambda driver: driver.client._kusto_cluster.startswith( + "https://test.kusto.windows.net" + ), lambda driver: driver.client._proxy_url == "https://test.com", ], additional_config={ @@ -337,8 +333,9 @@ class ConnectTest(NamedTuple): init_args={}, connect_args={"cluster": "https://random.kusto.windows.net"}, tests=[ - lambda driver: driver.client._kusto_cluster - == "https://random.kusto.windows.net" + lambda driver: driver.client._kusto_cluster.startswith( + "https://random.kusto.windows.net" + ) ], ), ConnectTest( @@ -382,9 +379,9 @@ def test_kusto_connect(test_config, monkeypatch): driver.connect(**(test_config.connect_args)) return driver.connect(**(test_config.connect_args)) - for test in test_config.tests: + for idx, test in enumerate(test_config.tests): print( - "Testcase:", + f"Testcase [{idx}]:", test_config.name, test.__code__.co_filename, "line:", @@ -402,7 +399,7 @@ def test_kusto_connect(test_config, monkeypatch): exp_cluster = cluster_name else: exp_cluster = f"https://{cluster_name.casefold()}.kusto.windows.net" - check.equal(driver.client._kusto_cluster, exp_cluster) + check.is_true(driver.client._kusto_cluster.startswith(exp_cluster)) class KustoResponseDataSet: From 9c9936e93d1a00bc6bf030284498b07d66cb0dcb Mon Sep 17 00:00:00 2001 From: jannieli <32920319+jannieli@users.noreply.github.com> Date: Mon, 22 May 2023 12:12:12 -0700 Subject: [PATCH 10/26] Write Sentinel queries to YAML for Github Browser (#491) * add code to write yaml files * PR comment changes * replace requests with httpx * remove separate download * add docs and parameter types * remove unnecessary comment * add tests and headers * Remove old code file * rename files * remove unused imports * black reformatting * isort imports * pylint adjustments * update yaml loader for bandit checks * try with no prospector in requirements-dev.txt * test changes * fix pytest errors * test github pytest print * test pytest again * test pytest with file path changes * test if directories are being listed out in pytest * see what files are in the github runner * check directory contents again * list data available in testdata * see contents of test data folder * fix typo * try different base_dir_test_folder * check yaml_files * test _import_sentinel_query output * change split for _import_sentinel_query * test import_sentinel_query * see yaml file order * check sample query in list instead of equal * fix typo * sort lists for equivalence in organize test * try sort again * test with no datetime.now() in metadata * fix value unpacking error * switch set to list to fix issue * test getting_test_data * pylint fixes * check disabling certain pylint errors * try pylint again * fix mypy * additional linting fixes * fix flake8 trailing whitespace * add author and version * requested changes * fix linting issues * fix typerror * fix linting pt 3 * test mypy * test mypy pt 2 * try mypy pt 4 * mypy fix --------- Co-authored-by: Jannie Li Co-authored-by: Pete Bryan Co-authored-by: Ian Hellen --- .pre-commit-config.yaml | 2 +- msticpy/data/drivers/sentinel_query_reader.py | 465 ++++++++++++++++++ .../drivers/test_sentinel_query_reader.py | 214 ++++++++ .../Detections/Anomalies/UnusualAnomaly.yaml | 54 ++ .../Detections/ZoomLogs/E2EEDisbaled.yaml | 38 ++ .../ZoomLogs/ExternalUserAccess.yaml | 47 ++ .../JoiningMeetingFromAnotherTimeZone.yaml | 54 ++ .../ZoomLogs/SupiciousLinkSharing.yaml | 41 ++ .../Detections/readme.md | 18 + ...wnloadinvokedfromcmdline(ASIMVersion).yaml | 37 ++ .../imProcess_Certutil-LOLBins.yaml | 27 + .../Hunting Queries/QUERY_TEMPLATE.md | 50 ++ .../Hunting Queries/ZoomLogs/HighCPURoom.yaml | 36 ++ .../Hunting Queries/readme.md | 14 + 14 files changed, 1096 insertions(+), 1 deletion(-) create mode 100644 msticpy/data/drivers/sentinel_query_reader.py create mode 100644 tests/data/drivers/test_sentinel_query_reader.py create mode 100644 tests/testdata/sentinel_query_import_data/Detections/Anomalies/UnusualAnomaly.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/E2EEDisbaled.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/ExternalUserAccess.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/JoiningMeetingFromAnotherTimeZone.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/SupiciousLinkSharing.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Detections/readme.md create mode 100644 tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/Discorddownloadinvokedfromcmdline(ASIMVersion).yaml create mode 100644 tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/imProcess_Certutil-LOLBins.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Hunting Queries/QUERY_TEMPLATE.md create mode 100644 tests/testdata/sentinel_query_import_data/Hunting Queries/ZoomLogs/HighCPURoom.yaml create mode 100644 tests/testdata/sentinel_query_import_data/Hunting Queries/readme.md diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index db1d0bc35..14b21eda2 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -52,4 +52,4 @@ repos: entry: python -m tools.create_reqs_all pass_filenames: False language: python - types: [python] + types: [python] \ No newline at end of file diff --git a/msticpy/data/drivers/sentinel_query_reader.py b/msticpy/data/drivers/sentinel_query_reader.py new file mode 100644 index 000000000..446916bd1 --- /dev/null +++ b/msticpy/data/drivers/sentinel_query_reader.py @@ -0,0 +1,465 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Github Sentinel Query repo import class and helpers.""" + +import os +import re +import warnings +import zipfile +from datetime import datetime +from pathlib import Path +from typing import Optional +import logging + +import attr +import httpx +import yaml +from attr import attrs +from tqdm.notebook import tqdm +from ..._version import VERSION + +__version__ = VERSION +__author__ = "Jannie Li" + +# pylint: disable=too-many-instance-attributes +# pylint: disable=consider-using-with +# pylint: disable=too-many-locals +# pylint: disable=unspecified-encoding + + +QUERY_METADATA_SECTION = [ + "name", + "severity", + "tags", + "required_data_connectors", + "query_frequency", + "query_period", + "trigger_operator", + "trigger_threshold", + "tactics", + "relevant_techniques", + "entity_mappings", + "custom_details", + "alert_details_override", + "version", + "kind", + "folder_name", + "source_file_name", + "query_type", +] + + +QUERY_DEFAULT_PARAMETER_SECTION = { + "add_query_items": { + "description": "Additional query clauses", + "type": "str", + "default": "", + }, + "start": {"description": "Query start time", "type": "datetime"}, + "end": {"description": "Query end time", "type": "datetime"}, +} + + +@attrs(auto_attribs=True) +class SentinelQuery: + """Attrs class that represents a Sentinel Query yaml file.""" + + query_id: str = attr.Factory(str) + name: str = attr.Factory(str) + description: str = attr.Factory(str) + severity: str = attr.Factory(str) + query_frequency: str = attr.Factory(str) + query_period: str = attr.Factory(str) + trigger_operator: str = attr.Factory(str) + trigger_threshold: str = attr.Factory(str) + version: str = attr.Factory(str) + kind: str = attr.Factory(str) + folder_name: str = attr.Factory(str) + source_file_name: str = attr.Factory(str) + query_type: str = attr.Factory(str) + tactics: list = attr.Factory(list) + relevant_techniques: list = attr.Factory(list) + query: str = attr.Factory(str) + entity_mappings: dict = attr.Factory(dict) + custom_details: dict = attr.Factory(dict) + alert_details_override: dict = attr.Factory(dict) + tags: list = attr.Factory(list) + required_data_connectors: dict = attr.Factory(dict) + + +def get_sentinel_queries_from_github( + git_url: Optional[ + str + ] = "https://github.com/Azure/Azure-Sentinel/archive/master.zip", + outputdir: Optional[str] = None, +) -> bool: + r""" + Download Microsoft Sentinel Github archive and extract detection and hunting queries. + + Parameters + ---------- + git_url : str, optional + URL of the GIT Repository to be downloaded, by default + "https://github.com/Azure/Azure-Sentinel/archive/master.zip" + outputdir : str, optional + Provide absolute path to the output folder to save downloaded archive + (e.g. '/usr/home' or 'C:\downloads'). + If no path provided, it will download to .msticpy dir under Azure-Sentinel directory. + + """ + if outputdir is None: + outputdir = str( + Path.joinpath(Path("~").expanduser(), ".msticpy", "Azure-Sentinel") + ) + + try: + with httpx.stream("GET", git_url, follow_redirects=True) as response: # type: ignore + progress_bar = tqdm( + desc="Downloading from Microsoft Sentinel Github", + initial=0, + unit="iB", + unit_scale=True, + ) + repo_zip = Path.joinpath(Path(outputdir), "Azure-Sentinel.zip") # type: ignore + with open(repo_zip, "wb") as file: + for data in response.iter_bytes(chunk_size=10000): + progress_bar.update(len(data)) + file.write(data) + progress_bar.close() + + archive = zipfile.ZipFile(repo_zip, mode="r") + + # Only extract Detections and Hunting Queries Folder + for file in archive.namelist(): # type: ignore + if file.startswith( # type: ignore + ( + "Azure-Sentinel-master/Detections/", + "Azure-Sentinel-master/Hunting Queries/", + ) + ): + archive.extract(file, path=outputdir) # type: ignore + print("Downloaded and Extracted Files successfully") + return True + + except httpx.HTTPError as http_err: + warnings.warn(f"HTTP error occurred trying to download from Github: {http_err}") + return False + + +def read_yaml_files(parent_dir: str, child_dir: str) -> dict: + """ + Create dictionary mapping query file paths with the yaml file text each contains. + + Parameters + ---------- + parent_dir : str + Directory storing the Hunting and Detections directories + child_dir : str + Either "Hunting Queries" or "Detections" or otherwise named query category + + Returns + ------- + dict + Dictionary mapping query file paths to corresponding yaml file text in the + parent_dir/child_dir specified. Only identifies .yaml files. + + """ + # enumerate the files and read the yaml + yaml_queries = list(Path(parent_dir, child_dir).rglob("*.yaml")) + yaml_queries_list = [str(Path(q)) for q in yaml_queries] + parsed_query_dict = {} + + for query in yaml_queries_list: + with open(query, encoding="utf8", errors="ignore") as opened_query_file: + parsed_query_dict[query] = opened_query_file.read() + + return parsed_query_dict + + +def import_sentinel_queries(yaml_files: dict, query_type: str) -> list: + """ + Create list of SentinelQuery attr objects. + + Parameters + ---------- + yaml_files : dict + Dictionary mapping query file addresses to yaml file text created by read_yaml_files + query_type : str + Either "Hunting Queries" or "Detections" or otherwise named query category + + Returns + ------- + list + Returns a list of SentinelQuery attr objects from a dict of yaml files and query type given + + """ + return [ + _import_sentinel_query(yaml_path, yaml_text, query_type) + for yaml_path, yaml_text in yaml_files.items() + ] + + +def _import_sentinel_query( + yaml_path: str, yaml_text: str, query_type: str +) -> SentinelQuery: + """ + Create a SentinelQuery attr object for a given yaml query. + + Parameters + ---------- + yaml_path : str + File path to a YAML Sentinel query + yaml_text : str + YAML text (Sentinel query) associated with the yaml_path + query_type : str + Either "Hunting Queries" or "Detections" or otherwise named query category + + Returns + ------- + SentinelQuery + Returns an attrs object called SentinelQuery with all the YAML query information + + """ + logger = logging.getLogger(__name__) + try: + parsed_yaml_dict = yaml.load(yaml_text, Loader=yaml.SafeLoader) + except yaml.YAMLError as error: + logger.warning("Failed to parse yaml for %s", yaml_path) + logger.warning(error) + return SentinelQuery() + + new_query = SentinelQuery( + name=parsed_yaml_dict.get("name"), + query_id=parsed_yaml_dict.get("id", ""), + description=parsed_yaml_dict.get("description", ""), + severity=parsed_yaml_dict.get("severity", ""), + tags=parsed_yaml_dict.get("tags_entry", []), + required_data_connectors=parsed_yaml_dict.get("requiredDataConnectors", {}), + query_frequency=parsed_yaml_dict.get("queryFrequency", ""), + query_period=parsed_yaml_dict.get("queryPeriod", ""), + trigger_operator=parsed_yaml_dict.get("triggerOperator", ""), + trigger_threshold=parsed_yaml_dict.get("triggerThreshold", ""), + tactics=parsed_yaml_dict.get("tactics", ""), + relevant_techniques=parsed_yaml_dict.get("relevantTechniques", []), + query=parsed_yaml_dict.get("query", ""), + version=parsed_yaml_dict.get("version", ""), + kind=parsed_yaml_dict.get("kind", ""), + folder_name=yaml_path.replace("\\", "/").split("/")[-2], + source_file_name=yaml_path, + query_type=query_type, + ) + return new_query + + +def _format_query_name(qname: str) -> str: + """ + Format the inputted query name for use as a file name. + + Parameters + ---------- + qname : str + Name of a Sentinel Query as read from the corresponding YAML file + + Returns + ------- + str + Returns the inputted string with non-alphanumeric characters removed and + spaces replaced with an underscore + + """ + del_chars_pattern = r"""[%*\*'\"\\.()[\]{}-]""" + repl_str = re.sub(del_chars_pattern, "", qname) + repl_str = repl_str.replace(" ", "_").replace("__", "_") + return repl_str + + +def _organize_query_list_by_folder(query_list: list) -> dict: + """ + Create a dictionary mapping each folder name with related SentinelQuery objects. + + Parameters + ---------- + query_list : list + List of SentinelQuery objects returned by import_sentinel_queries() + + Returns + ------- + dict + Returns a dictionary mapping each folder name with the SentinelQuery objects associated + with it + + """ + queries_by_folder = {} + for query in query_list: + if query.folder_name == "": + warnings.warn(f"query {query} has no folder_name") + if query.folder_name not in queries_by_folder: + queries_by_folder[query.folder_name] = [query] + else: + queries_by_folder[query.folder_name].append(query) + + return queries_by_folder + + +def _create_queryfile_metadata(folder_name: str) -> dict: # type: ignore + """ + Generate metadata section of the YAML file for the given folder_name. + + Parameters + ---------- + folder_name : str + Either "Hunting Queries" or "Detections" or otherwise named query category + + Returns + ------- + dict + Returns a generated metadata section for the YAML files in the given folder_name + + """ + dict_to_write = {"metadata": {}, "defaults": {}, "sources": {}} # type: ignore + dict_to_write["metadata"]["version"] = 1 # write update version functionality + dict_to_write["metadata"]["description"] = "Sentinel Alert Queries - " + folder_name + dict_to_write["metadata"]["data_environments"] = [ + "MSSentinel" + ] # write how to get this and what type this should be + dict_to_write["metadata"]["data_families"] = folder_name + dict_to_write["metadata"]["last_updated"] = str(datetime.now()) + return dict_to_write + + +def _create_yaml_source_sec(cur_query: dict) -> dict: + """ + Create the metadata section of the YAML for the current query. + + Parameters + ---------- + cur_query : SentinelQuery + The current SentinelQuery attrs object that a metadata section will be generated for. + + Returns + ------- + dict + Returns generated metadata section of the YAML for the given individual query. + + """ + source_dict = {} + source_dict["description"] = cur_query["description"] + source_dict["metadata"] = {} + source_dict["metadata"]["sentinel"] = {"query_id": cur_query["query_id"]} + for section in QUERY_METADATA_SECTION: + source_dict["metadata"][section] = cur_query[section] + source_dict["metadata"]["args"] = {} + source_dict["metadata"]["args"]["query"] = cur_query["query"] + source_dict["metadata"]["parameters"] = {} + return source_dict + + +def write_to_yaml(query_list: list, query_type: str, output_folder: str) -> bool: + """ + Write out generated YAML files of the given query_list into the given output_folder. + + Parameters + ---------- + query_list : list + List of SentinelQuery attr objects generated by import_sentinel_queries() + query_type : str + Either "Hunting Queries" or "Detections" or otherwise named query category + output_folder : str + The name of the folder you want the written YAML files to be stored in + + Returns + ------- + bool + True if succeeded; False if an error occurred + + """ + logger = logging.getLogger(__name__) + query_dict_by_folder = _organize_query_list_by_folder(query_list) + all_folders = query_dict_by_folder.keys() + + for source_folder in all_folders: + logger.info("now writing files from %s", source_folder) + dict_to_write = _create_queryfile_metadata(source_folder) + + if query_type == "Detections": + dict_to_write["defaults"]["parameters"] = QUERY_DEFAULT_PARAMETER_SECTION + + for cur_query in query_dict_by_folder[source_folder]: + # skipping instances where there is no name but should probably have a better solution + try: + formatted_qname = _format_query_name(cur_query.name) + dict_to_write["sources"][formatted_qname] = _create_yaml_source_sec( + attr.asdict(cur_query) + ) + + except TypeError as err: + logger.warning( + """Query name is most likely None at %s for + current query %r""", + source_folder, + cur_query, + ) + print(err) + + try: + query_text = yaml.safe_dump( + dict_to_write, encoding="utf-8", sort_keys=False + ) + except yaml.YAMLError as error: + print(error) + return False + + try: + def_path = Path.joinpath(Path(os.getcwd())) + path_main = Path(def_path, output_folder) + path_type = Path(output_folder, query_type) + if not path_main.is_dir(): + path = Path(def_path, output_folder) + os.mkdir(path) + if not path_type.is_dir(): + path = Path(output_folder, query_type) + os.mkdir(path) + Path(output_folder, query_type, source_folder).write_text( + query_text.decode("utf-8") + ) + except OSError as error: + print(error) + return False + logger.info("done writing files") + return True + + +def download_and_write_sentinel_queries( + query_type: str, yaml_output_folder: str, github_outputdir: Optional[str] = None +): + """ + Download queries from GitHub and write out YAML files for the given query type. + + Parameters + ---------- + query_type : str + Either "Hunting Queries" or "Detections" or otherwise named query category + yaml_output_folder : str + Path to the folder you want the new generated YAML files to be stored in + github_outputdir : Optional[str] + Path to the directory you want the Github download to be stored in + """ + print("Downloading files from GitHub") + get_sentinel_queries_from_github(outputdir=github_outputdir) + print("Reading yaml_files") + if github_outputdir is None: + github_outputdir = str( + Path.joinpath(Path("~").expanduser(), ".msticpy", "Azure-Sentinel") + ) + base_dir = str(Path(github_outputdir, "/Azure-Sentinel-master")) + yaml_files = read_yaml_files(parent_dir=base_dir, child_dir=query_type) + print("Generating a list of queries") + query_list = import_sentinel_queries(yaml_files, query_type=query_type) + query_list = [ + query for query in query_list if query.query_id != "" + ] # may need better solution for failed query definitions + print("Writing to YAML output folder") + write_to_yaml(query_list, query_type, yaml_output_folder) diff --git a/tests/data/drivers/test_sentinel_query_reader.py b/tests/data/drivers/test_sentinel_query_reader.py new file mode 100644 index 000000000..ff16edcce --- /dev/null +++ b/tests/data/drivers/test_sentinel_query_reader.py @@ -0,0 +1,214 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Github Sentinel Query repo import class and helpers tests.""" + +import os +from datetime import datetime +from pathlib import Path +from unittest.mock import patch + +import pytest +import pytest_check as check + +from msticpy.data.drivers.sentinel_query_reader import ( + SentinelQuery, + _create_queryfile_metadata, + _format_query_name, + _import_sentinel_query, + _organize_query_list_by_folder, + get_sentinel_queries_from_github, + import_sentinel_queries, + read_yaml_files, + write_to_yaml, +) + +from ...unit_test_lib import get_test_data_path + +# variables used throughout tests +DEF_PATH = Path.joinpath(Path(os.getcwd())) +BASE_DIR = Path.joinpath(DEF_PATH, Path("Azure-Sentinel-master")) +BASE_DIR_TEST_FOLDER = Path.joinpath(get_test_data_path(), "sentinel_query_import_data") +_SENTINEL_QUERY_READER = "msticpy.data.drivers.sentinel_query_reader" + + +@patch(f"{_SENTINEL_QUERY_READER}.get_sentinel_queries_from_github") +def test_get_sentinel_queries_from_github(get_sentinel_queries_from_github): + get_sentinel_queries_from_github.return_value = True + assert get_sentinel_queries_from_github(outputdir=os.getcwd()) + get_sentinel_queries_from_github.assert_called_once() + + +# original test case for downloading github zip +@pytest.mark.skip(reason="requires downloading the file directly during the test") +def test_get_sentinel_queries_from_github(): + assert get_sentinel_queries_from_github(outputdir=os.getcwd()) + + +def test_read_yaml_files(): + yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + assert yaml_files[str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml"))] + + +def test__import_sentinel_query(): + yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + query_type = "Detections" + yaml_path = str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")) + yaml_text = yaml_files[yaml_path] + sample_query = SentinelQuery( + query_id="d0255b5f-2a3c-4112-8744-e6757af3283a", + name="Unusual Anomaly", + description="'Anomaly Rules generate events in the Anomalies table. This scheduled rule tries to detect Anomalies that are not usual, they could be a type of Anomaly that has recently been activated, or an infrequent type. The detected Anomaly should be reviewed, if it is relevant enough, eventually a separate scheduled Analytics Rule could be created specifically for that Anomaly Type, so an alert and/or incident is generated everytime that type of Anomaly happens.'\n", + severity="Medium", + tags=[], + required_data_connectors=[], + query_frequency="1h", + query_period="4d", + trigger_operator="gt", + trigger_threshold=0, + tactics=[], + relevant_techniques=[], + query='// You can leave out Anomalies that are already monitored through other Analytics Rules\n//let _MonitoredRules = dynamic(["TestAlertName"]);\nlet query_frequency = 1h;\nlet query_lookback = 3d;\nAnomalies\n| where TimeGenerated > ago(query_frequency)\n//| where not(RuleName has_any (_MonitoredRules))\n| join kind = leftanti (\n Anomalies\n | where TimeGenerated between (ago(query_frequency + query_lookback)..ago(query_frequency))\n | distinct RuleName\n) on RuleName\n', + entity_mappings={}, + custom_details={}, + alert_details_override={}, + version="1.0.1", + kind="Scheduled", + folder_name="Anomalies", + source_file_name=yaml_path, + query_type="Detections", + ) + assert _import_sentinel_query(yaml_path, yaml_text, query_type) == sample_query + + +def test_import_sentinel_query(): + yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + yaml_path = str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")) + sample_query = SentinelQuery( + query_id="d0255b5f-2a3c-4112-8744-e6757af3283a", + name="Unusual Anomaly", + description="'Anomaly Rules generate events in the Anomalies table. This scheduled rule tries to detect Anomalies that are not usual, they could be a type of Anomaly that has recently been activated, or an infrequent type. The detected Anomaly should be reviewed, if it is relevant enough, eventually a separate scheduled Analytics Rule could be created specifically for that Anomaly Type, so an alert and/or incident is generated everytime that type of Anomaly happens.'\n", + severity="Medium", + tags=[], + required_data_connectors=[], + query_frequency="1h", + query_period="4d", + trigger_operator="gt", + trigger_threshold=0, + tactics=[], + relevant_techniques=[], + query='// You can leave out Anomalies that are already monitored through other Analytics Rules\n//let _MonitoredRules = dynamic(["TestAlertName"]);\nlet query_frequency = 1h;\nlet query_lookback = 3d;\nAnomalies\n| where TimeGenerated > ago(query_frequency)\n//| where not(RuleName has_any (_MonitoredRules))\n| join kind = leftanti (\n Anomalies\n | where TimeGenerated between (ago(query_frequency + query_lookback)..ago(query_frequency))\n | distinct RuleName\n) on RuleName\n', + entity_mappings={}, + custom_details={}, + alert_details_override={}, + version="1.0.1", + kind="Scheduled", + folder_name="Anomalies", + source_file_name=yaml_path, + query_type="Detections", + ) + assert ( + sample_query in import_sentinel_queries(yaml_files, query_type="Detections") + ) + + +@pytest.mark.parametrize( + "initial_str, expected_result", + [ + ( + "Discord download invoked from cmd line (ASIM Version)", + "Discord_download_invoked_from_cmd_line_ASIM_Version", + ), + ("Unusual Anomaly", "Unusual_Anomaly"), + ( + "Dev-0056 Command Line Activity November 2021 (ASIM Version)", + "Dev0056_Command_Line_Activity_November_2021_ASIM_Version", + ), + ], +) +def test__format_query_name(initial_str, expected_result): + assert _format_query_name(initial_str) == expected_result + + +@pytest.mark.parametrize( + "dict_section, expected_result", + [ + ( + "keys", + [ + "Anomalies", + "ZoomLogs", + ], + ), + ( + "Anomalies", + [ + SentinelQuery( + query_id="d0255b5f-2a3c-4112-8744-e6757af3283a", + name="Unusual Anomaly", + description="'Anomaly Rules generate events in the Anomalies table. This scheduled rule tries to detect Anomalies that are not usual, they could be a type of Anomaly that has recently been activated, or an infrequent type. The detected Anomaly should be reviewed, if it is relevant enough, eventually a separate scheduled Analytics Rule could be created specifically for that Anomaly Type, so an alert and/or incident is generated everytime that type of Anomaly happens.'\n", + severity="Medium", + tags=[], + required_data_connectors=[], + query_frequency="1h", + query_period="4d", + trigger_operator="gt", + trigger_threshold=0, + tactics=[], + relevant_techniques=[], + query='// You can leave out Anomalies that are already monitored through other Analytics Rules\n//let _MonitoredRules = dynamic(["TestAlertName"]);\nlet query_frequency = 1h;\nlet query_lookback = 3d;\nAnomalies\n| where TimeGenerated > ago(query_frequency)\n//| where not(RuleName has_any (_MonitoredRules))\n| join kind = leftanti (\n Anomalies\n | where TimeGenerated between (ago(query_frequency + query_lookback)..ago(query_frequency))\n | distinct RuleName\n) on RuleName\n', + entity_mappings={}, + custom_details={}, + alert_details_override={}, + version="1.0.1", + kind="Scheduled", + folder_name="Anomalies", + source_file_name=str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")), + query_type="Detections", + ) + ], + ), + ], +) +def test__organize_query_list_by_folder(dict_section, expected_result): + yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + query_list = import_sentinel_queries(yaml_files=yaml_files, query_type="Detections") + if dict_section == "keys": + assert ( + sorted(list(_organize_query_list_by_folder(query_list=query_list).keys())) + == sorted(expected_result) + ) + else: + assert ( + sorted(_organize_query_list_by_folder(query_list=query_list)[dict_section]) + == sorted(expected_result) + ) + + + +def test__create_queryfile_metadata(): + ignore_keys = ['last_updated'] #timing may differ but doesn't matter for test purposes + generated_dict = {k:v for k,v in _create_queryfile_metadata(folder_name="Detections")["metadata"].items() if k not in ignore_keys} + test_dict = { + "version": 1, + "description": "Sentinel Alert Queries - Detections", + "data_environments": ["MSSentinel"], + "data_families": "Detections" + } + assert generated_dict == test_dict + + +# original test case for generating new yaml files +@pytest.mark.skip(reason="requires downloading the file directly during the test") +def test_write_to_yaml(): + yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + query_list = import_sentinel_queries(yaml_files=yaml_files, query_type="Detections") + query_list = [l for l in query_list if l is not None] + write_to_yaml( + query_list=query_list, + query_type="Detections", + output_folder="github_test_yamls", + ) + assert os.path.isdir(os.getcwd() + "/github_test_yamls") diff --git a/tests/testdata/sentinel_query_import_data/Detections/Anomalies/UnusualAnomaly.yaml b/tests/testdata/sentinel_query_import_data/Detections/Anomalies/UnusualAnomaly.yaml new file mode 100644 index 000000000..f22dfef93 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/Anomalies/UnusualAnomaly.yaml @@ -0,0 +1,54 @@ +id: d0255b5f-2a3c-4112-8744-e6757af3283a +name: Unusual Anomaly +description: | + 'Anomaly Rules generate events in the Anomalies table. This scheduled rule tries to detect Anomalies that are not usual, they could be a type of Anomaly that has recently been activated, or an infrequent type. The detected Anomaly should be reviewed, if it is relevant enough, eventually a separate scheduled Analytics Rule could be created specifically for that Anomaly Type, so an alert and/or incident is generated everytime that type of Anomaly happens.' +severity: Medium +requiredDataConnectors: [] +queryFrequency: 1h +queryPeriod: 4d +triggerOperator: gt +triggerThreshold: 0 +tactics: [] +techniques: [] +query: | + // You can leave out Anomalies that are already monitored through other Analytics Rules + //let _MonitoredRules = dynamic(["TestAlertName"]); + let query_frequency = 1h; + let query_lookback = 3d; + Anomalies + | where TimeGenerated > ago(query_frequency) + //| where not(RuleName has_any (_MonitoredRules)) + | join kind = leftanti ( + Anomalies + | where TimeGenerated between (ago(query_frequency + query_lookback)..ago(query_frequency)) + | distinct RuleName + ) on RuleName +incidentConfiguration: + createIncident: true + groupingConfiguration: + enabled: true + reopenClosedIncident: false + lookbackDuration: 1h + matchingMethod: Selected + groupByEntities: [] + groupByAlertDetails: + - DisplayName + groupByCustomDetails: [] +eventGroupingSettings: + aggregationKind: AlertPerResult +alertDetailsOverride: + alertDisplayNameFormat: Unusual Anomaly - {{RuleName}} + alertTacticsColumnName: Tactics +sentinelEntitiesMappings: + - columnName: Entities +version: 1.0.1 +kind: Scheduled +metadata: + source: + kind: Community + author: + name: Jose Sebastian Canos + support: + tier: Community + categories: + domains: [ "Security - Others" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/E2EEDisbaled.yaml b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/E2EEDisbaled.yaml new file mode 100644 index 000000000..3f546a4e6 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/E2EEDisbaled.yaml @@ -0,0 +1,38 @@ +id: e4779bdc-397a-4b71-be28-59e6a1e1d16b +name: Zoom E2E Encryption Disabled +description: | + 'This alerts when end to end encryption is disabled for Zoom meetings.' +severity: Medium +requiredDataConnectors: [] +queryFrequency: 1d +queryPeriod: 1d +triggerOperator: gt +triggerThreshold: 0 +tactics: + - CredentialAccess + - Discovery +relevantTechniques: + - T1040 +query: | + ZoomLogs + | where Event =~ "account.settings_updated" + | extend NewE2ESetting = columnifexists("payload_object_settings_in_meeting_e2e_encryption_b", "") + | extend OldE2ESetting = columnifexists("payload_old_object_settings_in_meeting_e2e_encryption_b", "") + | where OldE2ESetting =~ 'false' and NewE2ESetting =~ 'true' + | extend timestamp = TimeGenerated, AccountCustomEntity = User +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity +version: 1.0.2 +kind: Scheduled +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Others" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/ExternalUserAccess.yaml b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/ExternalUserAccess.yaml new file mode 100644 index 000000000..90a152924 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/ExternalUserAccess.yaml @@ -0,0 +1,47 @@ +id: 8e267e91-6bda-4b3c-bf68-9f5cbdd103a3 +name: External User Access Enabled +description: | + 'This alerts when the account setting is changed to allow either external domain access or anonymous access to meetings.' +severity: Low +requiredDataConnectors: [] +queryFrequency: 1d +queryPeriod: 1d +triggerOperator: gt +triggerThreshold: 0 +tactics: + - CredentialAccess + - Persistence +relevantTechniques: + - T1098 + - T1556 +query: | + ZoomLogs + | where Event =~ "account.settings_updated" + | extend EnforceLogin = columnifexists("payload_object_settings_schedule_meeting_enfore_login_b", "") + | extend EnforceLoginDomain = columnifexists("payload_object_settings_schedule_meeting_enfore_login_b", "") + | extend GuestAlerts = columnifexists("payload_object_settings_in_meeting_alert_guest_join_b", "") + | where EnforceLogin == 'false' or EnforceLoginDomain == 'false' or GuestAlerts == 'false' + | extend SettingChanged = case(EnforceLogin == 'false' and EnforceLoginDomain == 'false' and GuestAlerts == 'false', "All settings changed", + EnforceLogin == 'false' and EnforceLoginDomain == 'false', "Enforced Logons and Restricted Domains Changed", + EnforceLoginDomain == 'false' and GuestAlerts == 'false', "Enforced Domains Changed", + EnforceLoginDomain == 'false', "Enfored Domains Changed", + GuestAlerts == 'false', "Guest Join Alerts Changed", + EnforceLogin == 'false', "Enforced Logins Changed", + "No Changes") + | extend timestamp = TimeGenerated, AccountCustomEntity = User +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity +version: 1.0.3 +kind: Scheduled +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Others", "Identity" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/JoiningMeetingFromAnotherTimeZone.yaml b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/JoiningMeetingFromAnotherTimeZone.yaml new file mode 100644 index 000000000..7c9069d77 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/JoiningMeetingFromAnotherTimeZone.yaml @@ -0,0 +1,54 @@ +id: 58fc0170-0877-4ea8-a9ff-d805e361cfae +name: User joining Zoom meeting from suspicious timezone +description: | + 'The alert shows users that join a Zoom meeting from a time zone other than the one the meeting was created in. + You can also whitelist known good time zones in the tz_whitelist value using the tz database name format https://en.wikipedia.org/wiki/List_of_tz_database_time_zones' +severity: Low +requiredDataConnectors: [] +queryFrequency: 1d +queryPeriod: 14d +triggerOperator: gt +triggerThreshold: 0 +tactics: + - InitialAccess + - PrivilegeEscalation +relevantTechniques: + - T1078 +query: | + let schedule_lookback = 14d; + let join_lookback = 1d; + // If you want to whitelist specific timezones include them in a list here + let tz_whitelist = dynamic([]); + let meetings = ( + ZoomLogs + | where TimeGenerated >= ago(schedule_lookback) + | where Event =~ "meeting.created" + | extend MeetingId = tostring(parse_json(MeetingEvents).MeetingId) + | extend SchedTimezone = tostring(parse_json(MeetingEvents).Timezone)); + ZoomLogs + | where TimeGenerated >= ago(join_lookback) + | where Event =~ "meeting.participant_joined" + | extend JoinedTimeZone = tostring(parse_json(MeetingEvents).Timezone) + | extend MeetingName = tostring(parse_json(MeetingEvents).MeetingName) + | extend MeetingId = tostring(parse_json(MeetingEvents).MeetingId) + | where JoinedTimeZone !in (tz_whitelist) + | join (meetings) on MeetingId + | where SchedTimezone != JoinedTimeZone + | project TimeGenerated, MeetingName, JoiningUser=payload_object_participant_user_name_s, JoinedTimeZone, SchedTimezone, MeetingScheduler=User1 + | extend timestamp = TimeGenerated, AccountCustomEntity = JoiningUser +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity +version: 1.0.3 +kind: Scheduled +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Others" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/SupiciousLinkSharing.yaml b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/SupiciousLinkSharing.yaml new file mode 100644 index 000000000..cd7a1abe6 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/ZoomLogs/SupiciousLinkSharing.yaml @@ -0,0 +1,41 @@ +id: 1218175f-c534-421c-8070-5dcaabf28067 +name: Suspicious link sharing pattern +description: | + 'Alerts in links that have been shared across multiple Zoom chat channels by the same user in a short space if time. + Adjust the threshold figure to change the number of channels a message needs to be posted in before an alert is raised.' +severity: Low +requiredDataConnectors: [] +queryFrequency: 1d +queryPeriod: 1d +triggerOperator: gt +triggerThreshold: 0 +tactics: + - CredentialAccess + - Persistence +query: | + let threshold = 3; + ZoomLogs + | where Event =~ "chat_message.sent" + | extend Channel = tostring(parse_json(ChatEvents).Channel) + | extend Message = tostring(parse_json(ChatEvents).Message) + | where Message matches regex "http(s?):\\/\\/" + | summarize Channels = makeset(Channel), StartTime = min(TimeGenerated), EndTime = max(TimeGenerated) by Message, User, UserId + | extend ChannelCount = arraylength(Channels) + | where ChannelCount > threshold + | extend timestamp = StartTime, AccountCustomEntity = User +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity +version: 1.0.2 +kind: Scheduled +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Others" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Detections/readme.md b/tests/testdata/sentinel_query_import_data/Detections/readme.md new file mode 100644 index 000000000..f75856436 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Detections/readme.md @@ -0,0 +1,18 @@ +# About + +This folder contains Detections based on different types of data sources that you can leverage in order to create alerts and respond to threats in your environment. These detections are termed as Analytics Rule templates in Microsoft Sentinel. + +**Note**: Many of these analytic rule templates are being delivered in Solutions for Microsoft Sentinel. You can discover and deploy those in [Microsoft Sentinel Content Hub](https://docs.microsoft.com/azure/sentinel/sentinel-solutions-deploy). These are available in this repository under [Solutions](https://github.com/Azure/Azure-Sentinel/tree/master/Solutions) folder. For example, Analytic rules for the McAfee ePolicy Orchestrator solution are found [here](https://github.com/Azure/Azure-Sentinel/tree/master/Solutions/McAfee%20ePolicy%20Orchestrator/Analytic%20Rules). + +For general information please start with the [Wiki](https://github.com/Azure/Azure-Sentinel/wiki) pages. + +More Specific to Detections: +* [Contribute](https://github.com/Azure/Azure-Sentinel/wiki/Contribute-to-Sentinel-GitHub-Community-of-Queries) to Analytic Templates (Detections) and Hunting queries +* Specifics on what is required for Detections and Hunting queries is in the [Query Style Guide](https://github.com/Azure/Azure-Sentinel/wiki/Query-Style-Guide) +* These detections are written using [KQL query langauge](https://docs.microsoft.com/azure/kusto/query/index) and will provide you a starting point to protect your environment and get familiar with the different data tables. +* To enable these detections in your environment follow the [out of the box guidance](https://docs.microsoft.com/azure/sentinel/tutorial-detect-threats-built-in) (Notice that after a detection is available in this GitHub, it might take up to 2 weeks before it is available in Microsoft Sentinel portal). +* The rule created will run the query on the scheduled time that was defined, and trigger an alert that will be seen both in the **SecurityAlert** table and in a case in the **Incidents** tab +* If you are contributing analytic rule templates as part of a solution, follow [guidance for solutions](https://github.com/Azure/Azure-Sentinel/tree/master/Solutions#step-1--create-your-content) to include those in the right folder paths. Do NOT include content to be packaged in solutions under the Detections folder. + +# Feedback +For questions or feedback, please contact AzureSentinel@microsoft.com diff --git a/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/Discorddownloadinvokedfromcmdline(ASIMVersion).yaml b/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/Discorddownloadinvokedfromcmdline(ASIMVersion).yaml new file mode 100644 index 000000000..df29f3e20 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/Discorddownloadinvokedfromcmdline(ASIMVersion).yaml @@ -0,0 +1,37 @@ +id: 3169dc83-9e97-452c-afcc-baebdb0ddf7c +name: Discord download invoked from cmd line (ASIM Version) +description: | + 'This hunting query looks for hosts that have attempted to interact with the Discord CDN. This activity is not normally invoked from the command line and could indicate C2, exfiltration, or malware delivery activity.' +requiredDataConnectors: [] +tactics: + - Execution + - CommandAndControl + - Exfiltration +relevantTechniques: + - T1204 + - T1102 + - T1567 +query: | + imProcess + | where Process has_any ("powershell.exe", "powershell_ise.exe", "cmd.exe") or CommandLine has "powershell" + | where CommandLine has_any ("cdn.discordapp.com", "moc.ppadrocsid.ndc") + | project-reorder TimeGenerated, Computer, Account, Process, CommandLine +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: Account + - entityType: Host + fieldMappings: + - identifier: FullName + columnName: Computer +version: 1.0.0 +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Threat Protection" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/imProcess_Certutil-LOLBins.yaml b/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/imProcess_Certutil-LOLBins.yaml new file mode 100644 index 000000000..f0383f83c --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Hunting Queries/ASimProcess/imProcess_Certutil-LOLBins.yaml @@ -0,0 +1,27 @@ +id: 28233666-c235-4d55-b456-5cfdda29d62d +name: Certutil (LOLBins and LOLScripts, Normalized Process Events) +description: | + 'This detection uses Normalized Process Events to hunt Certutil activities' + +requiredDataConnectors: [] +tactics: + - CommandAndControl +relevantTechniques: + - T1105 + +query: | + imProcessCreate + | where Process has "certutil.exe" + // Uncomment the next line and add your commandLine Whitelisted/ignore terms.For example "urlcache" + // | where CommandLine !contains ("urlcache") + | extend HostCustomEntity = Dvc, AccountCustomEntity = User + +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity + - entityType: Host + fieldMappings: + - identifier: FullName + columnName: HostCustomEntity diff --git a/tests/testdata/sentinel_query_import_data/Hunting Queries/QUERY_TEMPLATE.md b/tests/testdata/sentinel_query_import_data/Hunting Queries/QUERY_TEMPLATE.md new file mode 100644 index 000000000..e4da985e0 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Hunting Queries/QUERY_TEMPLATE.md @@ -0,0 +1,50 @@ +# Hunting Query Template + +### Use a short name: + // Name: + +### Add a GUID so that we can add to the UI: + // Id: + +### A good description of what the query does, inputs and outputs: + // Description: + +### The datasource for the query (examples): + // DataSource: #SecurityEvent, #Syslog + +### The MITRE ATT&CK Techniques that apply to the query (examples): + // Tactics: #InitialAccess, #Execution, #Persistance + +### Example Query: + // Name: Cscript script daily summary breakdown + // + // Id: 36abe031-962d-482e-8e1e-a556ed99d5a3 + // + // Description: breakdown of scripts running in the environment + // + // DataSource: #SecurityEvent + // + // Tactics: #Execution + // + let ProcessCreationEvents=() { + let processEvents=SecurityEvent + | where EventID==4688 + | project EventTime=TimeGenerated, ComputerName=Computer,AccountName=SubjectUserName, AccountDomain=SubjectDomainName, + FileName=tostring(split(NewProcessName, '\\')[-1]), + ProcessCommandLine = CommandLine, + InitiatingProcessFileName=ParentProcessName,InitiatingProcessCommandLine="",InitiatingProcessParentFileName=""; + processEvents; + }; + // Daily summary of cscript activity - extracting script name and parameters from commandline: + ProcessCreationEvents | where FileName =~ "cscript.exe" + | project removeSwitches = replace(@"/+[a-zA-Z0-9:]+", "", ProcessCommandLine) // remove commandline switches + | project CommandLine = trim(@"[a-zA-Z0-9\\:""]*cscript(.exe)?("")?(\s)+", removeSwitches) // remove the leading cscript.exe process name + // extract the script name: + | project ScriptName= iff(CommandLine startswith @"""", + extract(@"([:\\a-zA-Z_\-\s0-9\.()]+)(""?)", 0, CommandLine), // handle case where script name is enclosed in " characters + extract(@"([:\\a-zA-Z_\-0-9\.()]+)(""?)", 0, CommandLine)) // handle case where script name is not enclosed in quotes + , CommandLine + | project ScriptName=trim(@"""", ScriptName) , ScriptNameLength=strlen(ScriptName), CommandLine + // extract remainder of commandline as script parameters: + | project ScriptName, ScriptParams = iff(ScriptNameLength < strlen(CommandLine), substring(CommandLine, ScriptNameLength +1), "") + | summarize by ScriptName, ScriptParams \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Hunting Queries/ZoomLogs/HighCPURoom.yaml b/tests/testdata/sentinel_query_import_data/Hunting Queries/ZoomLogs/HighCPURoom.yaml new file mode 100644 index 000000000..b0cd45a06 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Hunting Queries/ZoomLogs/HighCPURoom.yaml @@ -0,0 +1,36 @@ +id: 79cf4646-0959-442f-9707-60fc66eb8145 +name: Zoom room high CPU alerts +description: | + 'This hunting query identifies Zoom room systems with high CPU alerts that may be a sign of device compromise.' +requiredDataConnectors: [] +tactics: + - DefenseEvasion + - Persistence +relevantTechniques: + - T1542 +query: | + + ZoomLogs + | where Event =~ "zoomroom.alert" + | extend AlertType = toint(parse_json(RoomEvents).AlertType), AlertKind = toint(parse_json(RoomEvents).AlertKind) + | extend RoomName = payload_object_room_name_s, User = payload_object_email_s + | where AlertType == 1 and AlertKind == 1 + | extend timestamp = TimeGenerated, AccountCustomEntity = User + // Uncomment the lines below to analyse event over time + //| summarize count() by bin(TimeGenerated, 1h), RoomName + //| render timechart +entityMappings: + - entityType: Account + fieldMappings: + - identifier: FullName + columnName: AccountCustomEntity +version: 1.0.0 +metadata: + source: + kind: Community + author: + name: Pete Bryan + support: + tier: Community + categories: + domains: [ "Security - Other" ] \ No newline at end of file diff --git a/tests/testdata/sentinel_query_import_data/Hunting Queries/readme.md b/tests/testdata/sentinel_query_import_data/Hunting Queries/readme.md new file mode 100644 index 000000000..75252d917 --- /dev/null +++ b/tests/testdata/sentinel_query_import_data/Hunting Queries/readme.md @@ -0,0 +1,14 @@ +# About + +This folder contains Hunting Queries based on different types of data sources that you can leverage in order to perform broad threat hunting in your environment. + +For general information please start with the [Wiki](https://github.com/Azure/Azure-Sentinel/wiki) pages. + +More Specific to Hunting Queries: +* [Contribute](https://github.com/Azure/Azure-Sentinel/wiki/Contribute-to-Sentinel-GitHub-Community-of-Queries) to Analytic Templates (Detections) and Hunting queries +* Specifics on what is required for Detections and Hunting queries is in the [Query Style Guide](https://github.com/Azure/Azure-Sentinel/wiki/Query-Style-Guide) +* These hunting queries are written using [KQL query langauge](https://docs.microsoft.com/azure/kusto/query/index) and will provide you a starting point to protect your environment and get familiar with the different data tables. +* Get started and learn how to [hunt for threats in your environment with Microsoft Sentinel](https://docs.microsoft.com/azure/sentinel/hunting). + +# Feedback +For questions or feedback, please contact AzureSentinel@microsoft.com From 1f87529a5217af6c0c56d7364850394d557d4b69 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Fri, 26 May 2023 15:56:06 -0700 Subject: [PATCH 11/26] Update _version.py to 2.5.0 --- msticpy/_version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/msticpy/_version.py b/msticpy/_version.py index 5df18a634..bfd18279a 100644 --- a/msticpy/_version.py +++ b/msticpy/_version.py @@ -1,2 +1,2 @@ """Version file.""" -VERSION = "2.4.0" +VERSION = "2.5.0" From 9466a7723aa49f74175c538fb69ecd11b2e2689d Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Fri, 2 Jun 2023 15:35:41 -0700 Subject: [PATCH 12/26] Hotfix for v2.5.1 (#672) Add more defense against import errors - in msticpy.__init__.py - this causes failures when help(msticpy) is used, causing loading of all dynamic attributes Better exception message on import error in azure_data.py Moving ResourceGraph query provider to only instantiate the provider when needed. Made data_query_reader.py produce warnings rather throw exceptions when encountering a bad query file --- msticpy/__init__.py | 11 +-- msticpy/_version.py | 2 +- msticpy/context/azure/azure_data.py | 6 +- msticpy/context/azure/sentinel_workspaces.py | 25 ++++--- msticpy/data/core/data_query_reader.py | 20 +++--- msticpy/data/drivers/sentinel_query_reader.py | 3 +- .../context/azure/test_sentinel_workspaces.py | 4 +- .../drivers/test_sentinel_query_reader.py | 71 ++++++++++++------- tests/data/test_dataqueries.py | 2 +- 9 files changed, 91 insertions(+), 53 deletions(-) diff --git a/msticpy/__init__.py b/msticpy/__init__.py index 80ca3dae5..19f728470 100644 --- a/msticpy/__init__.py +++ b/msticpy/__init__.py @@ -116,6 +116,7 @@ """ import importlib import os +import warnings from typing import Any, Iterable, Union from . import nbwidgets @@ -124,6 +125,7 @@ from ._version import VERSION from .common import pkg_config as settings from .common.check_version import check_version +from .common.exceptions import MsticpyException from .common.utility import search_name as search from .init.logging import set_logging_level, setup_logging @@ -146,8 +148,6 @@ "GeoLiteLookup": "msticpy.context.geoip", "init_notebook": "msticpy.init.nbinit", "reset_ipython_exception_handler": "msticpy.init.nbinit", - "IPStackLookup": "msticpy.context.geoip", - "MicrosoftSentinel": "msticpy.context.azure", "MpConfigEdit": "msticpy.config.mp_config_edit", "MpConfigFile": "msticpy.config.mp_config_file", "QueryProvider": "msticpy.data", @@ -180,8 +180,11 @@ def __getattr__(attrib: str) -> Any: """ if attrib in _DEFAULT_IMPORTS: - module = importlib.import_module(_DEFAULT_IMPORTS[attrib]) - return getattr(module, attrib) + try: + return getattr(importlib.import_module(_DEFAULT_IMPORTS[attrib]), attrib) + except (ImportError, MsticpyException): + warnings.warn("Unable to import msticpy.{attrib}", ImportWarning) + return None raise AttributeError(f"msticpy has no attribute {attrib}") diff --git a/msticpy/_version.py b/msticpy/_version.py index bfd18279a..d8deaac44 100644 --- a/msticpy/_version.py +++ b/msticpy/_version.py @@ -1,2 +1,2 @@ """Version file.""" -VERSION = "2.5.0" +VERSION = "2.5.1" diff --git a/msticpy/context/azure/azure_data.py b/msticpy/context/azure/azure_data.py index 440d6f1ab..2ec12b700 100644 --- a/msticpy/context/azure/azure_data.py +++ b/msticpy/context/azure/azure_data.py @@ -43,7 +43,11 @@ from azure.mgmt.compute.models import VirtualMachineInstanceView except ImportError as imp_err: raise MsticpyImportExtraError( - "Cannot use this feature without azure packages installed", + "Cannot use this feature without these azure packages installed", + "azure.mgmt.network", + "azure.mgmt.resource", + "azure.mgmt.monitor", + "azure.mgmt.compute", title="Error importing azure module", extra="azure", ) from imp_err diff --git a/msticpy/context/azure/sentinel_workspaces.py b/msticpy/context/azure/sentinel_workspaces.py index 86ce3e787..70b6f2f8f 100644 --- a/msticpy/context/azure/sentinel_workspaces.py +++ b/msticpy/context/azure/sentinel_workspaces.py @@ -34,7 +34,7 @@ class SentinelWorkspacesMixin: """Mixin class for Sentinel workspaces.""" _TENANT_URI = "{cloud_endpoint}/{tenant_name}/.well-known/openid-configuration" - _RES_GRAPH_PROV = QueryProvider("ResourceGraph") + _RES_GRAPH_PROV: Optional[QueryProvider] = None @classmethod def get_resource_id_from_url(cls, portal_url: str) -> str: @@ -238,6 +238,14 @@ def get_workspace_settings_by_name( ) return {} + @classmethod + def _get_resource_graph_provider(cls) -> QueryProvider: + if not cls._RES_GRAPH_PROV: + cls._RES_GRAPH_PROV = QueryProvider("ResourceGraph") + if not cls._RES_GRAPH_PROV.connected: + cls._RES_GRAPH_PROV.connect() # pragma: no cover + return cls._RES_GRAPH_PROV + @classmethod def _lookup_workspace_by_name( cls, @@ -245,9 +253,8 @@ def _lookup_workspace_by_name( subscription_id: str = "", resource_group: str = "", ) -> pd.DataFrame: - if not cls._RES_GRAPH_PROV.connected: - cls._RES_GRAPH_PROV.connect() # pragma: no cover - return cls._RES_GRAPH_PROV.Sentinel.list_sentinel_workspaces_for_name( + res_graph_prov = cls._get_resource_graph_provider() + return res_graph_prov.Sentinel.list_sentinel_workspaces_for_name( workspace_name=workspace_name, subscription_id=subscription_id, resource_group=resource_group, @@ -255,17 +262,15 @@ def _lookup_workspace_by_name( @classmethod def _lookup_workspace_by_ws_id(cls, workspace_id: str) -> pd.DataFrame: - if not cls._RES_GRAPH_PROV.connected: - cls._RES_GRAPH_PROV.connect() # pragma: no cover - return cls._RES_GRAPH_PROV.Sentinel.get_sentinel_workspace_for_workspace_id( + res_graph_prov = cls._get_resource_graph_provider() + return res_graph_prov.Sentinel.get_sentinel_workspace_for_workspace_id( workspace_id=workspace_id ) @classmethod def _lookup_workspace_by_res_id(cls, resource_id: str): - if not cls._RES_GRAPH_PROV.connected: - cls._RES_GRAPH_PROV.connect() # pragma: no cover - return cls._RES_GRAPH_PROV.Sentinel.get_sentinel_workspace_for_resource_id( + res_graph_prov = cls._get_resource_graph_provider() + return res_graph_prov.Sentinel.get_sentinel_workspace_for_resource_id( resource_id=resource_id ) diff --git a/msticpy/data/core/data_query_reader.py b/msticpy/data/core/data_query_reader.py index 55d3b6837..d49420408 100644 --- a/msticpy/data/core/data_query_reader.py +++ b/msticpy/data/core/data_query_reader.py @@ -4,6 +4,7 @@ # license information. # -------------------------------------------------------------------------- """Data query definition reader.""" +import logging from itertools import chain from pathlib import Path from typing import Any, Dict, Iterable, Tuple @@ -11,11 +12,12 @@ import yaml from ..._version import VERSION -from .query_defns import DataEnvironment __version__ = VERSION __author__ = "Ian Hellen" +logger = logging.getLogger(__name__) + def find_yaml_files(source_path: str, recursive: bool = True) -> Iterable[Path]: """ @@ -69,12 +71,16 @@ def read_query_def_file(query_file: str) -> Tuple[Dict, Dict, Dict]: # use safe_load instead load data_map = yaml.safe_load(f_handle) - validate_query_defs(query_def_dict=data_map) + try: + validate_query_defs(query_def_dict=data_map) + except ValueError as err: + logger.warning("Validation failed for %s\n%s", query_file, err, exc_info=True) defaults = data_map.get("defaults", {}) sources = data_map.get("sources", {}) metadata = data_map.get("metadata", {}) + logger.info("Read %s queries from %s", len(sources), query_file) return sources, defaults, metadata @@ -99,6 +105,8 @@ def validate_query_defs(query_def_dict: Dict[str, Any]) -> bool: exception message (arg[0]) """ + if query_def_dict is None or not query_def_dict: + raise ValueError("Imported file is empty") # verify that sources and metadata are in the data dict if "sources" not in query_def_dict or not query_def_dict["sources"]: raise ValueError("Imported file has no sources defined") @@ -119,14 +127,6 @@ def _validate_data_categories(query_def_dict: Dict): ): raise ValueError("Imported file has no data_environments defined") - for env in query_def_dict["metadata"]["data_environments"]: - if not DataEnvironment.parse(env): - raise ValueError( - f"Unknown data environment {env} in metadata. ", - "Valid values are\n", - ", ".join(e.name for e in DataEnvironment), - ) - if ( "data_families" not in query_def_dict["metadata"] or not query_def_dict["metadata"]["data_families"] diff --git a/msticpy/data/drivers/sentinel_query_reader.py b/msticpy/data/drivers/sentinel_query_reader.py index 446916bd1..94bb95a96 100644 --- a/msticpy/data/drivers/sentinel_query_reader.py +++ b/msticpy/data/drivers/sentinel_query_reader.py @@ -5,6 +5,7 @@ # -------------------------------------------------------------------------- """Github Sentinel Query repo import class and helpers.""" +import logging import os import re import warnings @@ -12,13 +13,13 @@ from datetime import datetime from pathlib import Path from typing import Optional -import logging import attr import httpx import yaml from attr import attrs from tqdm.notebook import tqdm + from ..._version import VERSION __version__ = VERSION diff --git a/tests/context/azure/test_sentinel_workspaces.py b/tests/context/azure/test_sentinel_workspaces.py index 588daee2e..12443b58c 100644 --- a/tests/context/azure/test_sentinel_workspaces.py +++ b/tests/context/azure/test_sentinel_workspaces.py @@ -14,6 +14,7 @@ from msticpy.auth.azure_auth_core import AzureCloudConfig from msticpy.context.azure import MicrosoftSentinel +from msticpy.data import QueryProvider # pylint: disable=protected-access @@ -380,7 +381,8 @@ def test_param_checks(): def _patch_qry_prov(patcher): - qry_prov = getattr(MicrosoftSentinel, "_RES_GRAPH_PROV") + qry_prov = QueryProvider("ResourceGraph") + setattr(MicrosoftSentinel, "_RES_GRAPH_PROV", qry_prov) qry_prov._query_provider._loaded = True qry_prov._query_provider._connected = True patcher.setattr(qry_prov, "connect", lambda: True) diff --git a/tests/data/drivers/test_sentinel_query_reader.py b/tests/data/drivers/test_sentinel_query_reader.py index ff16edcce..c4cd30233 100644 --- a/tests/data/drivers/test_sentinel_query_reader.py +++ b/tests/data/drivers/test_sentinel_query_reader.py @@ -48,14 +48,22 @@ def test_get_sentinel_queries_from_github(): def test_read_yaml_files(): - yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") - assert yaml_files[str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml"))] + yaml_files = read_yaml_files( + parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections" + ) + assert yaml_files[ + str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")) + ] def test__import_sentinel_query(): - yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + yaml_files = read_yaml_files( + parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections" + ) query_type = "Detections" - yaml_path = str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")) + yaml_path = str( + BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml") + ) yaml_text = yaml_files[yaml_path] sample_query = SentinelQuery( query_id="d0255b5f-2a3c-4112-8744-e6757af3283a", @@ -84,8 +92,12 @@ def test__import_sentinel_query(): def test_import_sentinel_query(): - yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") - yaml_path = str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")) + yaml_files = read_yaml_files( + parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections" + ) + yaml_path = str( + BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml") + ) sample_query = SentinelQuery( query_id="d0255b5f-2a3c-4112-8744-e6757af3283a", name="Unusual Anomaly", @@ -109,9 +121,7 @@ def test_import_sentinel_query(): source_file_name=yaml_path, query_type="Detections", ) - assert ( - sample_query in import_sentinel_queries(yaml_files, query_type="Detections") - ) + assert sample_query in import_sentinel_queries(yaml_files, query_type="Detections") @pytest.mark.parametrize( @@ -165,7 +175,11 @@ def test__format_query_name(initial_str, expected_result): version="1.0.1", kind="Scheduled", folder_name="Anomalies", - source_file_name=str(BASE_DIR_TEST_FOLDER.joinpath("Detections/Anomalies/UnusualAnomaly.yaml")), + source_file_name=str( + BASE_DIR_TEST_FOLDER.joinpath( + "Detections/Anomalies/UnusualAnomaly.yaml" + ) + ), query_type="Detections", ) ], @@ -173,29 +187,36 @@ def test__format_query_name(initial_str, expected_result): ], ) def test__organize_query_list_by_folder(dict_section, expected_result): - yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + yaml_files = read_yaml_files( + parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections" + ) query_list = import_sentinel_queries(yaml_files=yaml_files, query_type="Detections") if dict_section == "keys": - assert ( - sorted(list(_organize_query_list_by_folder(query_list=query_list).keys())) - == sorted(expected_result) - ) + assert sorted( + list(_organize_query_list_by_folder(query_list=query_list).keys()) + ) == sorted(expected_result) else: - assert ( - sorted(_organize_query_list_by_folder(query_list=query_list)[dict_section]) - == sorted(expected_result) - ) - + assert sorted( + _organize_query_list_by_folder(query_list=query_list)[dict_section] + ) == sorted(expected_result) def test__create_queryfile_metadata(): - ignore_keys = ['last_updated'] #timing may differ but doesn't matter for test purposes - generated_dict = {k:v for k,v in _create_queryfile_metadata(folder_name="Detections")["metadata"].items() if k not in ignore_keys} + ignore_keys = [ + "last_updated" + ] # timing may differ but doesn't matter for test purposes + generated_dict = { + k: v + for k, v in _create_queryfile_metadata(folder_name="Detections")[ + "metadata" + ].items() + if k not in ignore_keys + } test_dict = { "version": 1, "description": "Sentinel Alert Queries - Detections", "data_environments": ["MSSentinel"], - "data_families": "Detections" + "data_families": "Detections", } assert generated_dict == test_dict @@ -203,7 +224,9 @@ def test__create_queryfile_metadata(): # original test case for generating new yaml files @pytest.mark.skip(reason="requires downloading the file directly during the test") def test_write_to_yaml(): - yaml_files = read_yaml_files(parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections") + yaml_files = read_yaml_files( + parent_dir=BASE_DIR_TEST_FOLDER, child_dir="Detections" + ) query_list = import_sentinel_queries(yaml_files=yaml_files, query_type="Detections") query_list = [l for l in query_list if l is not None] write_to_yaml( diff --git a/tests/data/test_dataqueries.py b/tests/data/test_dataqueries.py index 15223d736..df88c93c7 100644 --- a/tests/data/test_dataqueries.py +++ b/tests/data/test_dataqueries.py @@ -225,7 +225,7 @@ def test_graph_load_query_exec(self): def test_load_yaml_def(self): """Test query loader rejecting badly formed query files.""" la_provider = self.la_provider - with self.assertRaises((MsticpyException, ValueError)) as cm: + with self.assertRaises((MsticpyException, ValueError, KeyError)) as cm: file_path = Path(_TEST_DATA, "data_q_meta_fail.yaml") la_provider.import_query_file(query_file=file_path) self.assertIn("no data families defined", str(cm.exception)) From f4e2cb08a646e9200aa75c7f05b21b53592e86f3 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Tue, 13 Jun 2023 16:51:43 -0700 Subject: [PATCH 13/26] Ianhelle/hotfix 2.5.2 2023 06 08 (#676) * Changed Bokeh requirements to work with panel 1.x Moved code from nbinit that checked CLI credentials to be run only in AML - avoiding non-core import. * Adding back MicrosoftSentinel as mp attribute Making the attribute failure more robust/informative. * Adding documentation URL to ignored links "https://username:password@proxy_host:port" appears in docstrings and documentation and is being checked for URL validity. * Checked in test version of readthedocs conf.py by mistake - reverting --- conda/conda-reqs.txt | 2 +- msticpy/__init__.py | 28 ++++++++++++++++++---- msticpy/_version.py | 2 +- msticpy/init/azure_ml_tools.py | 40 ++++++++++++++++++++++++++++--- msticpy/init/nbinit.py | 32 ++----------------------- requirements-all.txt | 2 +- requirements.txt | 2 +- tests/ignored_uri_links.txt | 3 ++- tests/init/test_azure_ml_tools.py | 8 +++---- 9 files changed, 72 insertions(+), 47 deletions(-) diff --git a/conda/conda-reqs.txt b/conda/conda-reqs.txt index 14f8aa860..8335879db 100644 --- a/conda/conda-reqs.txt +++ b/conda/conda-reqs.txt @@ -12,7 +12,7 @@ azure-mgmt-resource>=16.1.0 azure-monitor-query>=1.0.0 azure-storage-blob>=12.5.0 beautifulsoup4>=4.0.0 -bokeh>=1.4.0, <=2.4.3 +bokeh>=1.4.0, <4.0.0 cryptography>=3.1 deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 diff --git a/msticpy/__init__.py b/msticpy/__init__.py index 19f728470..fa549965a 100644 --- a/msticpy/__init__.py +++ b/msticpy/__init__.py @@ -116,6 +116,7 @@ """ import importlib import os +import traceback import warnings from typing import Any, Iterable, Union @@ -125,7 +126,11 @@ from ._version import VERSION from .common import pkg_config as settings from .common.check_version import check_version -from .common.exceptions import MsticpyException +from .common.exceptions import ( + MsticpyException, + MsticpyImportExtraError, + MsticpyMissingDependencyError, +) from .common.utility import search_name as search from .init.logging import set_logging_level, setup_logging @@ -148,6 +153,7 @@ "GeoLiteLookup": "msticpy.context.geoip", "init_notebook": "msticpy.init.nbinit", "reset_ipython_exception_handler": "msticpy.init.nbinit", + "MicrosoftSentinel": "msticpy.context.azure", "MpConfigEdit": "msticpy.config.mp_config_edit", "MpConfigFile": "msticpy.config.mp_config_file", "QueryProvider": "msticpy.data", @@ -182,10 +188,22 @@ def __getattr__(attrib: str) -> Any: if attrib in _DEFAULT_IMPORTS: try: return getattr(importlib.import_module(_DEFAULT_IMPORTS[attrib]), attrib) - except (ImportError, MsticpyException): - warnings.warn("Unable to import msticpy.{attrib}", ImportWarning) - return None - raise AttributeError(f"msticpy has no attribute {attrib}") + except (MsticpyImportExtraError, MsticpyImportExtraError): + raise + except (ImportError, MsticpyException) as err: + warnings.warn(f"Unable to import module for 'msticpy.{attrib}'") + print( + f"WARNING. The msticpy attribute '{attrib}' is not loadable.", + "You may need to install one or more additional dependencies.\n", + "Please check the exception details below for more information.", + "\n".join( + traceback.format_exception( + type(err), value=err, tb=err.__traceback__ + ) + ), + ) + raise AttributeError(f"msticpy failed to load '{attrib}'") from err + raise AttributeError(f"msticpy has no attribute '{attrib}'") def __dir__(): diff --git a/msticpy/_version.py b/msticpy/_version.py index d8deaac44..b0d1760e1 100644 --- a/msticpy/_version.py +++ b/msticpy/_version.py @@ -1,2 +1,2 @@ """Version file.""" -VERSION = "2.5.1" +VERSION = "2.5.2" diff --git a/msticpy/init/azure_ml_tools.py b/msticpy/init/azure_ml_tools.py index cb15c2312..eedcc8f4e 100644 --- a/msticpy/init/azure_ml_tools.py +++ b/msticpy/init/azure_ml_tools.py @@ -22,7 +22,7 @@ from .._version import VERSION from ..common.pkg_config import _HOME_PATH, refresh_config -from ..common.utility import search_for_file +from ..common.utility import search_for_file, unit_testing from ..config import MpConfigFile # pylint: disable=no-name-in-module __version__ = VERSION @@ -55,6 +55,17 @@ Please restart the notebook kernel and re-run this cell - it should run without error. """ +_AZ_CLI_WIKI_URI = ( + "https://github.com/Azure/Azure-Sentinel-Notebooks/wiki/" + "Caching-credentials-with-Azure-CLI" +) +_CLI_WIKI_MSSG_GEN = ( + f"For more information see " + "Caching credentials with Azure CLI" +) +_CLI_WIKI_MSSG_SHORT = ( + f"see Caching credentials with Azure CLI" +) MIN_PYTHON_VER_DEF = "3.6" MSTICPY_REQ_VERSION = __version__ @@ -73,7 +84,7 @@ def is_in_aml(): return os.environ.get("APPSETTING_WEBSITE_SITE_NAME") == "AMLComputeInstance" -def check_versions( +def check_aml_settings( min_py_ver: Union[str, Tuple] = MIN_PYTHON_VER_DEF, min_mp_ver: Union[str, Tuple] = MSTICPY_REQ_VERSION, extras: Optional[List[str]] = None, @@ -104,7 +115,7 @@ def check_versions( """ del kwargs - _disp_html("

Starting notebook pre-checks...

") + _disp_html("

Starting AML notebook pre-checks...

") if isinstance(min_py_ver, str): min_py_ver = _get_pkg_version(min_py_ver).release check_python_ver(min_py_ver=min_py_ver) @@ -114,9 +125,14 @@ def check_versions( _set_kql_env_vars(extras) _run_user_settings() _set_mpconfig_var() + _check_azure_cli_status() _disp_html("

Notebook pre-checks complete.

") +# retain previous name for backward compatibility +check_versions = check_aml_settings + + def check_python_ver(min_py_ver: Union[str, Tuple] = MIN_PYTHON_VER_DEF): """ Check the current version of the Python kernel. @@ -447,3 +463,21 @@ def _check_kql_prereqs(): " Please run the following command:
" "!conda install --yes -c conda-forge pygobject
" ) + + +def _check_azure_cli_status(): + """Check for Azure CLI credentials.""" + # import these only if we need them at runtime + # pylint: disable=import-outside-toplevel + from ..auth.azure_auth_core import AzureCliStatus, check_cli_credentials + + if not unit_testing(): + status, message = check_cli_credentials() + if status == AzureCliStatus.CLI_OK: + _disp_html(message) + elif status == AzureCliStatus.CLI_NOT_INSTALLED: + _disp_html( + "Azure CLI credentials not detected." f" ({_CLI_WIKI_MSSG_SHORT})" + ) + elif message: + _disp_html("\n".join([message, _CLI_WIKI_MSSG_GEN])) diff --git a/msticpy/init/nbinit.py b/msticpy/init/nbinit.py index d3d0b767d..737d57d27 100644 --- a/msticpy/init/nbinit.py +++ b/msticpy/init/nbinit.py @@ -66,7 +66,6 @@ sns = None from .._version import VERSION -from ..auth.azure_auth_core import AzureCliStatus, check_cli_credentials from ..common.check_version import check_version from ..common.exceptions import MsticpyException, MsticpyUserError from ..common.pkg_config import _HOME_PATH @@ -84,8 +83,7 @@ search_for_file, unit_testing, ) -from .azure_ml_tools import check_versions as check_versions_aml -from .azure_ml_tools import is_in_aml, populate_config_to_mp_config +from .azure_ml_tools import check_aml_settings, is_in_aml, populate_config_to_mp_config from .azure_synapse_tools import init_synapse, is_in_synapse from .pivot import Pivot from .user_config import load_user_defaults @@ -226,17 +224,6 @@ def _verbose(verbosity: Optional[int] = None) -> int: "Please run the Getting Started Guide for Azure Sentinel " + "ML Notebooks notebook." ) -_AZ_CLI_WIKI_URI = ( - "https://github.com/Azure/Azure-Sentinel-Notebooks/wiki/" - "Caching-credentials-with-Azure-CLI" -) -_CLI_WIKI_MSSG_GEN = ( - f"For more information see " - "Caching credentials with Azure CLI" -) -_CLI_WIKI_MSSG_SHORT = ( - f"see Caching credentials with Azure CLI" -) current_providers: Dict[str, Any] = {} # pylint: disable=invalid-name @@ -426,7 +413,7 @@ def init_notebook( logger.info("Starting Notebook initialization") # Check Azure ML environment if _detect_env("aml", **kwargs) and is_in_aml(): - check_versions_aml(*_get_aml_globals(namespace)) + check_aml_settings(*_get_aml_globals(namespace)) else: # If not in AML check and print version status stdout_cap = io.StringIO() @@ -459,7 +446,6 @@ def init_notebook( else: _pr_output("Checking configuration....") conf_ok = _get_or_create_config() - _check_azure_cli_status() # Notebook options _pr_output("Setting notebook options....") @@ -897,17 +883,3 @@ def reset_ipython_exception_handler(): """Remove MSTICPy custom exception handler.""" if hasattr(InteractiveShell.showtraceback, "__wrapped__"): InteractiveShell.showtraceback = InteractiveShell.showtraceback.__wrapped__ - - -def _check_azure_cli_status(): - """Check for Azure CLI credentials.""" - if not unit_testing(): - status, message = check_cli_credentials() - if status == AzureCliStatus.CLI_OK: - _pr_output(message) - elif status == AzureCliStatus.CLI_NOT_INSTALLED: - _pr_output( - "Azure CLI credentials not detected." f" ({_CLI_WIKI_MSSG_SHORT})" - ) - elif message: - _pr_output("\n".join([message, _CLI_WIKI_MSSG_GEN])) diff --git a/requirements-all.txt b/requirements-all.txt index d35ac4ec4..0a99fe62f 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -15,7 +15,7 @@ azure-mgmt-subscription>=3.0.0 azure-monitor-query>=1.0.0 azure-storage-blob>=12.5.0 beautifulsoup4>=4.0.0 -bokeh>=1.4.0, <=2.4.3 +bokeh>=1.4.0, <4.0.0 cryptography>=3.1 deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 diff --git a/requirements.txt b/requirements.txt index 376b65ca7..df9066a3a 100644 --- a/requirements.txt +++ b/requirements.txt @@ -4,7 +4,7 @@ azure-core>=1.24.0 azure-identity>=1.10.0 azure-mgmt-subscription>=3.0.0 beautifulsoup4>=4.0.0 -bokeh>=1.4.0, <=2.4.3 +bokeh>=1.4.0, <4.0.0 cryptography>=3.1 deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 diff --git a/tests/ignored_uri_links.txt b/tests/ignored_uri_links.txt index b3d0232e7..5e08b627b 100644 --- a/tests/ignored_uri_links.txt +++ b/tests/ignored_uri_links.txt @@ -8,4 +8,5 @@ https://api.xforce.ibmcloud.com/url/fkksjobnn43.org https://api.greynoise.io/v3/community/38.75.137.9 https://management.azure.com/.default https://www.maxmind.com/en/accounts/current/license-key -https://api.us2.sumologic.com/api \ No newline at end of file +https://api.us2.sumologic.com/api +https://username:password@proxy_host:port diff --git a/tests/init/test_azure_ml_tools.py b/tests/init/test_azure_ml_tools.py index c94a7c6e3..765717dab 100644 --- a/tests/init/test_azure_ml_tools.py +++ b/tests/init/test_azure_ml_tools.py @@ -137,14 +137,14 @@ def test_check_versions(monkeypatch, aml_file_sys, check_vers): if check_vers.excep: with pytest.raises(check_vers.excep): with change_directory(str(user_dir)): - aml.check_versions( + aml.check_aml_settings( min_py_ver=check_vers.py_req, min_mp_ver=check_vers.mp_req, extras=check_vers.extras, ) else: with change_directory(str(user_dir)): - aml.check_versions( + aml.check_aml_settings( min_py_ver=check_vers.py_req, min_mp_ver=check_vers.mp_req, extras=check_vers.extras, @@ -189,7 +189,7 @@ def test_check_versions_mpconfig(monkeypatch, aml_file_sys, test_case): _os.environ["APPSETTING_WEBSITE_SITE_NAME"] = "AMLComputeInstance" with change_directory(str(target_dir)): - aml.check_versions(min_py_ver=_MIN_PY_VER, min_mp_ver=_MIN_MP_VER) + aml.check_aml_settings(min_py_ver=_MIN_PY_VER, min_mp_ver=_MIN_MP_VER) if test_case.sub_dir and test_case.mpconf_exists: env = "MSTICPYCONFIG" @@ -215,7 +215,7 @@ def test_check_versions_nbuser_settings(monkeypatch, aml_file_sys): nb_user_settings.write_text(_NBUSER_SETTINGS, encoding="utf-8") with change_directory(str(user_dir)): - aml.check_versions(min_py_ver=_MIN_PY_VER, min_mp_ver=_MIN_MP_VER) + aml.check_aml_settings(min_py_ver=_MIN_PY_VER, min_mp_ver=_MIN_MP_VER) check.is_in("nbuser_settings", sys.modules) nbus_import = sys.modules["nbuser_settings"] From ab444ca27efb07a5fb790b4224ca9770685fdd47 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Wed, 21 Jun 2023 16:06:32 -0700 Subject: [PATCH 14/26] Azure monitor endpoint URL has changed format in v1.2.0 (#677) * Azure monitor endpoint URL has changed format in v1.2.0 Unfortunately, older versions break with new format - so need a version-specific code branch. * Bug for missing attribute in kusto_driver - due to code change in kql_driver.py. Added documentation of need for additional packages to DataProv-Kusto-New.rst and DataProv-MSSentinel-New.rst * Changing ipwidgets requirement to <9.0.0 * Update DataProv-Kusto-New.rst Fixing name of `azure-kusto-data` --- conda/conda-reqs.txt | 2 +- docs/source/data_acquisition/DataProv-Kusto-New.rst | 8 +++++--- docs/source/data_acquisition/DataProv-MSSentinel-New.rst | 4 +++- msticpy/data/drivers/azure_monitor_driver.py | 9 ++++++++- msticpy/data/drivers/kql_driver.py | 4 +++- requirements-all.txt | 2 +- requirements.txt | 2 +- 7 files changed, 22 insertions(+), 9 deletions(-) diff --git a/conda/conda-reqs.txt b/conda/conda-reqs.txt index 8335879db..3cd11cc53 100644 --- a/conda/conda-reqs.txt +++ b/conda/conda-reqs.txt @@ -21,7 +21,7 @@ geoip2>=2.9.0 html5lib httpx==0.24.0 ipython>=7.23.1 -ipywidgets>=7.4.2, <8.0.0 +ipywidgets>=7.4.2, <9.0.0 keyring>=13.2.1 lxml>=4.6.5 matplotlib>=3.0.0 diff --git a/docs/source/data_acquisition/DataProv-Kusto-New.rst b/docs/source/data_acquisition/DataProv-Kusto-New.rst index 7712368e2..8974e82b7 100644 --- a/docs/source/data_acquisition/DataProv-Kusto-New.rst +++ b/docs/source/data_acquisition/DataProv-Kusto-New.rst @@ -3,14 +3,16 @@ Azure Data Explorer/Kusto Provider - New Implementation This is a new implementation of the Azure Data Explorer/Kusto QueryProvider using the -`azure-data-kusto SDK `__ +`azure-kusto-data SDK `__ (the earlier implementation used `Kqlmagic `__). .. warning:: This provider currently in beta and is available for testing. It is available alongside the existing Kusto provider for you - to compare old and new. + to compare old and new. To use it you will need the ``azure-kusto-data`` + package installed. You can install this with ``pip install azure-kusto-data`` + or ``pip install msticpy[azure_query]``. If you are using the existing implementation, see :doc:`./DataProv-Kusto` Changes from the previous implementation @@ -485,4 +487,4 @@ For examples of using the Kusto provider, see the samples `Kusto Analysis Notebook `__ and `Kusto Ingest Notebook `__ -:py:mod:`Kusto driver API documentation` \ No newline at end of file +:py:mod:`Kusto driver API documentation` diff --git a/docs/source/data_acquisition/DataProv-MSSentinel-New.rst b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst index c7db7a53e..c0417cb02 100644 --- a/docs/source/data_acquisition/DataProv-MSSentinel-New.rst +++ b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst @@ -9,7 +9,9 @@ the .. note:: This provider currently in beta and is available for testing. It is available alongside the existing Sentinel provider for you - to compare old and new. + to compare old and new. To use it you will need the ``azure-monitor-query`` + package installed. You can install this with ``pip install azure-monitor-query`` + or ``pip install msticpy[azure_query]``. If you are using the existing implementation, see :doc:`./DataProv-MSSentinel` Changes from the previous implementation diff --git a/msticpy/data/drivers/azure_monitor_driver.py b/msticpy/data/drivers/azure_monitor_driver.py index 9d9b4d620..b54281b7b 100644 --- a/msticpy/data/drivers/azure_monitor_driver.py +++ b/msticpy/data/drivers/azure_monitor_driver.py @@ -22,6 +22,7 @@ import pandas as pd from azure.core.exceptions import HttpResponseError from azure.core.pipeline.policies import UserAgentPolicy +from pkg_resources import parse_version from ..._version import VERSION from ...auth.azure_auth import AzureCloudConfig, az_connect @@ -41,12 +42,14 @@ logger = logging.getLogger(__name__) +# pylint: disable=ungrouped-imports try: from azure.monitor.query import ( LogsQueryClient, LogsQueryPartialResult, LogsQueryResult, ) + from azure.monitor.query import __version__ as az_monitor_version except ImportError as imp_err: raise MsticpyMissingDependencyError( "Cannot use this feature without Azure monitor client installed", @@ -149,9 +152,13 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): @property def url_endpoint(self) -> str: """Return the current URL endpoint for Azure Monitor.""" - return _LOGANALYTICS_URL_BY_CLOUD.get( + base_url = _LOGANALYTICS_URL_BY_CLOUD.get( AzureCloudConfig().cloud, _LOGANALYTICS_URL_BY_CLOUD["global"] ) + # post v1.1.0 of azure-monitor-query, the API version requires a 'v1' suffix + if parse_version(az_monitor_version) > parse_version("1.1.0"): + return f"{base_url}v1" + return base_url def connect(self, connection_str: Optional[str] = None, **kwargs): """ diff --git a/msticpy/data/drivers/kql_driver.py b/msticpy/data/drivers/kql_driver.py index f2dc09f9a..9362dd50f 100644 --- a/msticpy/data/drivers/kql_driver.py +++ b/msticpy/data/drivers/kql_driver.py @@ -117,7 +117,7 @@ def __init__(self, connection_str: str = None, **kwargs): self._ip = get_ipython() self._debug = kwargs.get("debug", False) super().__init__(**kwargs) - + self.workspace_id: Optional[str] = None self.set_driver_property( DriverProps.FORMATTERS, {"datetime": self._format_datetime, "list": self._format_list}, @@ -437,6 +437,8 @@ def _get_kql_current_connection(): """Get the current connection Workspace ID from KQLMagic.""" connections = kql_exec("--conn") current_connection = [conn for conn in connections if conn.startswith(" * ")] + if not current_connection: + return "" return current_connection[0].strip(" * ").split("@")[0] def _set_kql_cloud(self): diff --git a/requirements-all.txt b/requirements-all.txt index 0a99fe62f..079a84505 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -25,7 +25,7 @@ httpx==0.24.0 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" -ipywidgets>=7.4.2, <8.0.0 +ipywidgets>=7.4.2, <9.0.0 keyring>=13.2.1 KqlmagicCustom[jupyter-basic,auth_code_clipboard]>=0.1.114.post22 KqlmagicCustom[jupyter-extended]>=0.1.114.post22 diff --git a/requirements.txt b/requirements.txt index df9066a3a..107732a5f 100644 --- a/requirements.txt +++ b/requirements.txt @@ -14,7 +14,7 @@ httpx==0.24.0 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" -ipywidgets>=7.4.2, <8.0.0 +ipywidgets>=7.4.2, <9.0.0 KqlmagicCustom[jupyter-basic,auth_code_clipboard]>=0.1.114.post22 lxml>=4.6.5 msal>=1.12.0 From 55c6c1aebb8505a220046705b7c74194f83d62f3 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Wed, 21 Jun 2023 20:00:01 -0700 Subject: [PATCH 15/26] Update _version.py Updating version to 2.5.3 --- msticpy/_version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/msticpy/_version.py b/msticpy/_version.py index b0d1760e1..4344fd749 100644 --- a/msticpy/_version.py +++ b/msticpy/_version.py @@ -1,2 +1,2 @@ """Version file.""" -VERSION = "2.5.2" +VERSION = "2.5.3" From 931656841fb5988a689f28c83ff3bb019b293931 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Sat, 24 Jun 2023 14:50:41 -0700 Subject: [PATCH 16/26] Update python-publish.yml --- .github/workflows/python-publish.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/workflows/python-publish.yml b/.github/workflows/python-publish.yml index 7f8fd7568..51bf0581d 100644 --- a/.github/workflows/python-publish.yml +++ b/.github/workflows/python-publish.yml @@ -11,6 +11,7 @@ name: Upload Python Package to PyPI Prod on: release: types: [published] + workflow_dispatch: permissions: contents: read From 290808391fa4ee869b04795d7391d3ca5b70d61c Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Mon, 3 Jul 2023 13:16:50 -0700 Subject: [PATCH 17/26] Ianhelle/velociraptor provider 2023 05 19 (#668) * Adding Velociraptor provider for local logs * Format of cluster name has changed in new KustoClient. Fixing test cases to allow for old and new format. * Minor updates for DataProv-Velociraptor.rst * Fixing comments in PR. Fixed bug in azure_kusto_driver and test_azure_kusto_driver Fixed some doc references. * Adding acknowledgement of Blue Team Village data --- docs/source/DataAcquisition.rst | 1 + ...data.drivers.local_velociraptor_driver.rst | 7 + docs/source/api/msticpy.data.drivers.rst | 1 + docs/source/api/msticpy.init.mp_plugins.rst | 7 + .../data_acquisition/DataProv-OSQuery.rst | 4 +- .../DataProv-Velociraptor.rst | 167 +++++++++++++ .../source/data_acquisition/DataProviders.rst | 6 +- msticpy/data/core/query_defns.py | 4 + msticpy/data/drivers/__init__.py | 4 + msticpy/data/drivers/azure_kusto_driver.py | 15 +- .../data/drivers/local_velociraptor_driver.py | 226 ++++++++++++++++++ .../data/drivers/test_velociraptor_driver.py | 59 +++++ .../velociraptor/Windows.Forensics.Lnk.json | 10 + .../Windows.Forensics.ProcessInfo.json | 10 + .../velociraptor/Windows.Forensics.Usn.json | 10 + .../Windows.Memory.Acquisition.json | 10 + .../Windows.Network.ArpCache.json | 10 + .../Windows.Network.InterfaceAddresses.json | 5 + .../Windows.Network.ListeningPorts.json | 10 + .../velociraptor/Windows.Network.Netstat.json | 10 + .../velociraptor/Windows.Sys.Users.json | 7 + .../Windows.Sysinternals.Autoruns.json | 10 + .../velociraptor/Windows.System.DNSCache.json | 10 + .../velociraptor/Windows.System.Pslist.json | 10 + 24 files changed, 607 insertions(+), 6 deletions(-) create mode 100644 docs/source/api/msticpy.data.drivers.local_velociraptor_driver.rst create mode 100644 docs/source/api/msticpy.init.mp_plugins.rst create mode 100644 docs/source/data_acquisition/DataProv-Velociraptor.rst create mode 100644 msticpy/data/drivers/local_velociraptor_driver.py create mode 100644 tests/data/drivers/test_velociraptor_driver.py create mode 100644 tests/testdata/velociraptor/Windows.Forensics.Lnk.json create mode 100644 tests/testdata/velociraptor/Windows.Forensics.ProcessInfo.json create mode 100644 tests/testdata/velociraptor/Windows.Forensics.Usn.json create mode 100644 tests/testdata/velociraptor/Windows.Memory.Acquisition.json create mode 100644 tests/testdata/velociraptor/Windows.Network.ArpCache.json create mode 100644 tests/testdata/velociraptor/Windows.Network.InterfaceAddresses.json create mode 100644 tests/testdata/velociraptor/Windows.Network.ListeningPorts.json create mode 100644 tests/testdata/velociraptor/Windows.Network.Netstat.json create mode 100644 tests/testdata/velociraptor/Windows.Sys.Users.json create mode 100644 tests/testdata/velociraptor/Windows.Sysinternals.Autoruns.json create mode 100644 tests/testdata/velociraptor/Windows.System.DNSCache.json create mode 100644 tests/testdata/velociraptor/Windows.System.Pslist.json diff --git a/docs/source/DataAcquisition.rst b/docs/source/DataAcquisition.rst index 8837d0cfc..d46944771 100644 --- a/docs/source/DataAcquisition.rst +++ b/docs/source/DataAcquisition.rst @@ -29,6 +29,7 @@ Individual Data Environments data_acquisition/DataProv-Kusto-New data_acquisition/DataProv-Cybereason data_acquisition/DataProv-OSQuery + data_acquisition/DataProv-Velociraptor Built-in Data Queries diff --git a/docs/source/api/msticpy.data.drivers.local_velociraptor_driver.rst b/docs/source/api/msticpy.data.drivers.local_velociraptor_driver.rst new file mode 100644 index 000000000..af10e49e0 --- /dev/null +++ b/docs/source/api/msticpy.data.drivers.local_velociraptor_driver.rst @@ -0,0 +1,7 @@ +msticpy.data.drivers.local\_velociraptor\_driver module +======================================================= + +.. automodule:: msticpy.data.drivers.local_velociraptor_driver + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.data.drivers.rst b/docs/source/api/msticpy.data.drivers.rst index 4bf42fe41..d60239fd3 100644 --- a/docs/source/api/msticpy.data.drivers.rst +++ b/docs/source/api/msticpy.data.drivers.rst @@ -21,6 +21,7 @@ Submodules msticpy.data.drivers.kusto_driver msticpy.data.drivers.local_data_driver msticpy.data.drivers.local_osquery_driver + msticpy.data.drivers.local_velociraptor_driver msticpy.data.drivers.mdatp_driver msticpy.data.drivers.mordor_driver msticpy.data.drivers.odata_driver diff --git a/docs/source/api/msticpy.init.mp_plugins.rst b/docs/source/api/msticpy.init.mp_plugins.rst new file mode 100644 index 000000000..932af5ded --- /dev/null +++ b/docs/source/api/msticpy.init.mp_plugins.rst @@ -0,0 +1,7 @@ +msticpy.init.mp\_plugins module +=============================== + +.. automodule:: msticpy.init.mp_plugins + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/data_acquisition/DataProv-OSQuery.rst b/docs/source/data_acquisition/DataProv-OSQuery.rst index a91164503..4f17b0d79 100644 --- a/docs/source/data_acquisition/DataProv-OSQuery.rst +++ b/docs/source/data_acquisition/DataProv-OSQuery.rst @@ -7,7 +7,7 @@ The ``OSQuery`` data provider can read OSQuery log files and provide convenient query functions for each OSQuery "table" (or event type) contained in the logs. -The provide can read in one or more log files, or multiple log files +The provider can read in one or more log files, or multiple log files in multiple folders. The files are read, converted to pandas DataFrames and grouped by table/event. In addition, date fields within the data are converted to pandas Timestamp format. @@ -16,7 +16,7 @@ within the data are converted to pandas Timestamp format. qry_prov = mp.QueryProvider("OSQueryLogs", data_paths=["~/my_logs"]) qry_prov.connect() - df_processes = qry_prov.processes() + df_processes = qry_prov.os_query.processes() The query provider query functions will ignore parameters and do no further filtering. You can use pandas to do additional filtering diff --git a/docs/source/data_acquisition/DataProv-Velociraptor.rst b/docs/source/data_acquisition/DataProv-Velociraptor.rst new file mode 100644 index 000000000..3f084f085 --- /dev/null +++ b/docs/source/data_acquisition/DataProv-Velociraptor.rst @@ -0,0 +1,167 @@ +The Velociraptor provider +========================= + +:py:mod:`Velociraptor driver documentation` + +The ``Velociraptor`` data provider can read Velociraptor +offline collection log files (see +`Velociraptor Offline Collection `__) +and provide convenient query functions for each data set +in the output logs. + +The provider can read files from one or more hosts, stored in +in separate folders. The files are read, converted to pandas +DataFrames and grouped by table/event. Multiple log files of the +same type (when reading in data from multiple hosts) are concatenated +into a single DataFrame. + +.. code::ipython3 + + qry_prov = mp.QueryProvider("Velociraptor", data_paths=["~/my_logs"]) + qry_prov.connect() + df_processes = qry_prov.velociraptor.Windows_Forensics_ProcessInfo() + +The query provider query functions will ignore parameters and do +no further filtering. You can use pandas to do additional filtering +and sorting of the data, or use it directly with other MSTICPy +functionality. + +.. note:: The examples used in this document were from data + provided by Blue Team Village at Defcon 30. You can find + this data at the + `Project-Obsidian-DC30 GitHub `__ + and more about + `Project Obsidian `__ + here. + +Velociraptor Configuration +-------------------------- + +You can (optionally) store your connection details in *msticpyconfig.yaml*, +instead of supplying the ``data_paths`` parameter to +the ``QueryProvider`` class. + +For more information on using and configuring *msticpyconfig.yaml* see +:doc:`msticpy Package Configuration <../getting_started/msticpyconfig>` +and :doc:`MSTICPy Settings Editor<../getting_started/SettingsEditor>` + +The Velociraptor settings in the file should look like the following: + +.. code:: yaml + + DataProviders: + ... + Velociraptor: + data_paths: + - /home/user1/sample_data + - /home/shared/sample_data + + +Expected log file format +------------------------ + +The log file format must be a text file of JSON records. An example +is shown below + +.. parsed-literal:: + + {"Pid":1664,"Ppid":540,"Name":"spoolsv.exe","Path":"C:\\Windows\\System32\\spoolsv.exe","CommandLine":"C:\\Windows\\System32\\spoolsv.exe","Hash":{"MD5":"c111e3d38c71808a8289b0e49db40c96","SHA1":"e56df979d776fe9e8c3b84e6fef8559d6811898d","SHA256":"0ed0c6f4ddc620039f05719d783585d69f03d950be97b49149d4addf23609902"},"Username":"NT AUTHORITY\\SYSTEM","Authenticode":{"Filename":"C:\\Windows\\System32\\spoolsv.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002ed2c45e4c145cf48440000000002ed","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows","Timestamp":null,"Trusted":"trusted","_ExtraInfo":{"Catalog":"C:\\Windows\\system32\\CatRoot\\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\\Package_6350_for_KB5007192~31bf3856ad364e35~amd64~~10.0.1.8.cat"}},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49697,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"} + {"Pid":548,"Ppid":416,"Name":"lsass.exe","Path":"C:\\Windows\\System32\\lsass.exe","CommandLine":"C:\\Windows\\system32\\lsass.exe","Hash":{"MD5":"93212fd52a9cd5addad2fd2a779355d2","SHA1":"49a814f72292082a1cfdf602b5e4689b0f942703","SHA256":"95888daefd187fac9c979387f75ff3628548e7ddf5d70ad489cf996b9cad7193"},"Username":"NT AUTHORITY\\SYSTEM","Authenticode":{"Filename":"C:\\Windows\\System32\\lsass.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002f49e469c54137b85e00000000002f4","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49722,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:54Z"} + {"Pid":540,"Ppid":416,"Name":"services.exe","Path":"C:\\Windows\\System32\\services.exe","CommandLine":"C:\\Windows\\system32\\services.exe","Hash":{"MD5":"fefc26105685c70d7260170489b5b520","SHA1":"d9b2cb9bf9d4789636b5fcdef0fdbb9d8bc0fb52","SHA256":"930f44f9a599937bdb23cf0c7ea4d158991b837d2a0975c15686cdd4198808e8"},"Username":"NT AUTHORITY\\SYSTEM","Authenticode":{"Filename":"C:\\Windows\\System32\\services.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002a5e1a081b7c895c0ed0000000002a5","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49728,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:57Z"} + + +The columns in each JSON will be used to create the pandas DataFrame columns. + + +Using the Velociraptor provider +------------------------------- + +To use the Velociraptor provider you need to create an QueryProvider +instance, passing the string "VelociraptorLogs" as the ``data_environment`` +parameter. If you have not configured ``data_paths`` in msticpyconfig.yaml, +you also need to add the ``data_paths`` parameter to specify +specific folders or files that you want to read. + +.. code::ipython3 + + qry_prov = mp.QueryProvider("VelociraptorLogs", data_paths=["~/my_logs"]) + +Calling the ``connect`` method triggers the provider to register the paths of the +log files to be read (although the log files are not read and parsed +until the related query is run - see below). + +.. code::ipython3 + + qry_prov.connect() + + + +Listing Velociraptor tables +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Until you run ``connect`` no queries will be available. After running +``connect`` you can list the available queries using the ``list_queries`` + +.. code:: ipython3 + + qry_prov.list_queries() + +.. parsed-literal:: + + ['velociraptor.Custom_Windows_NetBIOS', + 'velociraptor.Custom_Windows_Patches', + 'velociraptor.Custom_Windows_Sysinternals_PSInfo', + 'velociraptor.Custom_Windows_Sysinternals_PSLoggedOn', + 'velociraptor.Custom_Windows_System_Services', + 'velociraptor.Windows_Applications_Chrome_Cookies', + 'velociraptor.Windows_Applications_Chrome_Extensions', + 'velociraptor.Windows_Applications_Chrome_History', + 'velociraptor.Windows_Applications_Edge_History', + 'velociraptor.Windows_Forensics_Lnk', + 'velociraptor.Windows_Forensics_Prefetch', + 'velociraptor.Windows_Forensics_ProcessInfo', + 'velociraptor.Windows_Forensics_Usn', + ...] + +Querying Velociraptor table schema +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The schema of the log tables is built by sampling the first record +from each log file type, so is relatively fast to retrieve even +if you have large numbers and sizes of logs. + +.. code:: ipython3 + + vc_prov.schema["Windows_Network_InterfaceAddresses"] + +.. parsed-literal:: + + {'Index': 'int64', + 'MTU': 'int64', + 'Name': 'object', + 'HardwareAddr': 'object', + 'Flags': 'int64', + 'IP': 'object', + 'Mask': 'object'} + +Running a Velociraptor query +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Each query returns a pandas DataFrame retrieved +from the logs of that type (potentially containing records from +multiple hosts depending on the ``data_paths`` you specified). + +.. code:: python3 + + qry_prov.vc_prov.velociraptor.Windows_Forensics_ProcessInfo() + + +==== =========== ================ ===== =============================== ================================================================ ==================== =================================== + .. Name PebBaseAddress Pid ImagePathName CommandLine CurrentDirectory Env +==== =========== ================ ===== =============================== ================================================================ ==================== =================================== + 10 LogonUI.exe 0x95bd3d2000 804 C:\Windows\system32\LogonUI.exe "LogonUI.exe" /flags:0x2 /state0:0xa3b92855 /state1:0x41c64e6d C:\Windows\system32\ {'ALLUSERSPROFILE': 'C:\\ProgramD.. + 11 dwm.exe 0x6cf4351000 848 C:\Windows\system32\dwm.exe "dwm.exe" C:\Windows\system32\ {'ALLUSERSPROFILE': 'C:\\ProgramD.. + 12 svchost.exe 0x6cd64d000 872 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k termsvcs C:\Windows\system32\ {'ALLUSERSPROFILE': 'C:\\ProgramD.. + 13 svchost.exe 0x7d18e99000 912 C:\Windows\System32\svchost.exe C:\Windows\System32\svchost.exe -k LocalServiceNetworkRestricted C:\Windows\system32\ {'ALLUSERSPROFILE': 'C:\\ProgramD.. + 14 svchost.exe 0x5c762eb000 920 C:\Windows\system32\svchost.exe C:\Windows\system32\svchost.exe -k LocalService C:\Windows\system32\ {'ALLUSERSPROFILE': 'C:\\ProgramD.. +==== =========== ================ ===== =============================== ================================================================ ==================== =================================== diff --git a/docs/source/data_acquisition/DataProviders.rst b/docs/source/data_acquisition/DataProviders.rst index c1f07a903..ddd229f6c 100644 --- a/docs/source/data_acquisition/DataProviders.rst +++ b/docs/source/data_acquisition/DataProviders.rst @@ -85,7 +85,7 @@ would only use this parameter if you were building your own data driver backend, which is not common. 2. You can choose to import additional queries from a custom -query directory (see `Creating new queries`_ for more +query directory (see :doc:`../extending/Queries` for more details) with: .. code:: ipython3 @@ -494,7 +494,7 @@ for Timedelta in the .. warning:: There are some important caveats to this feature. 1. It currently only works with pre-defined queries (including ones - that you may create and add yourself, see `Creating new queries`_ + that you may create and add yourself, see :doc:`../extending/Queries` below). It does not work with `Running an ad hoc query`_ 2. If the query contains joins, the joins will only happen within the time ranges of each subquery. @@ -512,7 +512,7 @@ Dynamically adding new queries You can use the :py:meth:`msticpy.data.core.data_providers.QueryProvider.add_query` to add parameterized queries from a notebook or script. This let you use temporary parameterized queries without having to -add them to a YAML file (as described in `Creating new queries`_). +add them to a YAML file (as described in :doc:`../extending/Queries`). get_host_events diff --git a/msticpy/data/core/query_defns.py b/msticpy/data/core/query_defns.py index a42626e7d..87e16c949 100644 --- a/msticpy/data/core/query_defns.py +++ b/msticpy/data/core/query_defns.py @@ -91,6 +91,7 @@ class DataEnvironment(Enum): AzureSentinel = 1 # alias of LogAnalytics LogAnalytics = 1 Kusto = 2 + AzureDataExplorer = 2 # alias of Kusto AzureSecurityCenter = 3 MSGraph = 4 SecurityGraph = 4 @@ -106,8 +107,11 @@ class DataEnvironment(Enum): Cybereason = 12 Elastic = 14 OSQueryLogs = 15 + OSQuery = 15 MSSentinel_New = 16 Kusto_New = 17 + VelociraptorLogs = 18 + Velociraptor = 18 @classmethod def parse(cls, value: Union[str, int]) -> "DataEnvironment": diff --git a/msticpy/data/drivers/__init__.py b/msticpy/data/drivers/__init__.py index 3115ab0a5..6e934677f 100644 --- a/msticpy/data/drivers/__init__.py +++ b/msticpy/data/drivers/__init__.py @@ -34,6 +34,10 @@ DataEnvironment.Elastic: ("elastic_driver", "ElasticDriver"), DataEnvironment.MSSentinel_New: ("azure_monitor_driver", "AzureMonitorDriver"), DataEnvironment.Kusto_New: ("azure_kusto_driver", "AzureKustoDriver"), + DataEnvironment.VelociraptorLogs: ( + "local_velociraptor_driver", + "VelociraptorLogDriver", + ), } CUSTOM_PROVIDERS: Dict[str, type] = {} diff --git a/msticpy/data/drivers/azure_kusto_driver.py b/msticpy/data/drivers/azure_kusto_driver.py index e67ddccda..815787a69 100644 --- a/msticpy/data/drivers/azure_kusto_driver.py +++ b/msticpy/data/drivers/azure_kusto_driver.py @@ -162,7 +162,7 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): self._strict_query_match = kwargs.get("strict_query_match", False) self._kusto_settings: Dict[str, Dict[str, KustoConfig]] = _get_kusto_settings() self._default_database: Optional[str] = None - self.current_connection: Optional[str] = connection_str + self._current_connection: Optional[str] = connection_str self._current_config: Optional[KustoConfig] = None self.client: Optional[KustoClient] = None self._az_auth_types: Optional[List[str]] = None @@ -189,6 +189,18 @@ def _set_public_attribs(self): "set_database": self.set_database, } + @property + def current_connection(self) -> Optional[str]: + """Return current connection string or URI.""" + if self._current_connection: + return self._current_connection + return self.cluster_uri + + @current_connection.setter + def current_connection(self, value: str): + """Set current connection string or URI.""" + self._current_connection = value + @property def cluster_uri(self) -> str: """Return current cluster URI.""" @@ -318,6 +330,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): kusto_cs = self._get_connection_string_for_cluster(self._current_config) else: logger.info("Using connection string %s", connection_str) + self._current_connection = connection_str kusto_cs = connection_str self.client = KustoClient(kusto_cs) diff --git a/msticpy/data/drivers/local_velociraptor_driver.py b/msticpy/data/drivers/local_velociraptor_driver.py new file mode 100644 index 000000000..a8e8af90f --- /dev/null +++ b/msticpy/data/drivers/local_velociraptor_driver.py @@ -0,0 +1,226 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Local Velociraptor Data Driver class.""" +import logging +from collections import defaultdict +from functools import lru_cache +from pathlib import Path +from typing import Any, Dict, List, Optional, Union + +import pandas as pd +from tqdm.auto import tqdm + +from ..._version import VERSION +from ...common.exceptions import MsticpyDataQueryError +from ...common.provider_settings import get_provider_settings +from ...common.utility import export, valid_pyname +from .driver_base import DriverBase, QuerySource + +__version__ = VERSION +__author__ = "ianhelle" + +logger = logging.getLogger(__name__) + +_VELOCIRATOR_DOC_URL = ( + "https://msticpy.readthedocs.io/en/latest/data_acquisition/" + "DataProv-Velociraptor.html" +) + + +# pylint: disable=too-many-instance-attributes +@export +class VelociraptorLogDriver(DriverBase): + """Velociraptor driver class to ingest log data.""" + + def __init__(self, connection_str: Optional[str] = None, **kwargs): + """ + Instantiate Velociraptor driver and optionally connect. + + Parameters + ---------- + connection_str : str, optional + Connection string (not used) + data_paths : List[str], optional + Paths from which to load data files + progress : bool, optional + Show progress with tqdm, by default, True + + """ + del connection_str + if kwargs.get("debug", False): + logger.setLevel(logging.DEBUG) + super().__init__() + + self._paths: List[str] = ["."] + # If data paths specified, use these + # from kwargs or settings + if data_paths := kwargs.get("data_paths"): + self._paths = [path.strip() for path in data_paths] + logger.info("data paths read from param %s", str(self._paths)) + else: + prov_settings = get_provider_settings(config_section="DataProviders").get( + "VelociraptorLogs" + ) + if prov_settings: + self._paths = prov_settings.args.get("data_paths", []) or self._paths + logger.info("data paths read from config %s", str(self._paths)) + + self.data_files: Dict[str, List[Path]] = {} + self._schema: Dict[str, Any] = {} + self._query_map: Dict[str, List[str]] + self._progress = kwargs.pop("progress", True) + self._loaded = True + self.has_driver_queries = True + logger.info("data files to read %s", ",".join(self.data_files)) + + def connect(self, connection_str: Optional[str] = None, **kwargs): + """ + Connect to data source. + + Parameters + ---------- + connection_str : str + Connect to a data source + + """ + del connection_str + self.data_files = self._get_logfile_paths() + self._connected = True + + @property + def schema(self) -> Dict[str, Dict]: + """ + Return current data schema of connection. + + Returns + ------- + Dict[str, Dict] + Data schema of current connection. + + """ + if not self._schema: + if not self.data_files: + self.connect() + # read the first row of each file to get the schema + iter_data_files = ( + tqdm(self.data_files.items()) + if self._progress + else self.data_files.items() + ) + for table, files in iter_data_files: + if not files: + continue + sample_df = pd.read_json(files[0], lines=True, nrows=1) + self._schema[table] = { + col: dtype.name for col, dtype in sample_df.dtypes.to_dict().items() + } + logger.info("Reading schema for %d tables", len(self.data_files)) + + return self._schema + + def query( + self, query: str, query_source: Optional[QuerySource] = None, **kwargs + ) -> Union[pd.DataFrame, Any]: + """ + Execute query string and return DataFrame of results. + + Parameters + ---------- + query : str + The query to execute + query_source : QuerySource + The query definition object + + Returns + ------- + Union[pd.DataFrame, results.ResultSet] + A DataFrame (if successful) or + the underlying provider result if an error. + + """ + del kwargs + if not self.data_files: + self.connect() + + if query not in self.data_files: + raise MsticpyDataQueryError( + f"No data loaded for query {query}.", + "Please check that the data loaded from the log paths", + "has this data type.", + ) + return self._cached_query(query) + + @lru_cache(maxsize=256) + def _cached_query(self, query: str) -> pd.DataFrame: + iter_data_files = ( + tqdm(self.data_files[query]) if self._progress else self.data_files[query] + ) + dfs = [pd.read_json(file, lines=True) for file in iter_data_files] + query_df = pd.concat(dfs, ignore_index=True) + + logger.info("Query %s, returned %d rows", query, len(query_df)) + return query_df + + def query_with_results(self, query, **kwargs): + """Return query with fake results.""" + return self.query(query, **kwargs), "OK" + + @property + def driver_queries(self) -> List[Dict[str, Any]]: + """ + Return dynamic queries available on connection to data. + + Returns + ------- + List[Dict[str, Any]] + List of queries with properties: "name", "query", "container" + and (optionally) "description" + + Raises + ------ + MsticpyNotConnectedError + If called before driver is connected. + + """ + if not self.connected: + self.connect() + if self.data_files: + return [ + { + "name": query, + "query": query, + "query_paths": "velociraptor", + "description": f"Velociraptor {query} table.", + } + for query in self.data_files + ] + return [] + + def _get_logfile_paths(self) -> Dict[str, List[Path]]: + """Read files in data paths.""" + data_files: Dict[str, List[Path]] = defaultdict(list) + + for input_path in (Path(path_str) for path_str in self._paths): + files = { + file.relative_to(input_path): file + for file in input_path.rglob("*.json") + } + + file_names = [valid_pyname(str(file.with_suffix(""))) for file in files] + path_files = dict(zip(file_names, files.values())) + for file_name, file_path in path_files.items(): + data_files[file_name].append(file_path) + + logger.info("Found %d data file types", len(data_files)) + logger.info("Total data files: %d", sum(len(v) for v in data_files.values())) + if not data_files: + raise MsticpyDataQueryError( + "No usable data files found in supplied paths.", + f"Data paths supplied: {', '.join(self._paths)}", + title="No data files found", + help_uri=_VELOCIRATOR_DOC_URL, + ) + return data_files diff --git a/tests/data/drivers/test_velociraptor_driver.py b/tests/data/drivers/test_velociraptor_driver.py new file mode 100644 index 000000000..bbe2b25b5 --- /dev/null +++ b/tests/data/drivers/test_velociraptor_driver.py @@ -0,0 +1,59 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Module docstring.""" +import pandas as pd +import pytest +import pytest_check as check + +from msticpy.data.core.data_providers import QueryProvider +from msticpy.data.drivers.local_velociraptor_driver import VelociraptorLogDriver + +from ...unit_test_lib import get_test_data_path + +__author__ = "Ian Hellen" + +# pylint: disable=redefined-outer-name, protected-access + + +# change this for actual data + +_VR_LOG_PATH = "velociraptor" + +_EXPECTED_TABLES = [ + "Windows_Forensics_Lnk", + "Windows_Forensics_ProcessInfo", + "Windows_Forensics_Usn", + "Windows_Memory_Acquisition", + "Windows_Network_ArpCache", + "Windows_Network_InterfaceAddresses", + "Windows_Network_ListeningPorts", + "Windows_Network_Netstat", + "Windows_Sys_Users", + "Windows_Sysinternals_Autoruns", + "Windows_System_DNSCache", + "Windows_System_Pslist", +] + + +def test_read_log_files(): + """Test reading Velociraptor logs.""" + query_path = str(get_test_data_path().joinpath(_VR_LOG_PATH)) + vr_driver = VelociraptorLogDriver(data_paths=[query_path]) + + vr_driver.connect() + check.equal(len(vr_driver.data_files), len(_EXPECTED_TABLES)) + check.equal(len(vr_driver.schema), len(_EXPECTED_TABLES)) + + +def test_vr_query(): + """Test loading and querying velociraptor data.""" + os_query_path = str(get_test_data_path().joinpath(_VR_LOG_PATH)) + qry_prov = QueryProvider("Velociraptor", data_paths=[os_query_path]) + qry_prov.connect() + check.equal(len(qry_prov.velociraptor), len(_EXPECTED_TABLES)) + check.equal(len(qry_prov.velociraptor.Windows_Forensics_Lnk()), 10) + check.equal(len(qry_prov.velociraptor.Windows_Forensics_ProcessInfo()), 10) + check.equal(len(qry_prov.velociraptor.Windows_Sys_Users()), 7) diff --git a/tests/testdata/velociraptor/Windows.Forensics.Lnk.json b/tests/testdata/velociraptor/Windows.Forensics.Lnk.json new file mode 100644 index 000000000..df9137d88 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Forensics.Lnk.json @@ -0,0 +1,10 @@ +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\AWS.EC2.WindowsUpdate (2).lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkInfo","HasLinkTargetIDList","HasRelativePath","IsUnicode"],"FileAttributes":["FILE_ATTRIBUTE_DIRECTORY"],"CreationTime":"2017-06-15T17:00:11Z","AccessTime":"2017-06-15T17:00:11Z","WriteTime":"2017-06-15T17:00:11Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":443,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":25,"Offset":98,"Type":32,"Subtype":1,"ShellBag":{"Name":"C:\\","Description":{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"}}},{"ItemIDSize":96,"Offset":123,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-11-24T00:19:30Z","ShortName":"PROGRA~3","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-07-16T13:23:22Z","LastAccessed":"2016-11-24T00:19:30Z","MFTReference":{"MFTID":377,"SequenceNumber":281474976710656},"LongName":"ProgramData"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656}}},{"ItemIDSize":84,"Offset":219,"Type":48,"Subtype":1,"ShellBag":{"Size":84,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-10-18T02:51:34Z","ShortName":"Amazon","Extension":{"Size":62,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2016-10-18T02:51:34Z","MFTReference":{"MFTID":378,"SequenceNumber":281474976710656},"LongName":"Amazon"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656}}},{"ItemIDSize":96,"Offset":303,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-05-19T21:37:18Z","ShortName":"EC2-WI~1","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2017-05-19T21:37:18Z","MFTReference":{"MFTID":379,"SequenceNumber":281474976710656},"LongName":"EC2-Windows"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656}}},{"ItemIDSize":120,"Offset":399,"Type":48,"Subtype":1,"ShellBag":{"Size":120,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","Extension":{"Size":92,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156684,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}}}]},"LinkInfo":{"Offset":521,"LinkInfoSize":109,"LinkInfoFlags":["VolumeIDAndLocalBasePath"],"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}}},"NameInfo":{},"RelativePathInfo":{"Offset":630,"RelativePathInfoSize":146,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate"},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-06-15T19:16:55.502978Z","Atime":"2022-02-12T18:53:53.7569849Z","Ctime":"2017-06-15T19:16:55.502978Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"},{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}],"HeaderCreationTime":"2017-06-15T17:00:11Z","HeaderAccessTime":"2017-06-15T17:00:11Z","HeaderWriteTime":"2017-06-15T17:00:11Z","FileSize":0,"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}},"Name":null,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\AWS.EC2.WindowsUpdate.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkInfo","HasLinkTargetIDList","HasRelativePath","HasWorkingDir","IsUnicode"],"FileAttributes":["FILE_ATTRIBUTE_ARCHIVE"],"CreationTime":"2017-06-15T17:00:11Z","AccessTime":"2017-06-15T17:00:11Z","WriteTime":"2017-06-15T17:44:50Z","FileSize":11098,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":571,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":25,"Offset":98,"Type":32,"Subtype":1,"ShellBag":{"Name":"C:\\","Description":{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"}}},{"ItemIDSize":96,"Offset":123,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-11-24T00:19:30Z","ShortName":"PROGRA~3","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-07-16T13:23:22Z","LastAccessed":"2016-11-24T00:19:30Z","MFTReference":{"MFTID":377,"SequenceNumber":281474976710656},"LongName":"ProgramData"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656}}},{"ItemIDSize":84,"Offset":219,"Type":48,"Subtype":1,"ShellBag":{"Size":84,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-10-18T02:51:34Z","ShortName":"Amazon","Extension":{"Size":62,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2016-10-18T02:51:34Z","MFTReference":{"MFTID":378,"SequenceNumber":281474976710656},"LongName":"Amazon"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656}}},{"ItemIDSize":96,"Offset":303,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-05-19T21:37:18Z","ShortName":"EC2-WI~1","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2017-05-19T21:37:18Z","MFTReference":{"MFTID":379,"SequenceNumber":281474976710656},"LongName":"EC2-Windows"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656}}},{"ItemIDSize":120,"Offset":399,"Type":48,"Subtype":1,"ShellBag":{"Size":120,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","Extension":{"Size":92,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156684,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}}},{"ItemIDSize":128,"Offset":519,"Type":48,"Subtype":0,"ShellBag":{"Size":128,"Type":50,"SubType":["File","Unicode"],"LastModificationTime":"2017-06-15T17:44:52Z","ShortName":"AWSEC2~1.LOG","Extension":{"Size":100,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156686,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate.log"},"Description":{"Type":["File","Unicode"],"Modified":"2017-06-15T17:44:52Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.LOG","LongName":"AWS.EC2.WindowsUpdate.log","MFTID":156686,"MFTSeq":281474976710656}}}]},"LinkInfo":{"Offset":649,"LinkInfoSize":135,"LinkInfoFlags":["VolumeIDAndLocalBasePath"],"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}}},"NameInfo":{},"RelativePathInfo":{"Offset":784,"RelativePathInfoSize":198,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log"},"WorkingDirInfo":{"Offset":984,"WorkingDirInfoSize":110,"WorkingDir":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate"},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-06-15T19:16:55.4717422Z","Atime":"2022-02-12T18:53:53.7413598Z","Ctime":"2017-06-15T19:16:55.4717422Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"},{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656},{"Type":["File","Unicode"],"Modified":"2017-06-15T17:44:52Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.LOG","LongName":"AWS.EC2.WindowsUpdate.log","MFTID":156686,"MFTSeq":281474976710656}],"HeaderCreationTime":"2017-06-15T17:00:11Z","HeaderAccessTime":"2017-06-15T17:00:11Z","HeaderWriteTime":"2017-06-15T17:44:50Z","FileSize":11098,"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}},"Name":null,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","WorkingDir":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\All Tasks.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkTargetIDList","IsUnicode"],"FileAttributes":[],"CreationTime":"1601-01-01T00:00:00Z","AccessTime":"1601-01-01T00:00:00Z","WriteTime":"1601-01-01T00:00:00Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":22,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}}]},"LinkInfo":{},"NameInfo":{},"RelativePathInfo":{},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-05-19T20:56:39.7146821Z","Atime":"2022-02-12T18:53:53.8507453Z","Ctime":"2017-05-19T20:56:39.7146821Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"}],"HeaderCreationTime":"1601-01-01T00:00:00Z","HeaderAccessTime":"1601-01-01T00:00:00Z","HeaderWriteTime":"1601-01-01T00:00:00Z","FileSize":0,"Target":null,"Name":null,"RelativePath":null,"WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\Internet Options.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkTargetIDList","IsUnicode"],"FileAttributes":[],"CreationTime":"1601-01-01T00:00:00Z","AccessTime":"1601-01-01T00:00:00Z","WriteTime":"1601-01-01T00:00:00Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":64,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":12,"Offset":98,"Type":0,"Subtype":1,"ShellBag":null},{"ItemIDSize":30,"Offset":110,"Type":112,"Subtype":1,"ShellBag":null}]},"LinkInfo":{},"NameInfo":{},"RelativePathInfo":{},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2022-02-12T00:32:57.8617334Z","Atime":"2022-02-12T18:53:53.6632322Z","Ctime":"2022-02-12T00:32:57.8617334Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},null,null],"HeaderCreationTime":"1601-01-01T00:00:00Z","HeaderAccessTime":"1601-01-01T00:00:00Z","HeaderWriteTime":"1601-01-01T00:00:00Z","FileSize":0,"Target":null,"Name":null,"RelativePath":null,"WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\Uninstall a program.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkTargetIDList","IsUnicode"],"FileAttributes":[],"CreationTime":"1601-01-01T00:00:00Z","AccessTime":"1601-01-01T00:00:00Z","WriteTime":"1601-01-01T00:00:00Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":366,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":344,"Offset":98,"Type":0,"Subtype":0,"ShellBag":null}]},"LinkInfo":{},"NameInfo":{},"RelativePathInfo":{},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-05-19T20:56:39.6487129Z","Atime":"2022-02-12T18:53:53.6632322Z","Ctime":"2017-05-19T20:56:39.6487129Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},null],"HeaderCreationTime":"1601-01-01T00:00:00Z","HeaderAccessTime":"1601-01-01T00:00:00Z","HeaderWriteTime":"1601-01-01T00:00:00Z","FileSize":0,"Target":null,"Name":null,"RelativePath":null,"WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\domi.nusvir\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\WindowsUpdate.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkInfo","HasLinkTargetIDList","HasRelativePath","HasWorkingDir","IsUnicode"],"FileAttributes":["FILE_ATTRIBUTE_ARCHIVE"],"CreationTime":"2017-05-12T00:37:56Z","AccessTime":"2017-05-12T00:37:56Z","WriteTime":"2017-05-12T00:53:10Z","FileSize":194617,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":114,"IDList":[{"ItemIDSize":112,"Offset":78,"Type":48,"Subtype":0,"ShellBag":{"Size":112,"Type":50,"SubType":["File","Unicode"],"LastModificationTime":"2017-05-12T00:53:12Z","ShortName":"WINDOW~1.LOG","Extension":{"Size":84,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-05-12T00:37:58Z","LastAccessed":"2017-05-12T00:37:58Z","MFTReference":{"MFTID":153408,"SequenceNumber":562949953421312},"LongName":"WindowsUpdate.log"},"Description":{"Type":["File","Unicode"],"Modified":"2017-05-12T00:53:12Z","LastAccessed":"2017-05-12T00:37:58Z","CreateDate":"2017-05-12T00:37:58Z","ShortName":"WINDOW~1.LOG","LongName":"WindowsUpdate.log","MFTID":153408,"MFTSeq":562949953421312}}}]},"LinkInfo":{"Offset":192,"LinkInfoSize":102,"LinkInfoFlags":["VolumeIDAndLocalBasePath"],"Target":{"path":"C:\\Users\\Administrator\\Desktop\\WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":808864001,"VolumeLabel":"Windows"}}},"NameInfo":{},"RelativePathInfo":{"Offset":294,"RelativePathInfoSize":80,"RelativePath":"..\\..\\..\\..\\..\\Desktop\\WindowsUpdate.log"},"WorkingDirInfo":{"Offset":376,"WorkingDirInfoSize":60,"WorkingDir":"C:\\Users\\Administrator\\Desktop"},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-05-12T00:53:13.6114769Z","Atime":"2022-02-12T18:53:53.6632322Z","Ctime":"2017-05-12T00:53:13.6114769Z","_TargetIDInfo":[{"Type":["File","Unicode"],"Modified":"2017-05-12T00:53:12Z","LastAccessed":"2017-05-12T00:37:58Z","CreateDate":"2017-05-12T00:37:58Z","ShortName":"WINDOW~1.LOG","LongName":"WindowsUpdate.log","MFTID":153408,"MFTSeq":562949953421312}],"HeaderCreationTime":"2017-05-12T00:37:56Z","HeaderAccessTime":"2017-05-12T00:37:56Z","HeaderWriteTime":"2017-05-12T00:53:10Z","FileSize":194617,"Target":{"path":"C:\\Users\\Administrator\\Desktop\\WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":808864001,"VolumeLabel":"Windows"}},"Name":null,"RelativePath":"..\\..\\..\\..\\..\\Desktop\\WindowsUpdate.log","WorkingDir":"C:\\Users\\Administrator\\Desktop","Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\Administrator\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\AWS.EC2.WindowsUpdate (2).lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkInfo","HasLinkTargetIDList","HasRelativePath","IsUnicode"],"FileAttributes":["FILE_ATTRIBUTE_DIRECTORY"],"CreationTime":"2017-06-15T17:00:11Z","AccessTime":"2017-06-15T17:00:11Z","WriteTime":"2017-06-15T17:00:11Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":443,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":25,"Offset":98,"Type":32,"Subtype":1,"ShellBag":{"Name":"C:\\","Description":{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"}}},{"ItemIDSize":96,"Offset":123,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-11-24T00:19:30Z","ShortName":"PROGRA~3","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-07-16T13:23:22Z","LastAccessed":"2016-11-24T00:19:30Z","MFTReference":{"MFTID":377,"SequenceNumber":281474976710656},"LongName":"ProgramData"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656}}},{"ItemIDSize":84,"Offset":219,"Type":48,"Subtype":1,"ShellBag":{"Size":84,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-10-18T02:51:34Z","ShortName":"Amazon","Extension":{"Size":62,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2016-10-18T02:51:34Z","MFTReference":{"MFTID":378,"SequenceNumber":281474976710656},"LongName":"Amazon"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656}}},{"ItemIDSize":96,"Offset":303,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-05-19T21:37:18Z","ShortName":"EC2-WI~1","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2017-05-19T21:37:18Z","MFTReference":{"MFTID":379,"SequenceNumber":281474976710656},"LongName":"EC2-Windows"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656}}},{"ItemIDSize":120,"Offset":399,"Type":48,"Subtype":1,"ShellBag":{"Size":120,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","Extension":{"Size":92,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156684,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}}}]},"LinkInfo":{"Offset":521,"LinkInfoSize":109,"LinkInfoFlags":["VolumeIDAndLocalBasePath"],"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}}},"NameInfo":{},"RelativePathInfo":{"Offset":630,"RelativePathInfoSize":146,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate"},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-06-15T19:16:55.502978Z","Atime":"2022-02-12T01:17:15.156236Z","Ctime":"2017-06-15T19:16:55.502978Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"},{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}],"HeaderCreationTime":"2017-06-15T17:00:11Z","HeaderAccessTime":"2017-06-15T17:00:11Z","HeaderWriteTime":"2017-06-15T17:00:11Z","FileSize":0,"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}},"Name":null,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\Administrator\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\AWS.EC2.WindowsUpdate.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkInfo","HasLinkTargetIDList","HasRelativePath","HasWorkingDir","IsUnicode"],"FileAttributes":["FILE_ATTRIBUTE_ARCHIVE"],"CreationTime":"2017-06-15T17:00:11Z","AccessTime":"2017-06-15T17:00:11Z","WriteTime":"2017-06-15T17:44:50Z","FileSize":11098,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":571,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":25,"Offset":98,"Type":32,"Subtype":1,"ShellBag":{"Name":"C:\\","Description":{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"}}},{"ItemIDSize":96,"Offset":123,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-11-24T00:19:30Z","ShortName":"PROGRA~3","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-07-16T13:23:22Z","LastAccessed":"2016-11-24T00:19:30Z","MFTReference":{"MFTID":377,"SequenceNumber":281474976710656},"LongName":"ProgramData"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656}}},{"ItemIDSize":84,"Offset":219,"Type":48,"Subtype":1,"ShellBag":{"Size":84,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2016-10-18T02:51:34Z","ShortName":"Amazon","Extension":{"Size":62,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2016-10-18T02:51:34Z","MFTReference":{"MFTID":378,"SequenceNumber":281474976710656},"LongName":"Amazon"},"Description":{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656}}},{"ItemIDSize":96,"Offset":303,"Type":48,"Subtype":1,"ShellBag":{"Size":96,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-05-19T21:37:18Z","ShortName":"EC2-WI~1","Extension":{"Size":72,"Version":9,"Signature":"0xbeef0004","CreateDate":"2016-10-18T00:07:46Z","LastAccessed":"2017-05-19T21:37:18Z","MFTReference":{"MFTID":379,"SequenceNumber":281474976710656},"LongName":"EC2-Windows"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656}}},{"ItemIDSize":120,"Offset":399,"Type":48,"Subtype":1,"ShellBag":{"Size":120,"Type":49,"SubType":["Directory","Unicode"],"LastModificationTime":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","Extension":{"Size":92,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156684,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate"},"Description":{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656}}},{"ItemIDSize":128,"Offset":519,"Type":48,"Subtype":0,"ShellBag":{"Size":128,"Type":50,"SubType":["File","Unicode"],"LastModificationTime":"2017-06-15T17:44:52Z","ShortName":"AWSEC2~1.LOG","Extension":{"Size":100,"Version":9,"Signature":"0xbeef0004","CreateDate":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","MFTReference":{"MFTID":156686,"SequenceNumber":281474976710656},"LongName":"AWS.EC2.WindowsUpdate.log"},"Description":{"Type":["File","Unicode"],"Modified":"2017-06-15T17:44:52Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.LOG","LongName":"AWS.EC2.WindowsUpdate.log","MFTID":156686,"MFTSeq":281474976710656}}}]},"LinkInfo":{"Offset":649,"LinkInfoSize":135,"LinkInfoFlags":["VolumeIDAndLocalBasePath"],"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}}},"NameInfo":{},"RelativePathInfo":{"Offset":784,"RelativePathInfoSize":198,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log"},"WorkingDirInfo":{"Offset":984,"WorkingDirInfoSize":110,"WorkingDir":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate"},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-06-15T19:16:55.4717422Z","Atime":"2022-02-12T01:17:15.1552287Z","Ctime":"2017-06-15T19:16:55.4717422Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},{"LongName":"C:\\","ShortName":"C:\\","Type":"Volume"},{"Type":["Directory","Unicode"],"Modified":"2016-11-24T00:19:30Z","LastAccessed":"2016-11-24T00:19:30Z","CreateDate":"2016-07-16T13:23:22Z","ShortName":"PROGRA~3","LongName":"ProgramData","MFTID":377,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2016-10-18T02:51:34Z","LastAccessed":"2016-10-18T02:51:34Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"Amazon","LongName":"Amazon","MFTID":378,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-05-19T21:37:18Z","LastAccessed":"2017-05-19T21:37:18Z","CreateDate":"2016-10-18T00:07:46Z","ShortName":"EC2-WI~1","LongName":"EC2-Windows","MFTID":379,"MFTSeq":281474976710656},{"Type":["Directory","Unicode"],"Modified":"2017-06-15T17:00:12Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.WIN","LongName":"AWS.EC2.WindowsUpdate","MFTID":156684,"MFTSeq":281474976710656},{"Type":["File","Unicode"],"Modified":"2017-06-15T17:44:52Z","LastAccessed":"2017-06-15T17:00:12Z","CreateDate":"2017-06-15T17:00:12Z","ShortName":"AWSEC2~1.LOG","LongName":"AWS.EC2.WindowsUpdate.log","MFTID":156686,"MFTSeq":281474976710656}],"HeaderCreationTime":"2017-06-15T17:00:11Z","HeaderAccessTime":"2017-06-15T17:00:11Z","HeaderWriteTime":"2017-06-15T17:44:50Z","FileSize":11098,"Target":{"path":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","volume_info":{"DriveType":"DRIVE_FIXED","DriveSerialNumber":1982990285,"VolumeLabel":"Windows"}},"Name":null,"RelativePath":"..\\..\\..\\..\\..\\..\\..\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate\\AWS.EC2.WindowsUpdate.log","WorkingDir":"C:\\ProgramData\\Amazon\\EC2-Windows\\AWS.EC2.WindowsUpdate","Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\Administrator\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\All Tasks.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkTargetIDList","IsUnicode"],"FileAttributes":[],"CreationTime":"1601-01-01T00:00:00Z","AccessTime":"1601-01-01T00:00:00Z","WriteTime":"1601-01-01T00:00:00Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":22,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}}]},"LinkInfo":{},"NameInfo":{},"RelativePathInfo":{},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2017-05-19T20:56:39.7146821Z","Atime":"2022-02-12T01:17:15.2156524Z","Ctime":"2017-05-19T20:56:39.7146821Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"}],"HeaderCreationTime":"1601-01-01T00:00:00Z","HeaderAccessTime":"1601-01-01T00:00:00Z","HeaderWriteTime":"1601-01-01T00:00:00Z","FileSize":0,"Target":null,"Name":null,"RelativePath":null,"WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} +{"FullPath":"C:\\Users\\Administrator\\AppData\\Roaming\\Microsoft\\Windows\\Recent\\Internet Options.lnk","_Parsed":{"HeaderSize":76,"LinkClsID":"0114020000000000c000000000000046","LinkFlags":["DisableKnownFolderTracking","HasLinkTargetIDList","IsUnicode"],"FileAttributes":[],"CreationTime":"1601-01-01T00:00:00Z","AccessTime":"1601-01-01T00:00:00Z","WriteTime":"1601-01-01T00:00:00Z","FileSize":0,"IconIndex":0,"ShowCommand":1,"HotKey":0,"LinkTargetIDList":{"IDListSize":64,"IDList":[{"ItemIDSize":20,"Offset":78,"Type":16,"Subtype":1,"ShellBag":{"Description":{"ShortName":"My Computer","Type":"Root"}}},{"ItemIDSize":12,"Offset":98,"Type":0,"Subtype":1,"ShellBag":null},{"ItemIDSize":30,"Offset":110,"Type":112,"Subtype":1,"ShellBag":null}]},"LinkInfo":{},"NameInfo":{},"RelativePathInfo":{},"WorkingDirInfo":{},"ArgumentInfo":{},"IconInfo":{}},"Mtime":"2022-02-12T00:32:57.8617334Z","Atime":"2022-02-12T01:17:15.1058888Z","Ctime":"2022-02-12T00:32:57.8617334Z","_TargetIDInfo":[{"ShortName":"My Computer","Type":"Root"},null,null],"HeaderCreationTime":"1601-01-01T00:00:00Z","HeaderAccessTime":"1601-01-01T00:00:00Z","HeaderWriteTime":"1601-01-01T00:00:00Z","FileSize":0,"Target":null,"Name":null,"RelativePath":null,"WorkingDir":null,"Arguments":null,"Icons":null,"Upload":null} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Forensics.ProcessInfo.json b/tests/testdata/velociraptor/Windows.Forensics.ProcessInfo.json new file mode 100644 index 000000000..36c3c271e --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Forensics.ProcessInfo.json @@ -0,0 +1,10 @@ +{"Name":"System","PebBaseAddress":"0x0","Pid":4,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"smss.exe","PebBaseAddress":"0x5b6ea41000","Pid":268,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"csrss.exe","PebBaseAddress":"0x91fdbd0000","Pid":348,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"wininit.exe","PebBaseAddress":"0xc5f8d8e000","Pid":416,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"csrss.exe","PebBaseAddress":"0x65715c4000","Pid":424,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"winlogon.exe","PebBaseAddress":"0xaa675e0000","Pid":492,"ImagePathName":"C:\\Windows\\system32\\winlogon.exe","CommandLine":"winlogon.exe","CurrentDirectory":"C:\\Windows\\system32\\","Env":{"ALLUSERSPROFILE":"C:\\ProgramData","ChocolateyInstall":"C:\\ProgramData\\chocolatey","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"WKST01","ComSpec":"C:\\Windows\\system32\\cmd.exe","NUMBER_OF_PROCESSORS":"2","OS":"Windows_NT","Path":"C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Amazon\\cfn-bootstrap\\;C:\\ProgramData\\chocolatey\\bin;;C:\\Program Files\\Microsoft VS Code\\bin","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 7, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5507","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"%ProgramFiles%\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AWS Tools\\PowerShell\\","PUBLIC":"C:\\Users\\Public","SystemDrive":"C:","SystemRoot":"C:\\Windows","TEMP":"C:\\Windows\\TEMP","TMP":"C:\\Windows\\TEMP","USERNAME":"SYSTEM","USERPROFILE":"C:\\Windows\\system32\\config\\systemprofile","windir":"C:\\Windows"}} +{"Name":"services.exe","PebBaseAddress":"0x3917899000","Pid":540,"ImagePathName":null,"CommandLine":null,"CurrentDirectory":null,"Env":{}} +{"Name":"lsass.exe","PebBaseAddress":"0x7c55f75000","Pid":548,"ImagePathName":"C:\\Windows\\system32\\lsass.exe","CommandLine":"C:\\Windows\\system32\\lsass.exe","CurrentDirectory":"C:\\Windows\\system32\\","Env":{"ALLUSERSPROFILE":"C:\\ProgramData","ChocolateyInstall":"C:\\ProgramData\\chocolatey","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"WKST01","ComSpec":"C:\\Windows\\system32\\cmd.exe","NUMBER_OF_PROCESSORS":"2","OS":"Windows_NT","Path":"C:\\Windows\\System32","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 7, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5507","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"%ProgramFiles%\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AWS Tools\\PowerShell\\","PUBLIC":"C:\\Users\\Public","SystemDrive":"C:","SystemRoot":"C:\\Windows","TEMP":"C:\\Windows\\TEMP","TMP":"C:\\Windows\\TEMP","USERNAME":"SYSTEM","USERPROFILE":"C:\\Windows\\system32\\config\\systemprofile","windir":"C:\\Windows"}} +{"Name":"svchost.exe","PebBaseAddress":"0x1b865e5000","Pid":636,"ImagePathName":"C:\\Windows\\system32\\svchost.exe","CommandLine":"C:\\Windows\\system32\\svchost.exe -k DcomLaunch","CurrentDirectory":"C:\\Windows\\system32\\","Env":{"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Windows\\system32\\config\\systemprofile\\AppData\\Roaming","ChocolateyInstall":"C:\\ProgramData\\chocolatey","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"WKST01","ComSpec":"C:\\Windows\\system32\\cmd.exe","LOCALAPPDATA":"C:\\Windows\\system32\\config\\systemprofile\\AppData\\Local","NUMBER_OF_PROCESSORS":"2","OS":"Windows_NT","Path":"C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Amazon\\cfn-bootstrap\\;C:\\ProgramData\\chocolatey\\bin;;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Windows\\system32\\config\\systemprofile\\AppData\\Local\\Microsoft\\WindowsApps","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 7, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5507","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"%ProgramFiles%\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AWS Tools\\PowerShell\\","PUBLIC":"C:\\Users\\Public","SystemDrive":"C:","SystemRoot":"C:\\Windows","TEMP":"C:\\Windows\\TEMP","TMP":"C:\\Windows\\TEMP","USERDOMAIN":"MAGNUMTEMPUS","USERNAME":"WKST01$","USERPROFILE":"C:\\Windows\\system32\\config\\systemprofile","windir":"C:\\Windows"}} +{"Name":"svchost.exe","PebBaseAddress":"0xe60fe88000","Pid":692,"ImagePathName":"C:\\Windows\\system32\\svchost.exe","CommandLine":"C:\\Windows\\system32\\svchost.exe -k RPCSS","CurrentDirectory":"C:\\Windows\\system32\\","Env":{"ALLUSERSPROFILE":"C:\\ProgramData","APPDATA":"C:\\Windows\\ServiceProfiles\\NetworkService\\AppData\\Roaming","ChocolateyInstall":"C:\\ProgramData\\chocolatey","CommonProgramFiles":"C:\\Program Files\\Common Files","CommonProgramFiles(x86)":"C:\\Program Files (x86)\\Common Files","CommonProgramW6432":"C:\\Program Files\\Common Files","COMPUTERNAME":"WKST01","ComSpec":"C:\\Windows\\system32\\cmd.exe","LOCALAPPDATA":"C:\\Windows\\ServiceProfiles\\NetworkService\\AppData\\Local","NUMBER_OF_PROCESSORS":"2","OS":"Windows_NT","Path":"C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Program Files\\Amazon\\cfn-bootstrap\\;C:\\ProgramData\\chocolatey\\bin;;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Windows\\ServiceProfiles\\NetworkService\\AppData\\Local\\Microsoft\\WindowsApps","PATHEXT":".COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC","PROCESSOR_ARCHITECTURE":"AMD64","PROCESSOR_IDENTIFIER":"Intel64 Family 6 Model 85 Stepping 7, GenuineIntel","PROCESSOR_LEVEL":"6","PROCESSOR_REVISION":"5507","ProgramData":"C:\\ProgramData","ProgramFiles":"C:\\Program Files","ProgramFiles(x86)":"C:\\Program Files (x86)","ProgramW6432":"C:\\Program Files","PSModulePath":"%ProgramFiles%\\WindowsPowerShell\\Modules;C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\Modules;C:\\Program Files (x86)\\AWS Tools\\PowerShell\\","PUBLIC":"C:\\Users\\Public","SystemDrive":"C:","SystemRoot":"C:\\Windows","TEMP":"C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp","TMP":"C:\\Windows\\SERVIC~2\\NETWOR~1\\AppData\\Local\\Temp","USERDOMAIN":"MAGNUMTEMPUS","USERNAME":"WKST01$","USERPROFILE":"C:\\Windows\\ServiceProfiles\\NetworkService","windir":"C:\\Windows"}} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Forensics.Usn.json b/tests/testdata/velociraptor/Windows.Forensics.Usn.json new file mode 100644 index 000000000..090efa0fd --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Forensics.Usn.json @@ -0,0 +1,10 @@ +{"Usn":25165824,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25165920,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION","CLOSE"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166016,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166112,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166208,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION","CLOSE"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166304,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166400,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166496,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION","CLOSE"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166592,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} +{"Usn":25166688,"Timestamp":"2022-02-12T00:36:43.7895496Z","Filename":"setupapi.dev.log","FullPath":"Windows/INF/setupapi.dev.log","FileAttributes":["ARCHIVE"],"Reason":["DATA_EXTEND","DATA_TRUNCATION"],"SourceInfo":["ARCHIVE"],"_FileMFTID":1128,"_FileMFTSequence":3,"_ParentMFTID":3403,"_ParentMFTSequence":1} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Memory.Acquisition.json b/tests/testdata/velociraptor/Windows.Memory.Acquisition.json new file mode 100644 index 000000000..8250c19ed --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Memory.Acquisition.json @@ -0,0 +1,10 @@ +{"Stdout":"WinPmem64","Stderr":"","Upload":null} +{"Stdout":"Extracting driver to C:\\Users\\ADMINI~1.MAG\\AppData\\Local\\Temp\\pmeCEC5.tmp","Stderr":"","Upload":null} +{"Stdout":"Driver Unloaded.","Stderr":"","Upload":null} +{"Stdout":"Loaded Driver C:\\Users\\ADMINI~1.MAG\\AppData\\Local\\Temp\\pmeCEC5.tmp.","Stderr":"","Upload":null} +{"Stdout":"Deleting C:\\Users\\ADMINI~1.MAG\\AppData\\Local\\Temp\\pmeCEC5.tmp","Stderr":"","Upload":null} +{"Stdout":"The system time is: 23:59:48","Stderr":"","Upload":null} +{"Stdout":"Will generate a RAW image ","Stderr":"","Upload":null} +{"Stdout":" - buffer_size_: 0x1000","Stderr":"","Upload":null} +{"Stdout":"CR3: 0x00001AA002","Stderr":"","Upload":null} +{"Stdout":" 4 memory ranges:","Stderr":"","Upload":null} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Network.ArpCache.json b/tests/testdata/velociraptor/Windows.Network.ArpCache.json new file mode 100644 index 000000000..5b677e2d7 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Network.ArpCache.json @@ -0,0 +1,10 @@ +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"fe80::89af:1f88:9430:707","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:ff30:707","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-FF-30-07-07"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"172.16.50.130","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:ff30:707","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-FF-30-07-07"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"fe80::89af:1f88:9430:707","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:3","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-01-00-03"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"172.16.50.130","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:3","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-01-00-03"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"fe80::89af:1f88:9430:707","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:2","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-01-00-02"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"172.16.50.130","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::1:2","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-01-00-02"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"fe80::89af:1f88:9430:707","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::fb","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-00-00-FB"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"172.16.50.130","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::fb","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-00-00-FB"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"fe80::89af:1f88:9430:707","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::16","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-00-00-16"} +{"AddressFamily":"IPv6","Store":"Active","State":"Permanent","InterfaceIndex":2,"LocalAddress":"172.16.50.130","HardwareAddr":"06:f7:47:22:b0:c6","RemoteAddress":"ff02::16","InterfaceAlias":"Ethernet 2","RemoteMACAddress":"33-33-00-00-00-16"} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Network.InterfaceAddresses.json b/tests/testdata/velociraptor/Windows.Network.InterfaceAddresses.json new file mode 100644 index 000000000..890fba1d3 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Network.InterfaceAddresses.json @@ -0,0 +1,5 @@ +{"Index":2,"MTU":1500,"Name":"Ethernet 2","HardwareAddr":"06:f7:47:22:b0:c6","Flags":19,"IP":"fe80::89af:1f88:9430:707","Mask":"ffffffffffffffff0000000000000000"} +{"Index":2,"MTU":1500,"Name":"Ethernet 2","HardwareAddr":"06:f7:47:22:b0:c6","Flags":19,"IP":"172.16.50.130","Mask":"ffffff00"} +{"Index":1,"MTU":-1,"Name":"Loopback Pseudo-Interface 1","HardwareAddr":null,"Flags":21,"IP":"::1","Mask":"ffffffffffffffffffffffffffffffff"} +{"Index":1,"MTU":-1,"Name":"Loopback Pseudo-Interface 1","HardwareAddr":null,"Flags":21,"IP":"127.0.0.1","Mask":"ff000000"} +{"Index":7,"MTU":1280,"Name":"isatap.us-east-2.compute.internal","HardwareAddr":"00:00:00:00:00:00:00:e0","Flags":24,"IP":"fe80::5efe:ac10:3282","Mask":"ffffffffffffffffffffffffffffffff"} diff --git a/tests/testdata/velociraptor/Windows.Network.ListeningPorts.json b/tests/testdata/velociraptor/Windows.Network.ListeningPorts.json new file mode 100644 index 000000000..38501bcdf --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Network.ListeningPorts.json @@ -0,0 +1,10 @@ +{"Pid":692,"Name":"svchost.exe","Port":135,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":4,"Name":"System","Port":139,"Protocol":"TCP","Family":"IPv4","Address":"172.16.50.130"} +{"Pid":872,"Name":"svchost.exe","Port":3389,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":416,"Name":"wininit.exe","Port":49664,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":912,"Name":"svchost.exe","Port":49665,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":1284,"Name":"svchost.exe","Port":49667,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":548,"Name":"lsass.exe","Port":49669,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":1664,"Name":"spoolsv.exe","Port":49697,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":548,"Name":"lsass.exe","Port":49722,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} +{"Pid":540,"Name":"services.exe","Port":49728,"Protocol":"TCP","Family":"IPv4","Address":"0.0.0.0"} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Network.Netstat.json b/tests/testdata/velociraptor/Windows.Network.Netstat.json new file mode 100644 index 000000000..3c8afdd97 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Network.Netstat.json @@ -0,0 +1,10 @@ +{"Pid":692,"Name":"svchost.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":135,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:44Z"} +{"Pid":4,"Name":"System","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"172.16.50.130","Laddr.Port":139,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:44Z"} +{"Pid":872,"Name":"svchost.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":3389,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"} +{"Pid":872,"Name":"svchost.exe","Family":"IPv4","Type":"TCP","Status":"ESTAB","Laddr.IP":"172.16.50.130","Laddr.Port":3389,"Raddr.IP":"172.16.21.100","Raddr.Port":51588,"Timestamp":"2022-02-12T23:57:49Z"} +{"Pid":2288,"Name":"windows_exporter.exe","Family":"IPv4","Type":"TCP","Status":"ESTAB","Laddr.IP":"172.16.50.130","Laddr.Port":9100,"Raddr.IP":"172.16.10.100","Raddr.Port":39806,"Timestamp":"2022-02-12T19:35:58Z"} +{"Pid":416,"Name":"wininit.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49664,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:44Z"} +{"Pid":912,"Name":"svchost.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49665,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:44Z"} +{"Pid":1284,"Name":"svchost.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49667,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"} +{"Pid":548,"Name":"lsass.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49669,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"} +{"Pid":1664,"Name":"spoolsv.exe","Family":"IPv4","Type":"TCP","Status":"LISTEN","Laddr.IP":"0.0.0.0","Laddr.Port":49697,"Raddr.IP":"0.0.0.0","Raddr.Port":0,"Timestamp":"2022-02-12T19:35:45Z"} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.Sys.Users.json b/tests/testdata/velociraptor/Windows.Sys.Users.json new file mode 100644 index 000000000..2c9a73780 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Sys.Users.json @@ -0,0 +1,7 @@ +{"Uid":500,"Gid":513,"Name":"Administrator","Description":"Built-in account for administering the computer/domain","Directory":"C:\\Users\\Administrator","UUID":"S-1-5-21-2205072431-2485292186-3246622615-500","Mtime":"2022-02-12T18:50:11.0557956Z","Type":"local"} +{"Uid":503,"Gid":513,"Name":"DefaultAccount","Description":"A user account managed by the system.","Directory":null,"UUID":"S-1-5-21-2205072431-2485292186-3246622615-503","Mtime":null,"Type":"local"} +{"Uid":501,"Gid":513,"Name":"Guest","Description":"Built-in account for guest access to the computer/domain","Directory":null,"UUID":"S-1-5-21-2205072431-2485292186-3246622615-501","Mtime":null,"Type":"local"} +{"Uid":"","Gid":"","Name":"SYSTEM","Description":"\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\ProfileList\\S-1-5-18","Directory":"%systemroot%\\system32\\config\\systemprofile","UUID":"S-1-5-18","Mtime":"2016-07-16T13:24:47.6485195Z","Type":"roaming"} +{"Uid":"","Gid":"","Name":"LOCAL SERVICE","Description":"\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\ProfileList\\S-1-5-19","Directory":"C:\\Windows\\ServiceProfiles\\LocalService","UUID":"S-1-5-19","Mtime":"2016-09-12T11:33:51.7795967Z","Type":"roaming"} +{"Uid":"","Gid":"","Name":"NETWORK SERVICE","Description":"\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\ProfileList\\S-1-5-20","Directory":"C:\\Windows\\ServiceProfiles\\NetworkService","UUID":"S-1-5-20","Mtime":"2016-09-12T11:33:50.8358761Z","Type":"roaming"} +{"Uid":"","Gid":"","Name":"Administrator","Description":"\\HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\ProfileList\\S-1-5-21-2370586174-1517003462-1142029260-500","Directory":"C:\\Users\\Administrator.MAGNUMTEMPUS","UUID":"S-1-5-21-2370586174-1517003462-1142029260-500","Mtime":"2022-02-12T23:59:46.1745774Z","Type":"roaming"} diff --git a/tests/testdata/velociraptor/Windows.Sysinternals.Autoruns.json b/tests/testdata/velociraptor/Windows.Sysinternals.Autoruns.json new file mode 100644 index 000000000..429445550 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.Sysinternals.Autoruns.json @@ -0,0 +1,10 @@ +{"Time":"20220212-203745","Entry Location":"HKLM\\System\\CurrentControlSet\\Control\\Session Manager\\BootExecute","Entry":"","Enabled":"","Category":"Boot Execute","Profile":"System-wide","Description":"","Company":"","Image Path":"","Version":"","Launch String":"","MD5":"","SHA-1":"","PESHA-1":"","PESHA-256":"","SHA-256":"","IMP":""} +{"Time":"20210408-073952","Entry Location":"HKLM\\System\\CurrentControlSet\\Control\\Session Manager\\BootExecute","Entry":"autocheck autochk /q /v *","Enabled":"enabled","Category":"Boot Execute","Profile":"System-wide","Description":"Auto Check Utility","Company":"Microsoft Corporation","Image Path":"c:\\windows\\system32\\autochk.exe","Version":"10.0.14393.4350","Launch String":"autocheck autochk /q /v *","MD5":"A512733E2C767F87A8029400B4A48CD0","SHA-1":"E20DD6960F5EFB37D147D26910FF239D57EFFC06","PESHA-1":"FDE685A5880D3EF3A5DE738FBADB91480A8A8315","PESHA-256":"E746C91AB4AF82B5EF60792A6388793EB2ED6E32C919E5A428BAC9F515513C19","SHA-256":"1ED75EB59C2897304E0160E0605071178418802C31910D78A2076B0414047875","IMP":"1BF5E4792E849FE3BCFE23E7C1B21A3F"} +{"Time":"20220212-002036","Entry Location":"HKLM\\Software\\Microsoft\\Office\\Outlook\\Addins","Entry":"","Enabled":"","Category":"Office Addins","Profile":"System-wide","Description":"","Company":"","Image Path":"","Version":"","Launch String":"","MD5":"","SHA-1":"","PESHA-1":"","PESHA-256":"","SHA-256":"","IMP":""} +{"Time":"20210605-043306","Entry Location":"HKLM\\Software\\Microsoft\\Office\\Outlook\\Addins","Entry":"Windows_Search_OutlookToolbar","Enabled":"disabled","Category":"Office Addins","Profile":"System-wide","Description":"Outlook MSSearch Connector","Company":"Microsoft Corporation","Image Path":"c:\\windows\\system32\\mssphtb.dll","Version":"7.0.14393.4467","Launch String":"HKCR\\CLSID\\{F37AFD4F-E736-4980-8650-A486B1F2DF25}","MD5":"05BD2C094A2B52481554F6841149345D","SHA-1":"A3EAB25EE07AE1D9700C3CEDEEA588D66A8E49EF","PESHA-1":"E3013E221F9F6E24AB1BA8591C973540BD8D210B","PESHA-256":"71EE77030DF5F89D89B13DED3B65A1BB10B87FE227A186B33554450878B469F0","SHA-256":"6144C47A4F28EF44150E19E986F0B5FA1D28E5AA553EDC143DC23A8B50010082","IMP":"DBA3AD1CA0A0E7F336F8C0911CFC3BA8"} +{"Time":"20211231-082038","Entry Location":"HKLM\\Software\\Microsoft\\Office\\Outlook\\Addins","Entry":"LyncAddin Class","Enabled":"enabled","Category":"Office Addins","Profile":"System-wide","Description":"Skype for Business","Company":"Microsoft Corporation","Image Path":"c:\\program files\\microsoft office\\root\\office16\\ucaddin.dll","Version":"16.0.14827.20024","Launch String":"HKCR\\CLSID\\{a6a2383f-ad50-4d52-8110-3508275e77f7}","MD5":"792A6D548120A7C829E57BCF132446F9","SHA-1":"861CDF3C15ECC3562CA2C61BBD4317F6F56E20B3","PESHA-1":"47FA36A117350E465EBC80C9251616E76E68FBAD","PESHA-256":"38998B4C0F4470C005FBE7A347F14042854344E00BE5C5ECC3437615917B7D80","SHA-256":"B9554A476B88D77351369BBB2B1FC3A6D91F06FC8677AB0B2E9ACC289FAE4DE4","IMP":"52EE43B6C8076104B51CE3CC035CF7DA"} +{"Time":"20160716-132407","Entry Location":"HKLM\\Software\\Wow6432Node\\Microsoft\\Office\\Outlook\\Addins","Entry":"","Enabled":"","Category":"Office Addins","Profile":"System-wide","Description":"","Company":"","Image Path":"","Version":"","Launch String":"","MD5":"","SHA-1":"","PESHA-1":"","PESHA-256":"","SHA-256":"","IMP":""} +{"Time":"20210605-043226","Entry Location":"HKLM\\Software\\Wow6432Node\\Microsoft\\Office\\Outlook\\Addins","Entry":"Windows_Search_OutlookToolbar","Enabled":"enabled","Category":"Office Addins","Profile":"System-wide","Description":"Outlook MSSearch Connector","Company":"Microsoft Corporation","Image Path":"c:\\windows\\syswow64\\mssphtb.dll","Version":"7.0.14393.4467","Launch String":"HKCR\\CLSID\\{F37AFD4F-E736-4980-8650-A486B1F2DF25}","MD5":"2230582AABCA7A403368FBD1C37FF8A8","SHA-1":"AC123FA4906CB8BEFA060007E0A312E589FD16BC","PESHA-1":"6AA58860FF74584FB486CA1F857AB4C188281B02","PESHA-256":"1ABF1D4078AA35C8C33583F0B01BF55495C238DFCFA979AEF2A7B03186BAE450","SHA-256":"0549064689A3C2FA5862D4263F88F5179B766FB31896E6E208001F870AE89AAF","IMP":"22D2796DD696B78C9F461093647A54C1"} +{"Time":"20160912-113332","Entry Location":"HKLM\\SOFTWARE\\Classes\\Htmlfile\\Shell\\Open\\Command\\(Default)","Entry":"","Enabled":"","Category":"Hijacks","Profile":"System-wide","Description":"","Company":"","Image Path":"","Version":"","Launch String":"","MD5":"","SHA-1":"","PESHA-1":"","PESHA-256":"","SHA-256":"","IMP":""} +{"Time":"20180101-044027","Entry Location":"HKLM\\SOFTWARE\\Classes\\Htmlfile\\Shell\\Open\\Command\\(Default)","Entry":"C:\\Program Files\\Internet Explorer\\iexplore.exe","Enabled":"enabled","Category":"Hijacks","Profile":"System-wide","Description":"Internet Explorer","Company":"Microsoft Corporation","Image Path":"c:\\program files\\internet explorer\\iexplore.exe","Version":"11.0.14393.2007","Launch String":"","MD5":"DED3D744D46A5CE7965CE2B75B54958A","SHA-1":"D4ABAC114DBE28BAD8855C10D37F2B727177C9CA","PESHA-1":"C870605936B2E6D2CB2383CE4B856449FF93D09B","PESHA-256":"C656C4A4179CA12CF6F78FDA6D97BA00B575F9066B6E7A579B63BC41CCD76E50","SHA-256":"70C9616C026266BB3A1213BCC50E3A9A24238703FB7745746628D11163905D2F","IMP":"9BB01C801600CEBDCA166D0534E98CE6"} +{"Time":"20220212-235948","Entry Location":"HKLM\\System\\CurrentControlSet\\Services","Entry":"","Enabled":"","Category":"Services","Profile":"System-wide","Description":"","Company":"","Image Path":"","Version":"","Launch String":"","MD5":"","SHA-1":"","PESHA-1":"","PESHA-256":"","SHA-256":"","IMP":""} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.System.DNSCache.json b/tests/testdata/velociraptor/Windows.System.DNSCache.json new file mode 100644 index 000000000..ab227b297 --- /dev/null +++ b/tests/testdata/velociraptor/Windows.System.DNSCache.json @@ -0,0 +1,10 @@ +{"Name":"tr.blismedia.com","RecordType":1,"TTL":38407,"Type":"Answer","A":"34.96.105.8"} +{"Name":"nexusrules.officeapps.live.com","RecordType":5,"TTL":228,"Type":"Answer","A":"prod.nexusrules.live.com.akadns.net"} +{"Name":"cdn4.mxpnl.com","RecordType":1,"TTL":329,"Type":"Answer","A":"130.211.5.208"} +{"Name":"cdn.mxpnl.com","RecordType":1,"TTL":36989,"Type":"Answer","A":"130.211.5.208"} +{"Name":"openx.adhaven.com","RecordType":1,"TTL":36983,"Type":"Answer","A":"35.244.216.234"} +{"Name":"dev.visualwebsiteoptimizer.com","RecordType":1,"TTL":25331,"Type":"Answer","A":"34.96.102.137"} +{"Name":"dc.magnumtempus.financial","RecordType":1,"TTL":584,"Type":"Answer","A":"172.16.50.100"} +{"Name":"www.lucyinthesky.com","RecordType":5,"TTL":3503,"Type":"Answer","A":"lucyinthesky.com"} +{"Name":"free-website-translation.com","RecordType":1,"TTL":1592,"Type":"Answer","A":"109.239.60.158"} +{"Name":"velociraptor.magnumtempusfinancial.com","RecordType":1,"TTL":2183,"Type":"Answer","A":"18.188.230.95"} \ No newline at end of file diff --git a/tests/testdata/velociraptor/Windows.System.Pslist.json b/tests/testdata/velociraptor/Windows.System.Pslist.json new file mode 100644 index 000000000..e5b8d3acd --- /dev/null +++ b/tests/testdata/velociraptor/Windows.System.Pslist.json @@ -0,0 +1,10 @@ +{"Pid":4,"Ppid":0,"TokenIsElevated":false,"Name":"System","CommandLine":"","Exe":"","Hash":null,"Authenticode":null,"Username":"","WorkingSetSize":143360} +{"Pid":268,"Ppid":4,"TokenIsElevated":true,"Name":"smss.exe","CommandLine":"\\SystemRoot\\System32\\smss.exe","Exe":"C:\\Windows\\System32\\smss.exe","Hash":{"MD5":"725ec50d4b0f607bf5b45b5e0115770b","SHA1":"c9c133660468fd1d9905f598f5052dbb01f42eea","SHA256":"56881bcaeac350107a6453f38f020fe0e284dbe2e8a6f37ed482985e0dd98ea7"},"Authenticode":{"Filename":"C:\\Windows\\System32\\smss.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000001affae16562041426770000000001af","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":1277952} +{"Pid":348,"Ppid":340,"TokenIsElevated":true,"Name":"csrss.exe","CommandLine":"%SystemRoot%\\system32\\csrss.exe ObjectDirectory=\\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16","Exe":"C:\\Windows\\System32\\csrss.exe","Hash":{"MD5":"955e9227aa30a08b7465c109b863b886","SHA1":"563338b189de230aedf51b69e6d1601fba40292d","SHA256":"d896480bc8523fad3ae152c81a2b572022c3778a34a6d85e089d150a68e9165e"},"Authenticode":{"Filename":"C:\\Windows\\System32\\csrss.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000001affae16562041426770000000001af","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":4292608} +{"Pid":416,"Ppid":340,"TokenIsElevated":true,"Name":"wininit.exe","CommandLine":"wininit.exe","Exe":"C:\\Windows\\System32\\wininit.exe","Hash":{"MD5":"5a998f811d7805b79b8e769027f62fd2","SHA1":"46bbe99e579e9cae86a249b556243c5cbbcd00b4","SHA256":"8694c5732d26921eea29589a9fa4182139ef3d9ea6b6d0acca8994b4aa5defe5"},"Authenticode":{"Filename":"C:\\Windows\\System32\\wininit.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"330000016b5af7a2a57141582700000000016b","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":5050368} +{"Pid":424,"Ppid":408,"TokenIsElevated":true,"Name":"csrss.exe","CommandLine":"%SystemRoot%\\system32\\csrss.exe ObjectDirectory=\\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16","Exe":"C:\\Windows\\System32\\csrss.exe","Hash":{"MD5":"955e9227aa30a08b7465c109b863b886","SHA1":"563338b189de230aedf51b69e6d1601fba40292d","SHA256":"d896480bc8523fad3ae152c81a2b572022c3778a34a6d85e089d150a68e9165e"},"Authenticode":{"Filename":"C:\\Windows\\System32\\csrss.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000001affae16562041426770000000001af","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":3760128} +{"Pid":492,"Ppid":408,"TokenIsElevated":true,"Name":"winlogon.exe","CommandLine":"winlogon.exe","Exe":"C:\\Windows\\System32\\winlogon.exe","Hash":{"MD5":"dea4ce12f24601830083126e18a2c7c9","SHA1":"39a7038115ad1e578b15dd9fcb7772c1a83a898e","SHA256":"f002f8c2ea49d21f242996e3d57f5fdd7995fe6db524bb69bbd7f190cc0211a9"},"Authenticode":{"Filename":"C:\\Windows\\System32\\winlogon.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002ed2c45e4c145cf48440000000002ed","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows","Timestamp":null,"Trusted":"trusted","_ExtraInfo":{"Catalog":"C:\\Windows\\system32\\CatRoot\\{F750E6C3-38EE-11D1-85E5-00C04FC295EE}\\Package_2212_for_KB5007192~31bf3856ad364e35~amd64~~10.0.1.8.cat"}},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":8302592} +{"Pid":540,"Ppid":416,"TokenIsElevated":true,"Name":"services.exe","CommandLine":"C:\\Windows\\system32\\services.exe","Exe":"C:\\Windows\\System32\\services.exe","Hash":{"MD5":"fefc26105685c70d7260170489b5b520","SHA1":"d9b2cb9bf9d4789636b5fcdef0fdbb9d8bc0fb52","SHA256":"930f44f9a599937bdb23cf0c7ea4d158991b837d2a0975c15686cdd4198808e8"},"Authenticode":{"Filename":"C:\\Windows\\System32\\services.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002a5e1a081b7c895c0ed0000000002a5","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":8151040} +{"Pid":548,"Ppid":416,"TokenIsElevated":true,"Name":"lsass.exe","CommandLine":"C:\\Windows\\system32\\lsass.exe","Exe":"C:\\Windows\\System32\\lsass.exe","Hash":{"MD5":"93212fd52a9cd5addad2fd2a779355d2","SHA1":"49a814f72292082a1cfdf602b5e4689b0f942703","SHA256":"95888daefd187fac9c979387f75ff3628548e7ddf5d70ad489cf996b9cad7193"},"Authenticode":{"Filename":"C:\\Windows\\System32\\lsass.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000002f49e469c54137b85e00000000002f4","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":26480640} +{"Pid":636,"Ppid":540,"TokenIsElevated":true,"Name":"svchost.exe","CommandLine":"C:\\Windows\\system32\\svchost.exe -k DcomLaunch","Exe":"C:\\Windows\\System32\\svchost.exe","Hash":{"MD5":"36f670d89040709013f6a460176767ec","SHA1":"0dac68816ae7c09efc24d11c27c3274dfd147dee","SHA256":"438b6ccd84f4dd32d9684ed7d58fd7d1e5a75fe3f3d12ab6c788e6bb0ffad5e7"},"Authenticode":{"Filename":"C:\\Windows\\System32\\svchost.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000000cc4ee86d1a15af49950000000000cc","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\SYSTEM","WorkingSetSize":21061632} +{"Pid":692,"Ppid":540,"TokenIsElevated":true,"Name":"svchost.exe","CommandLine":"C:\\Windows\\system32\\svchost.exe -k RPCSS","Exe":"C:\\Windows\\System32\\svchost.exe","Hash":{"MD5":"36f670d89040709013f6a460176767ec","SHA1":"0dac68816ae7c09efc24d11c27c3274dfd147dee","SHA256":"438b6ccd84f4dd32d9684ed7d58fd7d1e5a75fe3f3d12ab6c788e6bb0ffad5e7"},"Authenticode":{"Filename":"C:\\Windows\\System32\\svchost.exe","ProgramName":"Microsoft Windows","PublisherLink":null,"MoreInfoLink":"http://www.microsoft.com/windows","SerialNumber":"33000000cc4ee86d1a15af49950000000000cc","IssuerName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Production PCA 2011","SubjectName":"C=US, ST=Washington, L=Redmond, O=Microsoft Corporation, CN=Microsoft Windows Publisher","Timestamp":null,"Trusted":"trusted","_ExtraInfo":null},"Username":"NT AUTHORITY\\NETWORK SERVICE","WorkingSetSize":11083776} \ No newline at end of file From 9c53daf8b04b08224b1a7a1218ac7af6577babd8 Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Mon, 3 Jul 2023 15:38:10 -0700 Subject: [PATCH 18/26] Updating github checkout and upload-artifact to v3 (#669) --- .github/workflows/python-package.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/python-package.yml b/.github/workflows/python-package.yml index 41dca6790..0afff0962 100644 --- a/.github/workflows/python-package.yml +++ b/.github/workflows/python-package.yml @@ -28,7 +28,7 @@ jobs: JOB_CONTEXT: ${{ toJSON(job) }} run: echo "$JOB_CONTEXT" # end print details - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: @@ -82,7 +82,7 @@ jobs: pytest tests -n auto --junitxml=junit/test-${{ matrix.python-version }}-results.xml --cov=msticpy --cov-report=xml if: ${{ always() }} - name: Upload pytest test results - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v3 with: name: pytest-results-${{ matrix.python-version }} path: junit/test-${{ matrix.python-version }}-results.xml @@ -95,7 +95,7 @@ jobs: matrix: python-version: ["3.8"] steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v3 - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v4 with: @@ -159,7 +159,7 @@ jobs: mypy --ignore-missing-imports --follow-imports=silent --show-column-numbers --show-error-end --show-error-context --disable-error-code annotation-unchecked --junit-xml junit/mypy-test-${{ matrix.python-version }}-results.xml msticpy if: ${{ always() }} - name: Upload mypy test results - uses: actions/upload-artifact@v2 + uses: actions/upload-artifact@v3 with: name: Mypy results ${{ matrix.python-version }} path: junit/mypy-test-${{ matrix.python-version }}-results.xml From 75048627fd9e051399b306d9abccd95cb776ef9f Mon Sep 17 00:00:00 2001 From: Joey Dreijer Date: Thu, 6 Jul 2023 00:39:56 +0200 Subject: [PATCH 19/26] Added multithreading support for additional connections (+fixes) (#645) * Multithreading support when using multiple connections * Renamed additional connection column in results df * Fix flake warning * Adding threaded execution for both multiple instances and split queries for drivers that support multi-threading in query_provider_connections_mixin.py Added unit tests for threading code in test_async_queries.py Added driver properties to azure_kusto_driver.py, azure_monitor_driver.py and odata.py (mdatp_driver and security_graph_driver) Fixed test in test_azure_kusto_driver.py Some doc fixes to docstring in DataProv-Kusto-New.rst, DataProv-MSSentinel-New.rst, DataProviders.rst Unrelated doc fixes in polling_detection.py, Installing.rst, SentinelIncidents.rst * Fixing issue with unit_test_lib not properly isolating temporary settings changes * Adding locking around pivot data providers loader to fix config file for pivot tests. Changing test_nbinit.py to avoid using config locking and just use monkeypatch.setenv * Fixing some bugs in multi-threaded code - ensuring that loop is available if nested threading is happening. Converting pd.Timestamps to datetimes to allow serialization in Azure-azure_monitor_driver (in AZmon SDK) Fixing some logger info outputs in nbinit.py - that normally have no output. * Fxing handling of datetime/pd.Timestamp in query_provider_connections_mixin. Add more logging to data_providers.QueryProvider and azure_monitor_driver.py * Typo in data_providers (self.logger instead of logger) * Typo calling logger.info in data_providers.py Format of cluster name has changed in new KustoClient. Fixing test cases to allow for old and new format. * Cleaned up and refactored code in query_provider_connections_mixin.py * Typo in type annotation in query_provider_connections_mixin Reverted change to pop start and end parameters Fixed failing test in test_dataqueries.py::test_split_query_err * Removing redundant line in mdatp_driver * Bug in commit from merge - missing self._connection_str attribute in azure_kusto_driver.py --------- Co-authored-by: Ian Hellen --- .../msticpy.analysis.polling_detection.rst | 7 + docs/source/api/msticpy.analysis.rst | 1 + .../data_acquisition/DataProv-Kusto-New.rst | 22 +- .../DataProv-MSSentinel-New.rst | 29 +- .../data_acquisition/DataProv-OSQuery.rst | 2 +- .../source/data_acquisition/DataProviders.rst | 76 +++- .../data_acquisition/SentinelIncidents.rst | 6 +- docs/source/getting_started/Installing.rst | 22 +- msticpy/analysis/polling_detection.py | 5 - msticpy/data/core/data_providers.py | 145 +++----- .../core/query_provider_connections_mixin.py | 346 +++++++++++++++++- msticpy/data/drivers/azure_kusto_driver.py | 83 ++++- msticpy/data/drivers/azure_monitor_driver.py | 55 ++- msticpy/data/drivers/driver_base.py | 3 + msticpy/data/drivers/mdatp_driver.py | 10 +- msticpy/data/drivers/odata_driver.py | 7 +- msticpy/data/drivers/security_graph_driver.py | 8 +- msticpy/init/nbinit.py | 7 +- tests/data/drivers/test_azure_kusto_driver.py | 11 +- tests/data/test_async_queries.py | 164 +++++++++ tests/data/test_dataqueries.py | 7 +- 21 files changed, 837 insertions(+), 179 deletions(-) create mode 100644 docs/source/api/msticpy.analysis.polling_detection.rst create mode 100644 tests/data/test_async_queries.py diff --git a/docs/source/api/msticpy.analysis.polling_detection.rst b/docs/source/api/msticpy.analysis.polling_detection.rst new file mode 100644 index 000000000..b9c035c23 --- /dev/null +++ b/docs/source/api/msticpy.analysis.polling_detection.rst @@ -0,0 +1,7 @@ +msticpy.analysis.polling\_detection module +========================================== + +.. automodule:: msticpy.analysis.polling_detection + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/api/msticpy.analysis.rst b/docs/source/api/msticpy.analysis.rst index b17d5e56a..615bb314c 100644 --- a/docs/source/api/msticpy.analysis.rst +++ b/docs/source/api/msticpy.analysis.rst @@ -25,5 +25,6 @@ Submodules msticpy.analysis.eventcluster msticpy.analysis.observationlist msticpy.analysis.outliers + msticpy.analysis.polling_detection msticpy.analysis.syslog_utils msticpy.analysis.timeseries diff --git a/docs/source/data_acquisition/DataProv-Kusto-New.rst b/docs/source/data_acquisition/DataProv-Kusto-New.rst index 8974e82b7..5e2e7645e 100644 --- a/docs/source/data_acquisition/DataProv-Kusto-New.rst +++ b/docs/source/data_acquisition/DataProv-Kusto-New.rst @@ -20,19 +20,27 @@ Changes from the previous implementation * Use the provider name ``Kusto_New`` when creating a QueryProvider instance. This will be changed to ``Kusto`` in a future release. +* The driver supports asynchronous execution of queries. This is used + when you create a Query provider with multiple connections (e.g. + to different clusters) and when you split queries into time chunks. + See :ref:`multiple_connections` and :ref:`splitting_query_execution` for + for more details. * The settings format has changed (although the existing format is still supported albeit with some limited functionality). +* Supports user-specified timeout for queries. +* Supports proxies (via MSTICPy config or the ``proxies`` parameter to + the ``connect`` method) * You could previously specify a new cluster to connect to in when executing a query. This is no longer supported. Once the provider is connected to a cluster it will only execute queries against - that cluster. (You can however, call the connect() function to connect + that cluster. (You can however, call the ``connect()`` function to connect the provider to a new cluster before running the query.) * Some of the previous parameters have been deprecated: - * ``mp_az_auth`` is replaced by ``auth_types`` (the former still works + * The ``mp_az_auth`` parameter is replaced by ``auth_types`` (the former still works but will be removed in a future release). * ``mp_az_auth_tenant_id`` is replaced by ``tenant_id`` (the former - is no longer supported + is no longer supported). Kusto Configuration ------------------- @@ -49,9 +57,9 @@ and :doc:`MSTICPy Settings Editor<../getting_started/SettingsEditor>` .. note:: The settings for the new Kusto provider are stored in the ``KustoClusters`` section of the configuration file. This cannot currently be edited from the MSTICPy Settings Editor - please - edit the *msticpyconfig.yaml* directly to edit these. + edit the *msticpyconfig.yaml* in a text editor to change these. -To accommodate the use of multiple clusters the new provider supports +To accommodate the use of multiple clusters, the new provider supports a different configuration format. The basic settings in the file should look like the following: @@ -92,7 +100,7 @@ for *clientsecret* authentication. The ClusterDefaults section ~~~~~~~~~~~~~~~~~~~~~~~~~~~ -If you have parameters that you want to apply to all clusters +If you have parameters that you want to apply to all clusters, you can add these to a ``ClusterDefaults`` section. .. code:: yaml @@ -117,7 +125,7 @@ cluster group name. This is useful if you have clusters in different regions that share the same schema and you want to run the same queries against all of them. -This is used primarily to support query templates, to match +ClusterGroups are used primarily to support query templates, to match queries to the correct cluster. See `Writing query templates for Kusto clusters`_ later in this document. diff --git a/docs/source/data_acquisition/DataProv-MSSentinel-New.rst b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst index c0417cb02..4abe5ccab 100644 --- a/docs/source/data_acquisition/DataProv-MSSentinel-New.rst +++ b/docs/source/data_acquisition/DataProv-MSSentinel-New.rst @@ -7,7 +7,7 @@ the (the earlier implementation used `Kqlmagic `__) -.. note:: This provider currently in beta and is available for testing. +.. warning:: This provider currently in beta and is available for testing. It is available alongside the existing Sentinel provider for you to compare old and new. To use it you will need the ``azure-monitor-query`` package installed. You can install this with ``pip install azure-monitor-query`` @@ -25,6 +25,12 @@ Changes from the previous implementation * Supports user-specified timeout for queries. * Supports proxies (via MSTICPy config or the ``proxies`` parameter to the ``connect`` method) +* The driver supports asynchronous execution of queries. This is used + when you create a Query provider with multiple connections (e.g. + to different clusters) and when you split queries into time chunks. + See :ref:`multiple_connections` and :ref:`splitting_query_execution` for + for more details. This is independent of the ability to specify + multiple workspaces in a single connection as described above. * Some of the previous parameters have been deprecated: * ``mp_az_auth`` is replaced by ``auth_types`` (the former still works @@ -129,7 +135,8 @@ Connecting to a MS Sentinel Workspace Once you've created a QueryProvider you need to authenticate to Sentinel Workspace. This is done by calling the connect() function of the Query -Provider. See :py:meth:`connect() ` +Provider. See +:py:meth:`connect() ` This function takes an initial parameter (called ``connection_str`` for historical reasons) that can be one of the following: @@ -160,7 +167,6 @@ an instance of WorkspaceConfig to the query provider's ``connect`` method. qry_prov.connect(WorkspaceConfig(workspace="MyOtherWorkspace")) - MS Sentinel Authentication options ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -177,15 +183,26 @@ You can override several authentication parameters including: * tenant_id - the Azure tenant ID to use for authentication If you are using a Sovereign cloud rather than the Azure global cloud, -you should follow the guidance in :doc:`Azure Authentication <../getting_started/AzureAuthentication>` +you should follow the guidance in +:doc:`Azure Authentication <../getting_started/AzureAuthentication>` to configure the correct cloud. + Connecting to multiple Sentinel workspaces ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The Sentinel data provider supports connecting to multiple workspaces. -You can pass a list of workspace names or workspace IDs to the ``connect`` method. +There are two mechanisms for querying multiple MS Sentinel workspaces. +One is a generic method common to all data providers. For more +information on this see :ref:`multiple_connections` in the main +Data Providers documentation. + +The other is specific to the Sentinel data provider and is provided +by the underlying Azure Monitor client. This latter capability is described in +this section. + +The Sentinel data provider supports connecting to multiple workspaces by +passing a list of workspace names or workspace IDs to the ``connect`` method. using the ``workspaces`` or ``workspace_ids`` parameters respectively. ``workspace_ids`` should be a list or tuple of workspace IDs. diff --git a/docs/source/data_acquisition/DataProv-OSQuery.rst b/docs/source/data_acquisition/DataProv-OSQuery.rst index 4f17b0d79..b6895ea9a 100644 --- a/docs/source/data_acquisition/DataProv-OSQuery.rst +++ b/docs/source/data_acquisition/DataProv-OSQuery.rst @@ -152,7 +152,7 @@ from the logs. qry_prov.osquery.processes() ================================== ================ ========================= ===== ========== ========= ====== ======== ======== ===== ========== -name hostIdentifier unixTime ... username cmdline euid name_ parent uid username +name hostIdentifier unixTime ... username cmdline euid tname_ parent uid username ================================== ================ ========================= ===== ========== ========= ====== ======== ======== ===== ========== pack_osquery-custom-pack_processes jumpvm 2023-03-16 03:08:58+00:00 ... LOGIN 0 kthreadd 2 0 root pack_osquery-custom-pack_processes jumpvm 2023-03-16 03:08:58+00:00 ... LOGIN 0 kthreadd 2 0 root diff --git a/docs/source/data_acquisition/DataProviders.rst b/docs/source/data_acquisition/DataProviders.rst index ddd229f6c..65f6561a1 100644 --- a/docs/source/data_acquisition/DataProviders.rst +++ b/docs/source/data_acquisition/DataProviders.rst @@ -449,6 +449,75 @@ TimeGenerated AlertDisplayName Severity 2019-07-22 07:02:42 Traffic from unrecommended IP addresses was de... Low Azure security center has detected incoming tr... {\r\n "Destination Port": "3389",\r\n "Proto... [\r\n {\r\n "$id": "4",\r\n "ResourceId... Detection =================== ================================================= ========== ================================================= ================================================ ========================================== ============== +.. _multiple_connections: + +Running a query across multiple connections +------------------------------------------- + +It is common for data services to be spread across multiple tenants or +workloads. For example, you may have multiple Sentinel workspaces, +Microsoft Defender subscriptions or Splunk instances. You can use the +``QueryProvider`` to run a query across multiple connections and return +the results in a single DataFrame. + +.. note:: This feature only works for multiple instances using the same + ``DataEnvironment`` (e.g. "MSSentinel", "Splunk", etc.) + +To create a multi-instance provider you first need to create an +instance of a QueryProvider for your data source and execute +the ``connect()`` method to connect to the first instance of your +data service. Which instance you choose is not important. +Then use the +:py:meth:`add_connection() ` +method. This takes the same parameters as the +:py:meth:`connect() ` +method (the parameters for this method vary by data provider). + +``add_connection()`` also supports an ``alias`` parameter to allow +you to refer to the connection by a friendly name. Otherwise, the +connection is just assigned an index number in the order that it was +added. + +Use the +:py:meth:`list_connections() ` +to see all of the current connections. + +.. code:: ipython3 + + qry_prov = QueryProvider("MSSentinel") + qry_prov.connect(workspace="Workspace1") + qry_prov.add_connection(workspace="Workspace2, alias="Workspace2") + qry_prov.list_connections() + +When you now run a query for this provider, the query will be run on +all of the connections and the results will be returned as a single +dataframe. + +.. code:: ipython3 + + test_query = ''' + SecurityAlert + | take 5 + ''' + + query_test = qry_prov.exec_query(query=test_query) + query_test.head() + +Some of the MSTICPy drivers support asynchronous execution of queries +against multiple instances, so that the time taken to run the query is +much reduced compared to running the queries sequentially. Drivers +that support asynchronous queries will use this automatically. + +By default, the queries will use at most 4 concurrent threads. You can +override this by initializing the QueryProvider with the +``max_threads`` parameter to set it to the number of threads you want. + +.. code:: ipython3 + + qry_prov = QueryProvider("MSSentinel", max_threads=10) + + +.. _splitting_query_execution: Splitting Query Execution into Chunks ------------------------------------- @@ -459,6 +528,11 @@ split a query into time ranges. Each sub-range is run as an independent query and the results are combined before being returned as a DataFrame. +.. note:: Some data drivers support running queries asynchronously. + This means that the time taken to run all chunks of the query is much reduced + compared to running these sequentially. Drivers that support + asynchronous queries will use this automatically. + To use this feature you must specify the keyword parameter ``split_query_by`` when executing the query function. The value to this parameter is a string that specifies a time period. The time range specified by the @@ -471,7 +545,7 @@ chunks. than the split period that you specified in the *split_query_by* parameter. This can happen if *start* and *end* are not aligned exactly on time boundaries (e.g. if you used a one hour split period - and *end* is 10 hours 15 min after *start*. The query split logic + and *end* is 10 hours 15 min after *start*). The query split logic will create a larger final slice if *end* is close to the final time range or it will insert an extra time range to ensure that the full *start** to *end* time range is covered. diff --git a/docs/source/data_acquisition/SentinelIncidents.rst b/docs/source/data_acquisition/SentinelIncidents.rst index 03435270e..da4d68610 100644 --- a/docs/source/data_acquisition/SentinelIncidents.rst +++ b/docs/source/data_acquisition/SentinelIncidents.rst @@ -16,10 +16,12 @@ See :py:meth:`list_incidents `_ and include the following key items: - $top: this controls how many incidents are returned - - $filter: this accepts an OData query that filters the returned item. https://learn.microsoft.com/graph/filter-query-parameter + - $filter: this accepts an OData query that filters the returned item. + (see `$filter parameter `_) - $orderby: this allows for sorting results by a specific column .. code:: ipython3 diff --git a/docs/source/getting_started/Installing.rst b/docs/source/getting_started/Installing.rst index faac0004e..b9c079d41 100644 --- a/docs/source/getting_started/Installing.rst +++ b/docs/source/getting_started/Installing.rst @@ -201,22 +201,24 @@ exception message: functionality you are trying to use. Installing in Managed Spark compute in Azure Machine Learning Notebooks -^^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -*MSTICPy* installation for Managed (Automatic) Spark Compute in Azure Machine Learning workspace requires -different instructions since library installation is different. +*MSTICPy* installation for Managed (Automatic) Spark Compute in Azure Machine Learning workspace requires +different instructions since library installation is different. -.. note:: These notebook requires Azure ML Spark Compute. If you are using it for the first time, follow the guidelines mentioned here :Attach and manage a Synapse Spark pool in Azure Machine Learning (preview): -.. _Attach and manage a Synapse Spark pool in Azure Machine Learning (preview): -https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-synapse-spark-pool?tabs=studio-ui +.. note:: These notebook requires Azure ML Spark Compute. If you are using + it for the first time, follow the guidelines mentioned here: + `Attach and manage a Synapse Spark pool in Azure Machine Learning (preview) `_ Once you have completed the pre-requisites, you will see AzureML Spark Compute in the dropdown menu for Compute. Select it and run any cell to start Spark Session. -Please refer the docs _Managed (Automatic) Spark compute in Azure Machine Learning Notebooks: for more detailed steps along with screenshots. -.. _Managed (Automatic) Spark compute in Azure Machine Learning Notebooks: -https://learn.microsoft.com/en-us/azure/machine-learning/interactive-data-wrangling-with-apache-spark-azure-ml +Please refer to +`Managed (Automatic) Spark compute in Azure Machine Learning Notebooks `_ +for more detailed steps along with screenshots. + + -In order to install any libraries in Spark compute, you need to use a conda file to configure a Spark session. +In order to install any libraries in Spark compute, you need to use a conda file to configure a Spark session. Please save below file as conda.yml , check the Upload conda file checkbox. You can modify the version number as needed. Then, select Browse, and choose the conda file saved earlier with the Spark session configuration you want. diff --git a/msticpy/analysis/polling_detection.py b/msticpy/analysis/polling_detection.py index f1803a2ac..9f35b5eb2 100644 --- a/msticpy/analysis/polling_detection.py +++ b/msticpy/analysis/polling_detection.py @@ -35,11 +35,6 @@ class PeriodogramPollingDetector: Dataframe containing the data to be analysed. Must contain a column of edges and a column of timestamps - Methods - ------- - detect_polling(timestamps, process_start, process_end, interval) - Detect strong periodic frequencies - """ def __init__(self, data: pd.DataFrame, copy: bool = False) -> None: diff --git a/msticpy/data/core/data_providers.py b/msticpy/data/core/data_providers.py index cbd681821..66a86157c 100644 --- a/msticpy/data/core/data_providers.py +++ b/msticpy/data/core/data_providers.py @@ -4,14 +4,12 @@ # license information. # -------------------------------------------------------------------------- """Data provider loader.""" -from datetime import datetime +import logging from functools import partial -from itertools import tee from pathlib import Path from typing import Any, Dict, Iterable, List, Optional, Tuple, Union import pandas as pd -from tqdm.auto import tqdm from ..._version import VERSION from ...common.pkg_config import get_config @@ -24,7 +22,6 @@ from .query_defns import DataEnvironment from .query_provider_connections_mixin import QueryProviderConnectionsMixin from .query_provider_utils_mixin import QueryProviderUtilsMixin -from .query_source import QuerySource from .query_store import QueryStore __version__ = VERSION @@ -39,6 +36,8 @@ "kusto_new": ["kusto"], } +logger = logging.getLogger(__name__) + # These are mixin classes that do not have an __init__ method # pylint: disable=super-init-not-called @@ -114,6 +113,8 @@ def __init__( # noqa: MC0001 driver.get_driver_property(DriverProps.EFFECTIVE_ENV) or self.environment_name ) + logger.info("Using data environment %s", self.environment_name) + logger.info("Driver class: %s", self.driver_class.__name__) self._additional_connections: Dict[str, DriverBase] = {} self._query_provider = driver @@ -124,15 +125,19 @@ def __init__( # noqa: MC0001 # Add any query files data_env_queries: Dict[str, QueryStore] = {} + self._query_paths = query_paths if driver.use_query_paths: + logger.info("Using query paths %s", query_paths) data_env_queries.update( self._read_queries_from_paths(query_paths=query_paths) ) self.query_store = data_env_queries.get( self.environment_name, QueryStore(self.environment_name) ) + logger.info("Adding query functions to provider") self._add_query_functions() self._query_time = QueryTime(units="day") + logger.info("Initialization complete.") def _check_environment( self, data_environment @@ -179,6 +184,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): Connection string for the data source """ + logger.info("Calling connect on driver") self._query_provider.connect(connection_str=connection_str, **kwargs) # If the driver has any attributes to expose via the provider @@ -194,6 +200,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): refresh_query_funcs = True # Add any built-in or dynamically retrieved queries from driver if self._query_provider.has_driver_queries: + logger.info("Adding driver queries to provider") driver_queries = self._query_provider.driver_queries self._add_driver_queries(queries=driver_queries) refresh_query_funcs = True @@ -202,6 +209,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): self._add_query_functions() # Since we're now connected, add Pivot functions + logger.info("Adding query pivot functions") self._add_pivots(lambda: self._query_time.timespan) def exec_query(self, query: str, **kwargs) -> Union[pd.DataFrame, Any]: @@ -230,12 +238,13 @@ def exec_query(self, query: str, **kwargs) -> Union[pd.DataFrame, Any]: """ query_options = kwargs.pop("query_options", {}) or kwargs query_source = kwargs.pop("query_source", None) - result = self._query_provider.query( - query, query_source=query_source, **query_options - ) + + logger.info("Executing query '%s...'", query[:40]) if not self._additional_connections: - return result - return self._exec_additional_connections(query, result, **kwargs) + return self._query_provider.query( + query, query_source=query_source, **query_options + ) + return self._exec_additional_connections(query, **kwargs) @property def query_time(self): @@ -265,6 +274,7 @@ def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: return None params, missing = extract_query_params(query_source, *args, **kwargs) + logger.info("Parameters for query: %s", params) query_options = { "default_time_params": self._check_for_time_params(params, missing) } @@ -272,12 +282,14 @@ def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: query_source.help() raise ValueError(f"No values found for these parameters: {missing}") - split_by = kwargs.pop("split_query_by", None) + split_by = kwargs.pop("split_query_by", kwargs.pop("split_by", None)) if split_by: + logger.info("Split query selected - interval - %s", split_by) split_result = self._exec_split_query( split_by=split_by, query_source=query_source, query_params=params, + debug=_debug_flag(*args, **kwargs), args=args, **kwargs, ) @@ -292,7 +304,10 @@ def _execute_query(self, *args, **kwargs) -> Union[pd.DataFrame, Any]: return query_str # Handle any query options passed and run the query - query_options.update(_get_query_options(params, kwargs)) + query_options.update(self._get_query_options(params, kwargs)) + logger.info( + "Running query '%s...' with params: %s", query_str[:40], query_options + ) return self.exec_query(query_str, query_source=query_source, **query_options) def _check_for_time_params(self, params, missing) -> bool: @@ -342,6 +357,7 @@ def _read_queries_from_paths(self, query_paths) -> Dict[str, QueryStore]: if param_qry_path: all_query_paths.append(param_qry_path) if all_query_paths: + logger.info("Reading queries from %s", all_query_paths) return QueryStore.import_files( source_path=all_query_paths, recursive=True, @@ -395,80 +411,23 @@ def _add_driver_queries(self, queries: Iterable[Dict[str, str]]): # queries it should not be noticeable. self._add_query_functions() - def _exec_split_query( - self, - split_by: str, - query_source: QuerySource, - query_params: Dict[str, Any], - args, - **kwargs, - ) -> Union[pd.DataFrame, str, None]: - start = query_params.pop("start", None) - end = query_params.pop("end", None) - if not (start or end): - print( - "Cannot split a query that does not have 'start' and 'end' parameters" - ) - return None - try: - split_delta = pd.Timedelta(split_by) - except ValueError: - split_delta = pd.Timedelta("1D") - - ranges = _calc_split_ranges(start, end, split_delta) - - split_queries = [ - query_source.create_query( - formatters=self._query_provider.formatters, - start=q_start, - end=q_end, - **query_params, - ) - for q_start, q_end in ranges - ] - # This looks for any of the "print query" debug args in args or kwargs - if _debug_flag(*args, **kwargs): - return "\n\n".join(split_queries) - - # Retrieve any query options passed (other than query params) - # and send to query function. - query_options = _get_query_options(query_params, kwargs) - query_dfs = [ - self.exec_query(query_str, query_source=query_source, **query_options) - for query_str in tqdm(split_queries, unit="sub-queries", desc="Running") - ] - - return pd.concat(query_dfs) - - -def _calc_split_ranges(start: datetime, end: datetime, split_delta: pd.Timedelta): - """Return a list of time ranges split by `split_delta`.""" - # Use pandas date_range and split the result into 2 iterables - s_ranges, e_ranges = tee(pd.date_range(start, end, freq=split_delta)) - next(e_ranges, None) # skip to the next item in the 2nd iterable - # Zip them together to get a list of (start, end) tuples of ranges - # Note: we subtract 1 nanosecond from the 'end' value of each range so - # to avoid getting duplicated records at the boundaries of the ranges. - # Some providers don't have nanosecond granularity so we might - # get duplicates in these cases - ranges = [ - (s_time, e_time - pd.Timedelta("1ns")) - for s_time, e_time in zip(s_ranges, e_ranges) - ] - - # Since the generated time ranges are based on deltas from 'start' - # we need to adjust the end time on the final range. - # If the difference between the calculated last range end and - # the query 'end' that the user requested is small (< 10% of a delta), - # we just replace the last "end" time with our query end time. - if (ranges[-1][1] - end) < (split_delta / 10): - ranges[-1] = ranges[-1][0], end - else: - # otherwise append a new range starting after the last range - # in ranges and ending in 'end" - # note - we need to add back our subtracted 1 nanosecond - ranges.append((ranges[-1][0] + pd.Timedelta("1ns"), end)) - return ranges + @staticmethod + def _get_query_options( + params: Dict[str, Any], kwargs: Dict[str, Any] + ) -> Dict[str, Any]: + # sourcery skip: inline-immediately-returned-variable, use-or-for-fallback + """Return any kwargs not already in params.""" + query_options = kwargs.pop("query_options", {}) + if not query_options: + # Any kwargs left over we send to the query provider driver + query_options = { + key: val for key, val in kwargs.items() if key not in params + } + query_options["time_span"] = { + "start": params.get("start"), + "end": params.get("end"), + } + return query_options def _resolve_package_path(config_path: str) -> Path: @@ -500,19 +459,3 @@ def _debug_flag(*args, **kwargs) -> bool: return any(db_arg for db_arg in _DEBUG_FLAGS if db_arg in args) or any( db_arg for db_arg in _DEBUG_FLAGS if kwargs.get(db_arg, False) ) - - -def _get_query_options( - params: Dict[str, Any], kwargs: Dict[str, Any] -) -> Dict[str, Any]: - # sourcery skip: inline-immediately-returned-variable, use-or-for-fallback - """Return any kwargs not already in params.""" - query_options = kwargs.pop("query_options", {}) - if not query_options: - # Any kwargs left over we send to the query provider driver - query_options = {key: val for key, val in kwargs.items() if key not in params} - query_options["time_span"] = { - "start": params.get("start"), - "end": params.get("end"), - } - return query_options diff --git a/msticpy/data/core/query_provider_connections_mixin.py b/msticpy/data/core/query_provider_connections_mixin.py index e436a3144..b40a757bd 100644 --- a/msticpy/data/core/query_provider_connections_mixin.py +++ b/msticpy/data/core/query_provider_connections_mixin.py @@ -4,19 +4,29 @@ # license information. # -------------------------------------------------------------------------- """Query Provider additional connection methods.""" -from typing import Any, Dict, List, Optional, Protocol +import asyncio +import logging +from concurrent.futures import ThreadPoolExecutor +from datetime import datetime +from functools import partial +from itertools import tee +from typing import Any, Dict, List, Optional, Protocol, Tuple, Union import pandas as pd +from tqdm.auto import tqdm from ..._version import VERSION from ...common.exceptions import MsticpyDataQueryError -from ..drivers.driver_base import DriverBase +from ..drivers.driver_base import DriverBase, DriverProps +from .query_source import QuerySource __version__ = VERSION __author__ = "Ian Hellen" +logger = logging.getLogger(__name__) -# pylint: disable=too-few-public-methods + +# pylint: disable=too-few-public-methods, unnecessary-ellipsis class QueryProviderProtocol(Protocol): """Protocol for required properties of QueryProvider class.""" @@ -25,6 +35,16 @@ class QueryProviderProtocol(Protocol): _additional_connections: Dict[str, Any] _query_provider: DriverBase + def exec_query(self, query: str, **kwargs) -> Union[pd.DataFrame, Any]: + """Execute a query against the provider.""" + ... + + @staticmethod + def _get_query_options( + params: Dict[str, Any], kwargs: Dict[str, Any] + ) -> Dict[str, Any]: + ... + # pylint: disable=super-init-not-called class QueryProviderConnectionsMixin(QueryProviderProtocol): @@ -49,7 +69,7 @@ def add_connection( Other Parameters ---------------- kwargs : Dict[str, Any] - Other parameters passed to the driver constructor. + Other connection parameters passed to the driver. Notes ----- @@ -67,7 +87,7 @@ def add_connection( def list_connections(self) -> List[str]: """ - Return a list of current connections or the default connection. + Return a list of current connections. Returns ------- @@ -81,18 +101,314 @@ def list_connections(self) -> List[str]: ] return [f"Default: {self._query_provider.current_connection}", *add_connections] - def _exec_additional_connections(self, query, result, **kwargs) -> pd.DataFrame: - """Return results of query run query against additional connections.""" - query_source = kwargs.get("query_source") - query_options = kwargs.get("query_options", {}) - results = [result] + # pylint: disable=too-many-locals + def _exec_additional_connections(self, query, **kwargs) -> pd.DataFrame: + """ + Return results of query run query against additional connections. + + Parameters + ---------- + query : str + The query to execute. + progress: bool, optional + Show progress bar, by default True + retry_on_error: bool, optional + Retry failed queries, by default False + **kwargs : Dict[str, Any] + Additional keyword arguments to pass to the query method. + + Returns + ------- + pd.DataFrame + The concatenated results of the query executed against all connections. + + Notes + ----- + This method executes the specified query against all additional connections + added to the query provider. + If the driver supports threading or async execution, the per-connection + queries are executed asynchronously. + Otherwise, the queries are executed sequentially. + + """ + progress = kwargs.pop("progress", True) + retry = kwargs.pop("retry_on_error", False) + # Add the initial connection + query_tasks = { + self._query_provider.current_connection + or "0": partial( + self._query_provider.query, + query, + **kwargs, + ) + } + # add the additional connections + query_tasks.update( + { + name: partial(connection.query, query, **kwargs) + for name, connection in self._additional_connections.items() + } + ) + + logger.info("Running queries for %s connections.", len(query_tasks)) + # Run the queries threaded if supported + if self._query_provider.get_driver_property(DriverProps.SUPPORTS_THREADING): + logger.info("Running threaded queries.") + event_loop = _get_event_loop() + return event_loop.run_until_complete( + self._exec_queries_threaded(query_tasks, progress, retry) + ) + + # standard synchronous execution print(f"Running query for {len(self._additional_connections)} connections.") - for con_name, connection in self._additional_connections.items(): - print(f"{con_name}...") + return self._exec_synchronous_queries(progress, query_tasks) + + def _exec_split_query( + self, + split_by: str, + query_source: QuerySource, + query_params: Dict[str, Any], + **kwargs, + ) -> Union[pd.DataFrame, str, None]: + """ + Execute a query that is split into multiple queries. + + Parameters + ---------- + split_by : str + The time interval to split the query by. + query_source : QuerySource + The query to execute. + query_params : Dict[str, Any] + The parameters to pass to the query. + + Other Parameters + ---------------- + debug: bool, optional + Return queries to be executed rather than execute them, by default False + progress: bool, optional + Show progress bar, by default True + retry_on_error: bool, optional + Retry failed queries, by default False + **kwargs : Dict[str, Any] + Additional keyword arguments to pass to the query method. + + Returns + ------- + pd.DataFrame + The concatenated results of the query executed against all connections. + + Notes + ----- + This method executes the time-chunks of the split query. + If the driver supports threading or async execution, the sub-queries are + executed asynchronously. Otherwise, the queries are executed sequentially. + + """ + start = query_params.pop("start", None) + end = query_params.pop("end", None) + progress = kwargs.pop("progress", True) + retry = kwargs.pop("retry_on_error", False) + debug = kwargs.pop("debug", False) + if not (start or end): + print("Cannot split a query with no 'start' and 'end' parameters") + return None + + split_queries = self._create_split_queries( + query_source=query_source, + query_params=query_params, + start=start, + end=end, + split_by=split_by, + ) + if debug: + return "\n\n".join( + f"{start}-{end}\n{query}" + for (start, end), query in split_queries.items() + ) + + query_tasks = self._create_split_query_tasks( + query_source, query_params, split_queries, **kwargs + ) + # Run the queries threaded if supported + if self._query_provider.get_driver_property(DriverProps.SUPPORTS_THREADING): + logger.info("Running threaded queries.") + event_loop = _get_event_loop() + return event_loop.run_until_complete( + self._exec_queries_threaded(query_tasks, progress, retry) + ) + + # or revert to standard synchronous execution + return self._exec_synchronous_queries(progress, query_tasks) + + def _create_split_query_tasks( + self, + query_source: QuerySource, + query_params: Dict[str, Any], + split_queries, + **kwargs, + ) -> Dict[str, partial]: + """Return dictionary of partials to execute queries.""" + # Retrieve any query options passed (other than query params) + query_options = self._get_query_options(query_params, kwargs) + logger.info("query_options: %s", query_options) + logger.info("kwargs: %s", kwargs) + if "time_span" in query_options: + del query_options["time_span"] + return { + f"{start}-{end}": partial( + self.exec_query, + query=query_str, + query_source=query_source, + time_span={"start": start, "end": end}, + **query_options, + ) + for (start, end), query_str in split_queries.items() + } + + @staticmethod + def _exec_synchronous_queries( + progress: bool, query_tasks: Dict[str, Any] + ) -> pd.DataFrame: + logger.info("Running queries sequentially.") + results: List[pd.DataFrame] = [] + if progress: + query_iter = tqdm(query_tasks.items(), unit="sub-queries", desc="Running") + else: + query_iter = query_tasks.items() + for con_name, query_task in query_iter: try: - results.append( - connection.query(query, query_source=query_source, **query_options) - ) + results.append(query_task()) except MsticpyDataQueryError: print(f"Query {con_name} failed.") return pd.concat(results) + + def _create_split_queries( + self, + query_source: QuerySource, + query_params: Dict[str, Any], + start: datetime, + end: datetime, + split_by: str, + ) -> Dict[Tuple[datetime, datetime], str]: + """Return separate queries for split time ranges.""" + try: + split_delta = pd.Timedelta(split_by) + except ValueError: + split_delta = pd.Timedelta("1D") + logger.info("Using split delta %s", split_delta) + + ranges = _calc_split_ranges(start, end, split_delta) + + split_queries = { + (q_start, q_end): query_source.create_query( + formatters=self._query_provider.formatters, + start=q_start, + end=q_end, + **query_params, + ) + for q_start, q_end in ranges + } + logger.info("Split query into %s chunks", len(split_queries)) + return split_queries + + async def _exec_queries_threaded( + self, + query_tasks: Dict[str, partial], + progress: bool = True, + retry: bool = False, + ) -> pd.DataFrame: + """Return results of multiple queries run as threaded tasks.""" + logger.info("Running threaded queries for %d connections.", len(query_tasks)) + + event_loop = _get_event_loop() + + with ThreadPoolExecutor( + max_workers=self._query_provider.get_driver_property( + DriverProps.MAX_PARALLEL + ) + ) as executor: + # add the additional connections + thread_tasks = { + query_id: event_loop.run_in_executor(executor, query_func) + for query_id, query_func in query_tasks.items() + } + results: List[pd.DataFrame] = [] + failed_tasks: Dict[str, asyncio.Future] = {} + if progress: + task_iter = tqdm( + asyncio.as_completed(thread_tasks.values()), + unit="sub-queries", + desc="Running", + ) + else: + task_iter = asyncio.as_completed(thread_tasks.values()) + ids_and_tasks = dict(zip(thread_tasks, task_iter)) + for query_id, thread_task in ids_and_tasks.items(): + try: + result = await thread_task + logger.info("Query task '%s' completed successfully.", query_id) + results.append(result) + except Exception: # pylint: disable=broad-except + logger.warning( + "Query task '%s' failed with exception", query_id, exc_info=True + ) + failed_tasks[query_id] = thread_task + + if retry and failed_tasks: + for query_id, thread_task in failed_tasks.items(): + try: + logger.info("Retrying query task '%s'", query_id) + result = await thread_task + results.append(result) + except Exception: # pylint: disable=broad-except + logger.warning( + "Retried query task '%s' failed with exception", + query_id, + exc_info=True, + ) + # Sort the results by the order of the tasks + results = [result for _, result in sorted(zip(thread_tasks, results))] + + return pd.concat(results, ignore_index=True) + + +def _get_event_loop() -> asyncio.AbstractEventLoop: + """Return the current event loop, or create a new one.""" + try: + loop = asyncio.get_running_loop() + except RuntimeError: + loop = asyncio.new_event_loop() + asyncio.set_event_loop(loop) + return loop + + +def _calc_split_ranges(start: datetime, end: datetime, split_delta: pd.Timedelta): + """Return a list of time ranges split by `split_delta`.""" + # Use pandas date_range and split the result into 2 iterables + s_ranges, e_ranges = tee(pd.date_range(start, end, freq=split_delta)) + next(e_ranges, None) # skip to the next item in the 2nd iterable + # Zip them together to get a list of (start, end) tuples of ranges + # Note: we subtract 1 nanosecond from the 'end' value of each range so + # to avoid getting duplicated records at the boundaries of the ranges. + # Some providers don't have nanosecond granularity so we might + # get duplicates in these cases + ranges = [ + (s_time, e_time - pd.Timedelta("1ns")) + for s_time, e_time in zip(s_ranges, e_ranges) + ] + + # Since the generated time ranges are based on deltas from 'start' + # we need to adjust the end time on the final range. + # If the difference between the calculated last range end and + # the query 'end' that the user requested is small (< 10% of a delta), + # we just replace the last "end" time with our query end time. + if (ranges[-1][1] - end) < (split_delta / 10): + ranges[-1] = ranges[-1][0], end + else: + # otherwise append a new range starting after the last range + # in ranges and ending in 'end" + # note - we need to add back our subtracted 1 nanosecond + ranges.append((ranges[-1][0] + pd.Timedelta("1ns"), end)) + + return ranges diff --git a/msticpy/data/drivers/azure_kusto_driver.py b/msticpy/data/drivers/azure_kusto_driver.py index 815787a69..160cb14a5 100644 --- a/msticpy/data/drivers/azure_kusto_driver.py +++ b/msticpy/data/drivers/azure_kusto_driver.py @@ -4,6 +4,7 @@ # license information. # -------------------------------------------------------------------------- """Kusto Driver subclass.""" +import base64 import dataclasses import json import logging @@ -17,6 +18,8 @@ KustoClient, KustoConnectionStringBuilder, ) +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.serialization import pkcs12 from ..._version import VERSION from ...auth.azure_auth import az_connect, get_default_resource_name @@ -34,6 +37,7 @@ from ..core.query_source import QuerySource from .driver_base import DriverBase, DriverProps +# pylint: disable=ungrouped-imports try: from azure.kusto.data.exceptions import KustoApiError, KustoServiceError from azure.kusto.data.helpers import dataframe_from_result_table @@ -78,6 +82,7 @@ class ConfigFields: CLIENT_SEC = "ClientSecret" ARGS = "Args" CLUSTER_GROUPS = "ClusterGroups" + CERTIFICATE = "Certificate" # pylint: disable=no-member @property @@ -174,6 +179,10 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): self.set_driver_property(DriverProps.PUBLIC_ATTRS, self._set_public_attribs()) self.set_driver_property(DriverProps.FILTER_ON_CONNECT, True) self.set_driver_property(DriverProps.EFFECTIVE_ENV, DataEnvironment.Kusto.name) + self.set_driver_property(DriverProps.SUPPORTS_THREADING, value=True) + self.set_driver_property( + DriverProps.MAX_PARALLEL, value=kwargs.get("max_threads", 4) + ) self._loaded = True def _set_public_attribs(self): @@ -204,16 +213,12 @@ def current_connection(self, value: str): @property def cluster_uri(self) -> str: """Return current cluster URI.""" - if not self._current_config: - return "" - return self._current_config.cluster + return "" if not self._current_config else self._current_config.cluster @property def cluster_name(self) -> str: """Return current cluster URI.""" - if not self._current_config: - return "" - return self._current_config.name + return self._current_config.name if self._current_config else "" @property def cluster_config_name(self) -> str: @@ -314,6 +319,7 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): ) cluster = kwargs.pop("cluster", None) + self.current_connection = connection_str or self.current_connection if not connection_str and not cluster: raise MsticpyParameterError( "Must specify either a connection string or a cluster name", @@ -322,15 +328,18 @@ def connect(self, connection_str: Optional[str] = None, **kwargs): if cluster: self._current_config = self._lookup_cluster_settings(cluster) + if not self._az_tenant_id: + self._az_tenant_id = self._current_config.tenant_id logger.info( - "Using cluster id: %s, retrieved %s", + "Using cluster id: %s, retrieved url %s to build connection string", cluster, self.cluster_uri, ) kusto_cs = self._get_connection_string_for_cluster(self._current_config) + self.current_connection = cluster else: logger.info("Using connection string %s", connection_str) - self._current_connection = connection_str + self.current_connection = connection_str kusto_cs = connection_str self.client = KustoClient(kusto_cs) @@ -551,14 +560,23 @@ def _get_connection_string_for_cluster( ) -> KustoConnectionStringBuilder: """Return full cluster URI and credential for cluster name or URI.""" auth_params = self._get_auth_params_from_config(cluster_config) + connect_auth_types = self._az_auth_types or AzureCloudConfig().auth_methods if auth_params.method == "clientsecret": - connect_auth_types = self._az_auth_types or AzureCloudConfig().auth_methods + logger.info("Client secret specified in config - using client secret authn") if "clientsecret" not in connect_auth_types: - connect_auth_types.append("clientsecret") + connect_auth_types.insert(0, "clientsecret") credential = az_connect( auth_types=connect_auth_types, **(auth_params.params) ) + elif auth_params.method == "certificate": + logger.info("Certificate specified in config - using certificate authn") + connect_auth_types.insert(0, "certificate") + credential = az_connect( + auth_types=self._az_auth_types, **(auth_params.params) + ) + return self._create_kusto_cert_connection_str(auth_params) else: + logger.info("Using integrated authn") credential = az_connect( auth_types=self._az_auth_types, **(auth_params.params) ) @@ -570,6 +588,41 @@ def _get_connection_string_for_cluster( user_token=token.token, ) + def _create_kql_cert_connection_str( + self, auth_params: AuthParams + ) -> KustoConnectionStringBuilder: + logger.info("Creating KQL connection string for certificate authentication") + if not self._az_tenant_id: + raise ValueError( + "Azure tenant ID must be set in config or connect parameter", + "to use certificate authentication", + ) + cert_bytes = base64.b64decode(auth_params.params["certificate"]) + ( + private_key, + certificate, + _, + ) = pkcs12.load_key_and_certificates(data=cert_bytes, password=None) + if private_key is None or certificate is None: + raise ValueError( + f"Could not load certificate for cluster {self.cluster_uri}" + ) + private_cert = private_key.private_bytes( + encoding=serialization.Encoding.PEM, + format=serialization.PrivateFormat.TraditionalOpenSSL, + encryption_algorithm=serialization.NoEncryption(), + ) + public_cert = certificate.public_bytes(encoding=serialization.Encoding.PEM) + thumbprint = certificate.fingerprint(hashes.SHA1()) + return KustoConnectionStringBuilder.with_aad_application_certificate_sni_authentication( + connection_string=self.cluster_uri, + aad_app_id=auth_params.params["client_id"], + private_certificate=private_cert.decode("utf-8"), + public_certificate=public_cert.decode("utf-8"), + thumbprint=thumbprint.hex().upper(), + authority_id=self._az_tenant_id, + ) + def _get_auth_params_from_config(self, cluster_config: KustoConfig) -> AuthParams: """Get authentication parameters for cluster from KustoConfig values.""" method = "integrated" @@ -581,6 +634,16 @@ def _get_auth_params_from_config(self, cluster_config: KustoConfig) -> AuthParam logger.info( "Using client secret authentication because client_secret in config" ) + elif ( + KFields.CERTIFICATE in cluster_config + and KFields.CLIENT_ID in cluster_config + ): + method = "certificate" + auth_params_dict["client_id"] = cluster_config.ClientId + auth_params_dict["certificate"] = cluster_config.Certificate + logger.info( + "Using client secret authentication because client_secret in config" + ) elif KFields.INTEG_AUTH in cluster_config: logger.info("Using integrated auth.") auth_params_dict["tenant_id"] = cluster_config.tenant_id diff --git a/msticpy/data/drivers/azure_monitor_driver.py b/msticpy/data/drivers/azure_monitor_driver.py index b54281b7b..c7e2f775b 100644 --- a/msticpy/data/drivers/azure_monitor_driver.py +++ b/msticpy/data/drivers/azure_monitor_driver.py @@ -133,6 +133,7 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): self._query_client: Optional[LogsQueryClient] = None self._az_tenant_id: Optional[str] = None self._ws_config: Optional[WorkspaceConfig] = None + self._ws_name: Optional[str] = None self._workspace_id: Optional[str] = None self._workspace_ids: List[str] = [] self._def_connection_str: Optional[str] = connection_str @@ -143,6 +144,10 @@ def __init__(self, connection_str: Optional[str] = None, **kwargs): self.set_driver_property( DriverProps.EFFECTIVE_ENV, DataEnvironment.MSSentinel.name ) + self.set_driver_property(DriverProps.SUPPORTS_THREADING, value=True) + self.set_driver_property( + DriverProps.MAX_PARALLEL, value=kwargs.get("max_threads", 4) + ) logger.info( "AzureMonitorDriver loaded. connect_str %s, kwargs: %s", connection_str, @@ -160,6 +165,29 @@ def url_endpoint(self) -> str: return f"{base_url}v1" return base_url + @property + def current_connection(self) -> str: + """Return the current connection name.""" + connection = self._ws_name + if ( + not connection + and self._ws_config + and WorkspaceConfig.CONF_WS_NAME_KEY in self._ws_config + ): + connection = self._ws_config[WorkspaceConfig.CONF_WS_NAME_KEY] + return ( + connection + or self._def_connection_str + or self._workspace_id + or next(iter(self._workspace_ids), "") + or "AzureMonitor" + ) + + @current_connection.setter + def current_connection(self, value: str): + """Allow attrib to be set but ignore.""" + del value + def connect(self, connection_str: Optional[str] = None, **kwargs): """ Connect to data source. @@ -303,13 +331,21 @@ def query_with_results( title="Workspace not connected.", help_uri=_HELP_URL, ) - logger.info("Query to run %s", query) time_span_value = self._get_time_span_value(**kwargs) - server_timeout = kwargs.pop("timeout", self._def_timeout) workspace_id = next(iter(self._workspace_ids), None) or self._workspace_id additional_workspaces = self._workspace_ids[1:] if self._workspace_ids else None + logger.info("Query to run %s", query) + logger.info( + "Workspaces %s", ",".join(self._workspace_ids) or self._workspace_id + ) + logger.info( + "Time span %s - %s", + str(time_span_value[0]) if time_span_value else "none", + str(time_span_value[1]) if time_span_value else "none", + ) + logger.info("Timeout %s", server_timeout) try: result = self._query_client.query_workspace( workspace_id=workspace_id, # type: ignore[arg-type] @@ -413,6 +449,7 @@ def _get_workspaces(self, connection_str: Optional[str] = None, **kwargs): help_uri=_HELP_URL, ) self._ws_config = ws_config + self._ws_name = workspace_name or ws_config.workspace_id if not self._az_tenant_id and WorkspaceConfig.CONF_TENANT_ID_KEY in ws_config: self._az_tenant_id = ws_config[WorkspaceConfig.CONF_TENANT_ID_KEY] self._workspace_id = ws_config[WorkspaceConfig.CONF_WS_ID_KEY] @@ -464,7 +501,19 @@ def _get_time_span_value(self, **kwargs): start=time_params["start"], end=time_params["end"], ) - time_span_value = time_span.start, time_span.end + # Azure Monitor API expects datetime objects, so + # convert to datetimes if we have pd.Timestamps + t_start = ( + time_span.start.to_pydatetime(warn=False) + if isinstance(time_span.start, pd.Timestamp) + else time_span.start + ) + t_end = ( + time_span.end.to_pydatetime(warn=False) + if isinstance(time_span.end, pd.Timestamp) + else time_span.end + ) + time_span_value = t_start, t_end logger.info("Time parameters set %s", str(time_span)) return time_span_value diff --git a/msticpy/data/drivers/driver_base.py b/msticpy/data/drivers/driver_base.py index b4c5a6def..bfb3d79de 100644 --- a/msticpy/data/drivers/driver_base.py +++ b/msticpy/data/drivers/driver_base.py @@ -88,6 +88,7 @@ def __init__(self, **kwargs): self.data_environment = kwargs.get("data_environment") self._query_filter: Dict[str, Set[str]] = defaultdict(set) self._instance: Optional[str] = None + self.properties = DriverProps.defaults() self.set_driver_property( name=DriverProps.EFFECTIVE_ENV, @@ -97,6 +98,8 @@ def __init__(self, **kwargs): else self.data_environment or "" ), ) + self.set_driver_property(DriverProps.SUPPORTS_THREADING, False) + self.set_driver_property(DriverProps.MAX_PARALLEL, kwargs.get("max_threads", 4)) def __getattr__(self, attrib): """Return item from the properties dictionary as an attribute.""" diff --git a/msticpy/data/drivers/mdatp_driver.py b/msticpy/data/drivers/mdatp_driver.py index 19cbe0d39..0c1bf9214 100644 --- a/msticpy/data/drivers/mdatp_driver.py +++ b/msticpy/data/drivers/mdatp_driver.py @@ -4,7 +4,7 @@ # license information. # -------------------------------------------------------------------------- """MDATP OData Driver class.""" -from typing import Any, Union +from typing import Any, Optional, Union import pandas as pd @@ -27,7 +27,9 @@ class MDATPDriver(OData): CONFIG_NAME = "MicrosoftDefender" _ALT_CONFIG_NAMES = ["MDATPApp"] - def __init__(self, connection_str: str = None, instance: str = "Default", **kwargs): + def __init__( + self, connection_str: Optional[str] = None, instance: str = "Default", **kwargs + ): """ Instantiate MSDefenderDriver and optionally connect. @@ -74,7 +76,7 @@ def __init__(self, connection_str: str = None, instance: str = "Default", **kwar self.connect(connection_str) def query( - self, query: str, query_source: QuerySource = None, **kwargs + self, query: str, query_source: Optional[QuerySource] = None, **kwargs ) -> Union[pd.DataFrame, Any]: """ Execute query string and return DataFrame of results. @@ -89,7 +91,7 @@ def query( Returns ------- Union[pd.DataFrame, results.ResultSet] - A DataFrame (if successfull) or + A DataFrame (if successful) or the underlying provider result if an error. """ diff --git a/msticpy/data/drivers/odata_driver.py b/msticpy/data/drivers/odata_driver.py index ebc3204d4..fd2c1c394 100644 --- a/msticpy/data/drivers/odata_driver.py +++ b/msticpy/data/drivers/odata_driver.py @@ -18,7 +18,7 @@ from ...common.pkg_config import get_config from ...common.provider_settings import get_provider_settings from ...common.utility import mp_ua_header -from .driver_base import DriverBase, QuerySource +from .driver_base import DriverBase, DriverProps, QuerySource __version__ = VERSION __author__ = "Pete Bryan" @@ -66,6 +66,11 @@ def __init__(self, **kwargs): self.scopes = None self.msal_auth = None + self.set_driver_property(DriverProps.SUPPORTS_THREADING, value=True) + self.set_driver_property( + DriverProps.MAX_PARALLEL, value=kwargs.get("max_threads", 4) + ) + @abc.abstractmethod def query( self, query: str, query_source: QuerySource = None, **kwargs diff --git a/msticpy/data/drivers/security_graph_driver.py b/msticpy/data/drivers/security_graph_driver.py index 48b3aa671..a4f5c0433 100644 --- a/msticpy/data/drivers/security_graph_driver.py +++ b/msticpy/data/drivers/security_graph_driver.py @@ -4,7 +4,7 @@ # license information. # -------------------------------------------------------------------------- """Security Graph OData Driver class.""" -from typing import Any, Union +from typing import Any, Optional, Union import pandas as pd @@ -24,7 +24,7 @@ class SecurityGraphDriver(OData): CONFIG_NAME = "MicrosoftGraph" _ALT_CONFIG_NAMES = ["SecurityGraphApp"] - def __init__(self, connection_str: str = None, **kwargs): + def __init__(self, connection_str: Optional[str] = None, **kwargs): """ Instantiate MSGraph driver and optionally connect. @@ -54,7 +54,7 @@ def __init__(self, connection_str: str = None, **kwargs): self.connect(connection_str) def query( - self, query: str, query_source: QuerySource = None, **kwargs + self, query: str, query_source: Optional[QuerySource] = None, **kwargs ) -> Union[pd.DataFrame, Any]: """ Execute query string and return DataFrame of results. @@ -69,7 +69,7 @@ def query( Returns ------- Union[pd.DataFrame, results.ResultSet] - A DataFrame (if successfull) or + A DataFrame (if successful) or the underlying provider result if an error. """ diff --git a/msticpy/init/nbinit.py b/msticpy/init/nbinit.py index 737d57d27..170d606ed 100644 --- a/msticpy/init/nbinit.py +++ b/msticpy/init/nbinit.py @@ -421,7 +421,7 @@ def init_notebook( check_version() output = stdout_cap.getvalue() _pr_output(output) - logger.info(output) + logger.info("Check version failures: %s", output) if _detect_env("synapse", **kwargs) and is_in_synapse(): synapse_params = { @@ -438,7 +438,7 @@ def init_notebook( ) output = stdout_cap.getvalue() _pr_output(output) - logger.info(output) + logger.info("Import failures: %s", output) # Configuration check if no_config_check: @@ -468,7 +468,7 @@ def init_notebook( _load_pivots(namespace=namespace) output = stdout_cap.getvalue() _pr_output(output) - logger.info(output) + logger.info("Pivot load failures: %s", output) # User defaults stdout_cap = io.StringIO() @@ -478,6 +478,7 @@ def init_notebook( output = stdout_cap.getvalue() _pr_output(output) logger.info(output) + logger.info("User default load failures: %s", output) if prov_dict: namespace.update(prov_dict) diff --git a/tests/data/drivers/test_azure_kusto_driver.py b/tests/data/drivers/test_azure_kusto_driver.py index db8e8f0e1..be1663913 100644 --- a/tests/data/drivers/test_azure_kusto_driver.py +++ b/tests/data/drivers/test_azure_kusto_driver.py @@ -93,9 +93,14 @@ def get_test_df(): def test_init(): - # Test that __init__ sets the current_connection property correctly - driver = AzureKustoDriver(connection_str="https://test.kusto.windows.net") - assert driver.current_connection == "https://test.kusto.windows.net" + """Test initialization of AzureKustoDriver.""" + driver = AzureKustoDriver( + connection_str="cluster='https://test.kusto.windows.net', db='Security'" + ) + assert ( + driver.current_connection + == "cluster='https://test.kusto.windows.net', db='Security'" + ) # Test that __init__ sets the _connection_props property correctly driver = AzureKustoDriver(timeout=300) diff --git a/tests/data/test_async_queries.py b/tests/data/test_async_queries.py new file mode 100644 index 000000000..50c052de9 --- /dev/null +++ b/tests/data/test_async_queries.py @@ -0,0 +1,164 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +"""Test async connections and split queries.""" + +from datetime import datetime, timedelta, timezone + +import pandas as pd +import pytest_check as check + +from msticpy.data.core.data_providers import QueryProvider +from msticpy.data.core.query_provider_connections_mixin import _calc_split_ranges +from msticpy.data.drivers.driver_base import DriverProps + +from ..unit_test_lib import get_test_data_path + +_LOCAL_DATA_PATHS = [str(get_test_data_path().joinpath("localdata"))] + +# pylint: disable=protected-access + + +def test_multiple_connections_sync(): + """Test adding connection instance to provider.""" + prov_args = dict(query_paths=_LOCAL_DATA_PATHS, data_paths=_LOCAL_DATA_PATHS) + # create local provider and run a query + local_prov = QueryProvider("LocalData", **prov_args) + start = datetime.now(timezone.utc) - timedelta(days=1) + end = datetime.now(timezone.utc) + single_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + + # add another connection (to same folder) + local_prov.add_connection(alias="SecondInst", **prov_args) + connections = local_prov.list_connections() + # verify second connection is listed + check.equal(len(connections), 2) + check.is_in("Default:", connections[0]) + check.is_in("SecondInst:", connections[1]) + + # run query again + multi_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + # verify len of result is 2x single_result + check.equal(single_results.shape[0] * 2, multi_results.shape[0]) + # verify columns/schema is the same. + check.equal(list(single_results.columns), list(multi_results.columns)) + + +def test_multiple_connections_threaded(): + """Test adding connection instance to provider.""" + prov_args = dict(query_paths=_LOCAL_DATA_PATHS, data_paths=_LOCAL_DATA_PATHS) + # create local provider and run a query + local_prov = QueryProvider("LocalData", **prov_args) + local_prov._query_provider.set_driver_property( + DriverProps.SUPPORTS_THREADING, value=True + ) + start = datetime.now(timezone.utc) - timedelta(days=1) + end = datetime.now(timezone.utc) + single_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + + # add another 2 named connections + for idx in range(1, 3): + local_prov.add_connection(alias=f"Instance {idx}", **prov_args) + # add another 2 unnamed connections + for _ in range(2): + local_prov.add_connection(**prov_args) + + connections = local_prov.list_connections() + # verify second connection is listed + check.equal(len(connections), 5) + check.is_in("Default:", connections[0]) + check.is_in("Instance 1", connections[1]) + + # run query again + multi_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + # verify len of result is 2x single_result + check.equal(single_results.shape[0] * 5, multi_results.shape[0]) + # verify columns/schema is the same. + check.equal(list(single_results.columns), list(multi_results.columns)) + + +def test_split_queries_sync(): + """Test queries split into time segments.""" + prov_args = dict(query_paths=_LOCAL_DATA_PATHS, data_paths=_LOCAL_DATA_PATHS) + local_prov = QueryProvider("LocalData", **prov_args) + + start = datetime.now(timezone.utc) - pd.Timedelta("5H") + end = datetime.now(timezone.utc) + pd.Timedelta("5min") + delta = pd.Timedelta("1H") + + ranges = _calc_split_ranges(start, end, delta) + local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + result_queries = local_prov.WindowsSecurity.list_host_logons( + "print", host_name="DESKTOP-12345", start=start, end=end, split_query_by="1H" + ) + queries = result_queries.split("\n\n") + check.equal(len(queries), 5) + + for idx, (st_time, e_time) in enumerate(ranges): + check.is_in(st_time.isoformat(sep=" "), queries[idx]) + check.is_in(e_time.isoformat(sep=" "), queries[idx]) + check.is_in(start.isoformat(sep=" "), queries[0]) + check.is_in(end.isoformat(sep=" "), queries[-1]) + + single_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + result_queries = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end, split_query_by="1H" + ) + # verify len of result is 2x single_result + check.equal(single_results.shape[0] * 5, result_queries.shape[0]) + # verify columns/schema is the same. + check.equal(list(single_results.columns), list(result_queries.columns)) + + +def test_split_queries_async(): + """Test queries split into time segments threaded execution.""" + prov_args = dict(query_paths=_LOCAL_DATA_PATHS, data_paths=_LOCAL_DATA_PATHS) + local_prov = QueryProvider("LocalData", **prov_args) + local_prov._query_provider.set_driver_property( + DriverProps.SUPPORTS_THREADING, value=True + ) + + start = datetime.now(timezone.utc) - pd.Timedelta("5H") + end = datetime.now(timezone.utc) + pd.Timedelta("5min") + delta = pd.Timedelta("1H") + + ranges = _calc_split_ranges(start, end, delta) + local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + result_queries = local_prov.WindowsSecurity.list_host_logons( + "print", host_name="DESKTOP-12345", start=start, end=end, split_query_by="1H" + ) + queries = result_queries.split("\n\n") + check.equal(len(queries), 5) + + for idx, (st_time, e_time) in enumerate(ranges): + check.is_in(st_time.isoformat(sep=" "), queries[idx]) + check.is_in(e_time.isoformat(sep=" "), queries[idx]) + check.is_in(start.isoformat(sep=" "), queries[0]) + check.is_in(end.isoformat(sep=" "), queries[-1]) + + single_results = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end + ) + result_queries = local_prov.WindowsSecurity.list_host_logons( + host_name="DESKTOP-12345", start=start, end=end, split_query_by="1H" + ) + # verify len of result is 2x single_result + check.equal(single_results.shape[0] * 5, result_queries.shape[0]) + # verify columns/schema is the same. + check.equal(list(single_results.columns), list(result_queries.columns)) diff --git a/tests/data/test_dataqueries.py b/tests/data/test_dataqueries.py index df88c93c7..756c23180 100644 --- a/tests/data/test_dataqueries.py +++ b/tests/data/test_dataqueries.py @@ -20,8 +20,9 @@ from msticpy.common import pkg_config from msticpy.common.exceptions import MsticpyException -from msticpy.data.core.data_providers import QueryProvider, _calc_split_ranges +from msticpy.data.core.data_providers import QueryProvider from msticpy.data.core.query_container import QueryContainer +from msticpy.data.core.query_provider_connections_mixin import _calc_split_ranges from msticpy.data.core.query_source import QuerySource from msticpy.data.drivers.driver_base import DriverBase, DriverProps @@ -404,7 +405,7 @@ def test_split_queries_err(self): queries = result_queries.split("\n\n") # if no start and end - provider prints message and returns None self.assertEqual(len(queries), 1) - self.assertIn("Cannot split a query that", mssg.getvalue()) + self.assertIn("Cannot split a query", mssg.getvalue()) # With invalid split_query_by value it will default to 1D start = datetime.utcnow() - pd.Timedelta("5D") @@ -420,7 +421,7 @@ def test_split_queries_err(self): _LOCAL_DATA_PATHS = [str(get_test_data_path().joinpath("localdata"))] -def test_add_provider(): +def test_multiple_connections(): """Test adding connection instance to provider.""" prov_args = dict(query_paths=_LOCAL_DATA_PATHS, data_paths=_LOCAL_DATA_PATHS) # create local provider and run a query From 7dbc9a49f10b4b4f9b4dd08baece84214549f09a Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 5 Jul 2023 16:21:47 -0700 Subject: [PATCH 20/26] Bump readthedocs-sphinx-ext from 2.2.0 to 2.2.2 (#679) Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.2.0 to 2.2.2. - [Commits](https://github.com/readthedocs/readthedocs-sphinx-ext/compare/2.2.0...2.2.2) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen --- docs/requirements.txt | 2 +- requirements-dev.txt | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index 9a4e881c1..481bc7fb2 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -11,7 +11,7 @@ python-dateutil>=2.8.1 pytz>=2019.2 pyyaml>=3.13 typing-extensions>=4.2.0 -readthedocs-sphinx-ext==2.2.0 +readthedocs-sphinx-ext==2.2.2 seed_intersphinx_mapping sphinx-rtd-theme==1.2.0 sphinx==6.1.3 diff --git a/requirements-dev.txt b/requirements-dev.txt index 79bc60cd9..d073ef955 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -28,7 +28,7 @@ pytest-check>=1.0.1 pytest-cov>=2.11.1 pytest-xdist>=2.5.0 pytest>=5.0.1 -readthedocs-sphinx-ext==2.2.0 +readthedocs-sphinx-ext==2.2.2 responses>=0.13.2 respx>=0.20.1 sphinx-rtd-theme>=1.0.0 From a00ab8ff016fbf780ea003891807510f06d58eda Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 5 Jul 2023 16:22:10 -0700 Subject: [PATCH 21/26] Bump sphinx-rtd-theme from 1.2.0 to 1.2.2 (#675) Bumps [sphinx-rtd-theme](https://github.com/readthedocs/sphinx_rtd_theme) from 1.2.0 to 1.2.2. - [Changelog](https://github.com/readthedocs/sphinx_rtd_theme/blob/master/docs/changelog.rst) - [Commits](https://github.com/readthedocs/sphinx_rtd_theme/compare/1.2.0...1.2.2) --- updated-dependencies: - dependency-name: sphinx-rtd-theme dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen --- docs/requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements.txt b/docs/requirements.txt index 481bc7fb2..852f8975d 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -13,6 +13,6 @@ pyyaml>=3.13 typing-extensions>=4.2.0 readthedocs-sphinx-ext==2.2.2 seed_intersphinx_mapping -sphinx-rtd-theme==1.2.0 +sphinx-rtd-theme==1.2.2 sphinx==6.1.3 sphinxcontrib-jquery==4.1 From 56560b47558e5e2ca91c86846bc397c932a0127d Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Thu, 6 Jul 2023 08:35:57 -0700 Subject: [PATCH 22/26] Bump httpx from 0.24.0 to 0.24.1 (#666) Bumps [httpx](https://github.com/encode/httpx) from 0.24.0 to 0.24.1. - [Release notes](https://github.com/encode/httpx/releases) - [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md) - [Commits](https://github.com/encode/httpx/compare/0.24.0...0.24.1) --- updated-dependencies: - dependency-name: httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen --- conda/conda-reqs.txt | 2 +- docs/requirements.txt | 2 +- requirements-all.txt | 2 +- requirements.txt | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/conda/conda-reqs.txt b/conda/conda-reqs.txt index 3cd11cc53..622bb0262 100644 --- a/conda/conda-reqs.txt +++ b/conda/conda-reqs.txt @@ -19,7 +19,7 @@ dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 html5lib -httpx==0.24.0 +httpx==0.24.1 ipython>=7.23.1 ipywidgets>=7.4.2, <9.0.0 keyring>=13.2.1 diff --git a/docs/requirements.txt b/docs/requirements.txt index 852f8975d..78781c3f9 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -2,7 +2,7 @@ attrs>=18.2.0 cryptography deprecated>=1.2.4 docutils<0.20.0 -httpx==0.24.0 +httpx==0.24.1 ipython >= 7.1.1 jinja2<3.2.0 numpy>=1.15.4 diff --git a/requirements-all.txt b/requirements-all.txt index 079a84505..00430245d 100644 --- a/requirements-all.txt +++ b/requirements-all.txt @@ -21,7 +21,7 @@ deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 -httpx==0.24.0 +httpx==0.24.1 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" diff --git a/requirements.txt b/requirements.txt index 107732a5f..e71d51074 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,7 +10,7 @@ deprecated>=1.2.4 dnspython>=2.0.0, <3.0.0 folium>=0.9.0 geoip2>=2.9.0 -httpx==0.24.0 +httpx==0.24.1 html5lib ipython >= 7.1.1; python_version < "3.8" ipython >= 7.23.1; python_version >= "3.8" From 555b58aa83a6e55ce69cd5da43b2e97494040fac Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Thu, 6 Jul 2023 15:57:53 -0700 Subject: [PATCH 23/26] Ianhelle/fix func query names 2023 06 30 (#680) * Changes to doc strings and queries to improve descriptions of pivot functions and queries. Renamed m365 query files Some corrections to kql_mdatp_file.yaml and kql_m365_file.yaml to reference DeviceFileEvents instead of DeviceProcessEvents * Missing column from query in kql_mdatp_file.yaml * Bunch of minor typo/URL link fixes in documents * Some missing doc files from module generation --- docs/source/Development.rst | 4 +- docs/source/api/msticpy.common.utility.rst | 1 - .../msticpy.common.utility.yaml_loader.rst | 7 - docs/source/api/msticpy.data.drivers.rst | 1 + ...cpy.data.drivers.sentinel_query_reader.rst | 7 + .../data_acquisition/AzureBlobStorage.rst | 2 +- .../source/data_acquisition/DataProviders.rst | 2 +- docs/source/data_acquisition/DataQueries.rst | 490 ++++++++---------- .../data_acquisition/SentinelIncidents.rst | 6 +- docs/source/data_analysis/PivotFunctions.rst | 4 +- .../source/extending/WritingDataProviders.rst | 12 +- docs/source/getting_started/Installing.rst | 29 +- docs/source/visualization/FoliumMap.rst | 2 +- msticpy/context/tilookup.py | 32 +- msticpy/context/vtlookupv3/vtlookupv3.py | 22 +- msticpy/data/core/data_providers.py | 2 +- .../data/core/query_provider_utils_mixin.py | 6 +- ...mdatp_alerts.yaml => kql_m365_alerts.yaml} | 0 ...kql_mdatp_file.yaml => kql_m365_file.yaml} | 34 +- ...atp_hunting.yaml => kql_m365_hunting.yaml} | 2 +- ...atp_network.yaml => kql_m365_network.yaml} | 12 +- ...atp_process.yaml => kql_m365_process.yaml} | 12 +- ...kql_mdatp_user.yaml => kql_m365_user.yaml} | 14 +- .../data/queries/mde/kql_mdatp_alerts.yaml | 12 +- msticpy/data/queries/mde/kql_mdatp_file.yaml | 79 ++- .../data/queries/mde/kql_mdatp_hunting.yaml | 2 +- .../data/queries/mde/kql_mdatp_network.yaml | 12 +- .../data/queries/mde/kql_mdatp_process.yaml | 12 +- msticpy/data/queries/mde/kql_mdatp_user.yaml | 10 +- .../queries/mssentinel/kql_sent_alert.yaml | 8 +- .../queries/mssentinel/kql_sent_az_dns.yaml | 6 +- .../mssentinel/kql_sent_az_network.yaml | 2 +- .../queries/mssentinel/kql_sent_azure.yaml | 18 +- .../mssentinel/kql_sent_azuresentinel.yaml | 14 +- .../kql_sent_lxsyslog_activity.yaml | 10 +- .../mssentinel/kql_sent_lxsyslog_apps.yaml | 2 +- .../mssentinel/kql_sent_lxsyslog_logon.yaml | 12 +- .../mssentinel/kql_sent_lxsyslog_sysmon.yaml | 2 +- .../data/queries/mssentinel/kql_sent_net.yaml | 26 +- .../queries/mssentinel/kql_sent_o365.yaml | 8 +- .../mssentinel/kql_sent_threatintel.yaml | 14 +- .../mssentinel/kql_sent_timeseries.yaml | 12 +- .../queries/mssentinel/kql_sent_winevent.yaml | 16 +- .../mssentinel/kql_sent_winevent_logon.yaml | 16 +- .../mssentinel/kql_sent_winevent_proc.yaml | 18 +- msticpy/init/pivot_init/vt_pivot.py | 30 +- 46 files changed, 585 insertions(+), 459 deletions(-) delete mode 100644 docs/source/api/msticpy.common.utility.yaml_loader.rst create mode 100644 docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst rename msticpy/data/queries/m365d/{kql_mdatp_alerts.yaml => kql_m365_alerts.yaml} (100%) rename msticpy/data/queries/m365d/{kql_mdatp_file.yaml => kql_m365_file.yaml} (69%) rename msticpy/data/queries/m365d/{kql_mdatp_hunting.yaml => kql_m365_hunting.yaml} (99%) rename msticpy/data/queries/m365d/{kql_mdatp_network.yaml => kql_m365_network.yaml} (86%) rename msticpy/data/queries/m365d/{kql_mdatp_process.yaml => kql_m365_process.yaml} (87%) rename msticpy/data/queries/m365d/{kql_mdatp_user.yaml => kql_m365_user.yaml} (86%) diff --git a/docs/source/Development.rst b/docs/source/Development.rst index 385096c7d..6a86185cd 100644 --- a/docs/source/Development.rst +++ b/docs/source/Development.rst @@ -16,8 +16,8 @@ an improvement to an existing feature that you have thought about such as a new data connector or threat intelligence provider, or a completely new feature category. -If you don’t have a specific idea in mind take a look at the -Issues page on GitHub: `https://github.com/microsoft/msticpy/issues`__ +If you don't have a specific idea in mind take a look at the +`Issues page on GitHub `__ This page tracks a range of issues, enhancements, and features that members of the community have thought of. The MSTICPy team uses these diff --git a/docs/source/api/msticpy.common.utility.rst b/docs/source/api/msticpy.common.utility.rst index 6f622c02d..cd235f316 100644 --- a/docs/source/api/msticpy.common.utility.rst +++ b/docs/source/api/msticpy.common.utility.rst @@ -16,4 +16,3 @@ Submodules msticpy.common.utility.ipython msticpy.common.utility.package msticpy.common.utility.types - msticpy.common.utility.yaml_loader diff --git a/docs/source/api/msticpy.common.utility.yaml_loader.rst b/docs/source/api/msticpy.common.utility.yaml_loader.rst deleted file mode 100644 index 21b164f63..000000000 --- a/docs/source/api/msticpy.common.utility.yaml_loader.rst +++ /dev/null @@ -1,7 +0,0 @@ -msticpy.common.utility.yaml\_loader module -========================================== - -.. automodule:: msticpy.common.utility.yaml_loader - :members: - :undoc-members: - :show-inheritance: diff --git a/docs/source/api/msticpy.data.drivers.rst b/docs/source/api/msticpy.data.drivers.rst index d60239fd3..b44059dab 100644 --- a/docs/source/api/msticpy.data.drivers.rst +++ b/docs/source/api/msticpy.data.drivers.rst @@ -27,5 +27,6 @@ Submodules msticpy.data.drivers.odata_driver msticpy.data.drivers.resource_graph_driver msticpy.data.drivers.security_graph_driver + msticpy.data.drivers.sentinel_query_reader msticpy.data.drivers.splunk_driver msticpy.data.drivers.sumologic_driver diff --git a/docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst b/docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst new file mode 100644 index 000000000..69968e584 --- /dev/null +++ b/docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst @@ -0,0 +1,7 @@ +msticpy.data.drivers.sentinel\_query\_reader module +=================================================== + +.. automodule:: msticpy.data.drivers.sentinel_query_reader + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/source/data_acquisition/AzureBlobStorage.rst b/docs/source/data_acquisition/AzureBlobStorage.rst index 4e4fdcdee..1db23c2bc 100644 --- a/docs/source/data_acquisition/AzureBlobStorage.rst +++ b/docs/source/data_acquisition/AzureBlobStorage.rst @@ -129,7 +129,7 @@ See :py:mod:`delete_blob`_ for the specified blob. +``get_sas_token`` generates a `SAS token `__ for the specified blob. By default the token generated is valid for read access for 7 days but permissions can be modified with the ``permission`` keyword, and validity time-frame with the ``start`` and ``end`` keywords. The returned string is a full URI for the blob, with the SAS token appended. diff --git a/docs/source/data_acquisition/DataProviders.rst b/docs/source/data_acquisition/DataProviders.rst index 65f6561a1..38601e807 100644 --- a/docs/source/data_acquisition/DataProviders.rst +++ b/docs/source/data_acquisition/DataProviders.rst @@ -96,7 +96,7 @@ details) with: ) -For more details see :py:class:`QueryProvider API`. +For more details see :py:class:`QueryProvider API `. Connecting to a Data Environment diff --git a/docs/source/data_acquisition/DataQueries.rst b/docs/source/data_acquisition/DataQueries.rst index 012264bfc..58755b5d4 100644 --- a/docs/source/data_acquisition/DataQueries.rst +++ b/docs/source/data_acquisition/DataQueries.rst @@ -10,196 +10,181 @@ Data Environment identifier: MSSentinel ================== ================================ ================================================================================================================================== =============================================================================================================== =========================== QueryGroup Query Description Req-Params Table ================== ================================ ================================================================================================================================== =============================================================================================================== =========================== -Azure get_vmcomputer_for_host Gets latest VMComputer record for Host end (datetime), host_name (str), start (datetime) VMComputer -Azure get_vmcomputer_for_ip Gets latest VMComputer record for IPAddress end (datetime), ip_address (str), start (datetime) VMComputer -Azure list_aad_signins_for_account Lists Azure AD Signins for Account end (datetime), start (datetime) SigninLogs -Azure list_aad_signins_for_ip Lists Azure AD Signins for an IP Address end (datetime), ip_address_list (list), start (datetime) SigninLogs +Azure get_vmcomputer_for_host Returns most recent VMComputer record for Host end (datetime), host_name (str), start (datetime) VMComputer +Azure get_vmcomputer_for_ip Returns most recent VMComputer record for IPAddress end (datetime), ip_address (str), start (datetime) VMComputer +Azure list_aad_signins_for_account Returns Azure AD Signins for Account end (datetime), start (datetime) SigninLogs +Azure list_aad_signins_for_ip Returns Azure AD Signins for an IP Address end (datetime), ip_address_list (list), start (datetime) SigninLogs Azure list_all_signins_geo Gets Signin data used by morph charts end (datetime), start (datetime) SigninLogs -Azure list_azure_activity_for_account Lists Azure Activity for Account account_name (str), end (datetime), start (datetime) AzureActivity -Azure list_azure_activity_for_ip Lists Azure Activity for Caller IP Address(es) end (datetime), ip_address_list (list), start (datetime) AzureActivity -Azure list_azure_activity_for_resource Lists Azure Activity for a Resource end (datetime), resource_id (str), start (datetime) AzureActivity -Azure list_storage_ops_for_hash no description end (datetime), file_hash (str), start (datetime) StorageFileLogs -Azure list_storage_ops_for_ip no description end (datetime), ip_address (str), start (datetime) StorageFileLogs +Azure list_azure_activity_for_account Returns Azure Activity for Account account_name (str), end (datetime), start (datetime) AzureActivity +Azure list_azure_activity_for_ip Returns Azure Activity for Caller IP Address(es) end (datetime), ip_address_list (list), start (datetime) AzureActivity +Azure list_azure_activity_for_resource Returns Azure Activity for an Azure Resource ID end (datetime), resource_id (str), start (datetime) AzureActivity +Azure list_storage_ops_for_hash Returns Azure Storage Operations for an MD5 file hash end (datetime), file_hash (str), start (datetime) StorageFileLogs +Azure list_storage_ops_for_ip Returns Storage Operations for an IP Address end (datetime), ip_address (str), start (datetime) StorageFileLogs AzureNetwork all_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog -AzureNetwork az_net_analytics All Azure Network Analytics Data end (datetime), start (datetime) AzureNetworkAnalytics_CL -AzureNetwork dns_lookups_for_domain Dns queries for a domain domain (str), end (datetime), start (datetime) DnsEvents -AzureNetwork dns_lookups_for_ip Dns queries for a domain end (datetime), ip_address (str), start (datetime) DnsEvents -AzureNetwork dns_lookups_from_ip Dns queries for a domain end (datetime), ip_address (str), start (datetime) DnsEvents -AzureNetwork get_heartbeat_for_host Retrieves latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat -AzureNetwork get_heartbeat_for_ip Retrieves latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat -AzureNetwork get_host_for_ip Gets the latest AzureNetworkAnalytics interface event for a host. end (datetime), ip_address (str), start (datetime) AzureNetworkAnalytics_CL -AzureNetwork get_ips_for_host Gets the latest AzureNetworkAnalytics interface event for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL -AzureNetwork host_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog +AzureNetwork az_net_analytics Returns all Azure Network Flow (NSG) Data for a given host end (datetime), start (datetime) AzureNetworkAnalytics_CL +AzureNetwork dns_lookups_for_domain Returns DNS query events for a specified domain domain (str), end (datetime), start (datetime) DnsEvents +AzureNetwork dns_lookups_for_ip Returns Dns query events that contain a resolved IP address end (datetime), ip_address (str), start (datetime) DnsEvents +AzureNetwork dns_lookups_from_ip Returns Dns queries originating from a specified IP address end (datetime), ip_address (str), start (datetime) DnsEvents +AzureNetwork get_heartbeat_for_host Returns latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat +AzureNetwork get_heartbeat_for_ip Returns latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat +AzureNetwork get_host_for_ip Returns the most recent Azure NSG Interface event for an IP Address. end (datetime), ip_address (str), start (datetime) AzureNetworkAnalytics_CL +AzureNetwork get_ips_for_host Returns the most recent Azure Network NSG Interface event for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL +AzureNetwork host_network_connections_csl Returns network connections to and from a host (CommonSecurityLog) end (datetime), start (datetime) CommonSecurityLog AzureNetwork hosts_by_ip_csl no description end (datetime), start (datetime) CommonSecurityLog AzureNetwork ip_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog AzureNetwork ips_by_host_csl no description end (datetime), start (datetime) CommonSecurityLog -AzureNetwork list_azure_network_flows_by_host Retrieves Azure network analytics flow events. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL -AzureNetwork list_azure_network_flows_by_ip Retrieves Azure network analytics flow events. end (datetime), ip_address_list (list), start (datetime) AzureNetworkAnalytics_CL -AzureNetwork network_connections_to_url List of network connections to a URL end (datetime), start (datetime), url (str) CommonSecurityLog -AzureSentinel get_bookmark_by_id Retrieves a single Bookmark by BookmarkId bookmark_id (str), end (datetime), start (datetime) HuntingBookmark +AzureNetwork list_azure_network_flows_by_host Returns Azure NSG flow events for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL +AzureNetwork list_azure_network_flows_by_ip Returns Azure NSG flow events for an IP Address. end (datetime), ip_address_list (list), start (datetime) AzureNetworkAnalytics_CL +AzureNetwork network_connections_to_url Returns connections to a URL or domain (CommonSecurityLog) end (datetime), start (datetime), url (str) CommonSecurityLog +AzureSentinel get_bookmark_by_id Returns a single Bookmark by BookmarkId bookmark_id (str), end (datetime), start (datetime) HuntingBookmark AzureSentinel get_bookmark_by_name Retrieves one or more Bookmarks by Bookmark Name bookmark_name (str), end (datetime), start (datetime) HuntingBookmark -AzureSentinel get_dynamic_summary_by_id Retrieves Dynamic Summary by SummaryId end (datetime), start (datetime), summary_id (str) DynamicSummary -AzureSentinel get_dynamic_summary_by_name Retrieves Dynamic Summary by Name end (datetime), start (datetime), summary_name (str) DynamicSummary -AzureSentinel list_bookmarks Retrieves list of bookmarks end (datetime), start (datetime) HuntingBookmark -AzureSentinel list_bookmarks_for_entity Retrieves bookmarks for entity string end (datetime), start (datetime) HuntingBookmark -AzureSentinel list_bookmarks_for_tags Retrieves Bookmark by one or mare Tags bookmark_tags (list), end (datetime), start (datetime) HuntingBookmark -AzureSentinel list_dynamic_summaries Retrieves Dynamic Summaries by date range end (datetime), start (datetime) DynamicSummary -Heartbeat get_heartbeat_for_host Retrieves latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat -Heartbeat get_heartbeat_for_ip Retrieves latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat +AzureSentinel get_dynamic_summary_by_id Returns a Dynamic Summary by SummaryId end (datetime), start (datetime), summary_id (str) DynamicSummary +AzureSentinel get_dynamic_summary_by_name Returns a Dynamic Summary by Name end (datetime), start (datetime), summary_name (str) DynamicSummary +AzureSentinel list_bookmarks Retrieves list of bookmarks for a time range end (datetime), start (datetime) HuntingBookmark +AzureSentinel list_bookmarks_for_entity Retrieves bookmarks for a host, account, ip address, domain, url or other entity identifier end (datetime), start (datetime) HuntingBookmark +AzureSentinel list_bookmarks_for_tags Returns Bookmark by one or more Tags bookmark_tags (list), end (datetime), start (datetime) HuntingBookmark +AzureSentinel list_dynamic_summaries Returns all Dynamic Summaries by time range end (datetime), start (datetime) DynamicSummary +Heartbeat get_heartbeat_for_host Returns latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat +Heartbeat get_heartbeat_for_ip Returns latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat Heartbeat get_info_by_hostname Deprecated - use 'get_heartbeat_for_host' end (datetime), host_name (str), start (datetime) Heartbeat Heartbeat get_info_by_ipaddress Deprecated - use 'get_heartbeat_for_ip' end (datetime), ip_address (str), start (datetime) Heartbeat LinuxAudit auditd_all Extract all audit messages grouped by mssg_id end (datetime), start (datetime) AuditLog_CL LinuxSyslog all_syslog Returns all syslog activity for a host end (datetime), start (datetime) Syslog -LinuxSyslog cron_activity All cron activity end (datetime), start (datetime) Syslog -LinuxSyslog list_account_logon_failures All failed user logon events from an IP address account_name (str), end (datetime), start (datetime) Syslog -LinuxSyslog list_host_logon_failures All failed user logon events on a host end (datetime), host_name (str), start (datetime) Syslog -LinuxSyslog list_ip_logon_failures All failed user logon events from an IP address end (datetime), ip_address (str), start (datetime) Syslog +LinuxSyslog cron_activity Returns all cron activity for a host end (datetime), start (datetime) Syslog +LinuxSyslog list_account_logon_failures All failed user logon events for account name account_name (str), end (datetime), start (datetime) Syslog +LinuxSyslog list_host_logon_failures Failed user logon events on a host end (datetime), host_name (str), start (datetime) Syslog +LinuxSyslog list_ip_logon_failures Failed user logon events from an IP address end (datetime), ip_address (str), start (datetime) Syslog LinuxSyslog list_logon_failures All failed user logon events on any host end (datetime), start (datetime) Syslog -LinuxSyslog list_logons_for_account All successful user logon events for account (all hosts) account_name (str), end (datetime), start (datetime) Syslog +LinuxSyslog list_logons_for_account Successful user logon events for account name (all hosts) account_name (str), end (datetime), start (datetime) Syslog LinuxSyslog list_logons_for_host All logon events on a host end (datetime), host_name (str), start (datetime) Syslog -LinuxSyslog list_logons_for_source_ip All successful user logon events for source IP (all hosts) end (datetime), ip_address (str), start (datetime) Syslog -LinuxSyslog notable_events Returns all syslog activity for a host end (datetime), start (datetime) Syslog -LinuxSyslog squid_activity All squid proxy activity end (datetime), host_name (str), start (datetime) Syslog -LinuxSyslog sudo_activity All sudo activity end (datetime), start (datetime) Syslog -LinuxSyslog summarize_events Returns all syslog activity for a host end (datetime), start (datetime) Syslog -LinuxSyslog sysmon_process_events Get Process Events from a specified host end (datetime), host_name (str), start (datetime) - -LinuxSyslog user_group_activity All user/group additions, deletions, and modifications end (datetime), start (datetime) Syslog -LinuxSyslog user_logon All user logon events on a host end (datetime), host_name (str), start (datetime) Syslog -MDATP file_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceProcessEvents -MDATP host_connections Lists connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents -MDATP ip_connections Lists network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents -MDATP list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents -MDATP list_filehash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceProcessEvents -MDATP list_files Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceProcessEvents -MDATP list_host_processes Lists all process creations for a host end (datetime), host_name (str), start (datetime) DeviceProcessEvents -MDATP process_cmd_line Lists all processes with a command line containing a string cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents -MDATP process_creations Lists all processes created by name or hash end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents -MDATP process_paths Lists all processes created from a path end (datetime), file_path (str), start (datetime) DeviceProcessEvents -MDATP protocol_connections Lists connections associated with a specified protocol end (datetime), protocol (str), start (datetime) DeviceNetworkEvents -MDATP url_connections Lists connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents -MDATP user_files Lists all files created by a user account_name (str), end (datetime), start (datetime) - -MDATP user_logons Lists all user logons by user account_name (str), end (datetime), start (datetime) - -MDATP user_network Lists all network connections associated with a user account_name (str), end (datetime), start (datetime) - -MDATP user_processes Lists all processes created by a user account_name (str), end (datetime), start (datetime) - -MDATPHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - -MDATPHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - -MDATPHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - -MDATPHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - -MDATPHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - -MDATPHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - -MDATPHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - -MDATPHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - -MDATPHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - -MDATPHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - -MDATPHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - -MDATPHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - -MDATPHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - -MDATPHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - -MDATPHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - -MDATPHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - -MDATPHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - -MDATPHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - -MDATPHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - -MDATPHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - -MDE accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - -MDE av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - -MDE b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - -MDE brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - -MDE cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - -MDE cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - -MDE cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - -MDE doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - -MDE dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - -MDE email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - -MDE email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - -MDE file_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceProcessEvents -MDE host_connections Lists connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents -MDE ip_connections Lists network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents -MDE list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents -MDE list_filehash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceProcessEvents -MDE list_files Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceProcessEvents -MDE list_host_processes Lists all process creations for a host end (datetime), host_name (str), start (datetime) DeviceProcessEvents -MDE malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - -MDE network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - -MDE powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - -MDE process_cmd_line Lists all processes with a command line containing a string cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents -MDE process_creations Lists all processes created by name or hash end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents -MDE process_paths Lists all processes created from a path end (datetime), file_path (str), start (datetime) DeviceProcessEvents -MDE protocol_connections Lists connections associated with a specified protocol end (datetime), protocol (str), start (datetime) DeviceNetworkEvents -MDE service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - -MDE smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - -MDE smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - -MDE tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - -MDE uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - -MDE url_connections Lists connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents -MDE user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - -MDE user_files Lists all files created by a user account_name (str), end (datetime), start (datetime) - -MDE user_logons Lists all user logons by user account_name (str), end (datetime), start (datetime) - -MDE user_network Lists all network connections associated with a user account_name (str), end (datetime), start (datetime) - -MDE user_processes Lists all processes created by a user account_name (str), end (datetime), start (datetime) - -MSSentinel get_bookmark_by_id Retrieves a single Bookmark by BookmarkId bookmark_id (str), end (datetime), start (datetime) HuntingBookmark +LinuxSyslog list_logons_for_source_ip Successful user logon events for source IP (all hosts) end (datetime), ip_address (str), start (datetime) Syslog +LinuxSyslog notable_events Returns all 'alert' and 'crit' syslog activity for a host end (datetime), start (datetime) Syslog +LinuxSyslog squid_activity Returns all squid proxy activity for a host end (datetime), host_name (str), start (datetime) Syslog +LinuxSyslog sudo_activity Returns all sudo activity for a host and account name end (datetime), start (datetime) Syslog +LinuxSyslog summarize_events Returns summarized syslog activity for a host end (datetime), start (datetime) Syslog +LinuxSyslog sysmon_process_events Sysmon Process Events on host end (datetime), host_name (str), start (datetime) - +LinuxSyslog user_group_activity Returns all user/group additions, deletions, and modifications for a host end (datetime), start (datetime) Syslog +LinuxSyslog user_logon User logon events on a host end (datetime), host_name (str), start (datetime) Syslog +M365D host_connections Returns connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents +M365D ip_connections Returns network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents +M365D list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents +M365D list_file_events_for_filename Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceFileEvents +M365D list_file_events_for_hash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceFileEvents +M365D list_file_events_for_host Lists all file events for a host/device end (datetime), start (datetime) DeviceFileEvents +M365D list_file_events_for_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceFileEvents +M365D list_host_processes Return all process creations for a host for the specified time range end (datetime), host_name (str), start (datetime) DeviceProcessEvents +M365D process_cmd_line Lists all processes with a command line containing a string (all hosts) cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents +M365D process_creations Return all processes with matching name or hash (all hosts) end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents +M365D process_paths Return all processes with a matching path (part path) (all hosts) end (datetime), file_path (str), start (datetime) DeviceProcessEvents +M365D protocol_connections Returns connections associated with a specified protocol (port number) end (datetime), protocol (str), start (datetime) DeviceNetworkEvents +M365D url_connections Returns connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents +M365D user_files Return all files created by a user account_name (str), end (datetime), start (datetime) - +M365D user_logons Return all user logons for user name account_name (str), end (datetime), start (datetime) - +M365D user_network Return all network connections associated with a user account_name (str), end (datetime), start (datetime) - +M365D user_processes Return all processes created by a user account_name (str), end (datetime), start (datetime) - +M365DHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - +M365DHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - +M365DHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - +M365DHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - +M365DHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - +M365DHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - +M365DHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - +M365DHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - +M365DHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - +M365DHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - +M365DHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - +M365DHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - +M365DHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - +M365DHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - +M365DHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - +M365DHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - +M365DHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - +M365DHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - +M365DHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - +M365DHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - +MDEHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - +MDEHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - +MDEHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - +MDEHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - +MDEHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - +MDEHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - +MDEHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - +MDEHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - +MDEHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - +MDEHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - +MDEHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - +MDEHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - +MDEHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - +MDEHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - +MDEHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - +MDEHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - +MDEHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - +MDEHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - +MDEHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - +MDEHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - +MSSentinel get_bookmark_by_id Returns a single Bookmark by BookmarkId bookmark_id (str), end (datetime), start (datetime) HuntingBookmark MSSentinel get_bookmark_by_name Retrieves one or more Bookmarks by Bookmark Name bookmark_name (str), end (datetime), start (datetime) HuntingBookmark -MSSentinel get_dynamic_summary_by_id Retrieves Dynamic Summary by SummaryId end (datetime), start (datetime), summary_id (str) DynamicSummary -MSSentinel get_dynamic_summary_by_name Retrieves Dynamic Summary by Name end (datetime), start (datetime), summary_name (str) DynamicSummary -MSSentinel list_bookmarks Retrieves list of bookmarks end (datetime), start (datetime) HuntingBookmark -MSSentinel list_bookmarks_for_entity Retrieves bookmarks for entity string end (datetime), start (datetime) HuntingBookmark -MSSentinel list_bookmarks_for_tags Retrieves Bookmark by one or mare Tags bookmark_tags (list), end (datetime), start (datetime) HuntingBookmark -MSSentinel list_dynamic_summaries Retrieves Dynamic Summaries by date range end (datetime), start (datetime) DynamicSummary -MultiDataSource get_timeseries_anomalies Time Series filtered anomalies detected using built-in KQL time series function-series_decompose_anomalies end (datetime), start (datetime), table (str) na -MultiDataSource get_timeseries_data Retrieves TimeSeriesData prepared to use with built-in KQL time series functions end (datetime), start (datetime), table (str) na -MultiDataSource get_timeseries_decompose Time Series decomposition and anomalies generated using built-in KQL time series function- series_decompose end (datetime), start (datetime), table (str) na -MultiDataSource plot_timeseries_datawithbaseline Plot timeseries data using built-in KQL time series decomposition using built-in KQL render method end (datetime), start (datetime), table (str) na -MultiDataSource plot_timeseries_scoreanomolies Plot timeseries anomaly score using built-in KQL render method end (datetime), start (datetime), table (str) na +MSSentinel get_dynamic_summary_by_id Returns a Dynamic Summary by SummaryId end (datetime), start (datetime), summary_id (str) DynamicSummary +MSSentinel get_dynamic_summary_by_name Returns a Dynamic Summary by Name end (datetime), start (datetime), summary_name (str) DynamicSummary +MSSentinel list_bookmarks Retrieves list of bookmarks for a time range end (datetime), start (datetime) HuntingBookmark +MSSentinel list_bookmarks_for_entity Retrieves bookmarks for a host, account, ip address, domain, url or other entity identifier end (datetime), start (datetime) HuntingBookmark +MSSentinel list_bookmarks_for_tags Returns Bookmark by one or more Tags bookmark_tags (list), end (datetime), start (datetime) HuntingBookmark +MSSentinel list_dynamic_summaries Returns all Dynamic Summaries by time range end (datetime), start (datetime) DynamicSummary +MultiDataSource get_timeseries_anomalies Time Series filtered anomalies using native KQL analysis (series_decompose_anomalies) end (datetime), start (datetime), table (str) na +MultiDataSource get_timeseries_data Generic query to return TimeSeriesData for use with native KQL time series functions end (datetime), start (datetime), table (str) na +MultiDataSource get_timeseries_decompose Generic Time Series decomposition using native KQL analysis (series_decompose) end (datetime), start (datetime), table (str) na +MultiDataSource plot_timeseries_datawithbaseline Plot of Time Series data using native KQL analysis and plot rendering (KQLMagic only) end (datetime), start (datetime), table (str) na +MultiDataSource plot_timeseries_scoreanomolies Plot Time Series anomaly score using native KQL render (KQLMagic only) end (datetime), start (datetime), table (str) na Network all_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog -Network get_heartbeat_for_host Retrieves latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat -Network get_heartbeat_for_ip Retrieves latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat -Network get_host_for_ip Gets the latest AzureNetworkAnalytics interface event for a host. end (datetime), ip_address (str), start (datetime) AzureNetworkAnalytics_CL -Network get_ips_for_host Gets the latest AzureNetworkAnalytics interface event for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL -Network host_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog +Network get_heartbeat_for_host Returns latest OMS Heartbeat event for host. end (datetime), host_name (str), start (datetime) Heartbeat +Network get_heartbeat_for_ip Returns latest OMS Heartbeat event for ip address. end (datetime), ip_address (str), start (datetime) Heartbeat +Network get_host_for_ip Returns the most recent Azure NSG Interface event for an IP Address. end (datetime), ip_address (str), start (datetime) AzureNetworkAnalytics_CL +Network get_ips_for_host Returns the most recent Azure Network NSG Interface event for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL +Network host_network_connections_csl Returns network connections to and from a host (CommonSecurityLog) end (datetime), start (datetime) CommonSecurityLog Network hosts_by_ip_csl no description end (datetime), start (datetime) CommonSecurityLog Network ip_network_connections_csl no description end (datetime), start (datetime) CommonSecurityLog Network ips_by_host_csl no description end (datetime), start (datetime) CommonSecurityLog -Network list_azure_network_flows_by_host Retrieves Azure network analytics flow events. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL -Network list_azure_network_flows_by_ip Retrieves Azure network analytics flow events. end (datetime), ip_address_list (list), start (datetime) AzureNetworkAnalytics_CL -Network network_connections_to_url List of network connections to a URL end (datetime), start (datetime), url (str) CommonSecurityLog -Office365 list_activity_for_account Lists Office Activity for Account account_name (str), end (datetime), start (datetime) OfficeActivity -Office365 list_activity_for_ip Lists Office Activity for Caller IP Address(es) end (datetime), ip_address_list (list), start (datetime) OfficeActivity -Office365 list_activity_for_resource Lists Office Activity for a Resource end (datetime), resource_id (str), start (datetime) OfficeActivity +Network list_azure_network_flows_by_host Returns Azure NSG flow events for a host. end (datetime), host_name (str), start (datetime) AzureNetworkAnalytics_CL +Network list_azure_network_flows_by_ip Returns Azure NSG flow events for an IP Address. end (datetime), ip_address_list (list), start (datetime) AzureNetworkAnalytics_CL +Network network_connections_to_url Returns connections to a URL or domain (CommonSecurityLog) end (datetime), start (datetime), url (str) CommonSecurityLog +Office365 list_activity_for_account Lists Office/O365 Activity for Account account_name (str), end (datetime), start (datetime) OfficeActivity +Office365 list_activity_for_ip Lists Office/O365 Activity for Caller IP Address(es) end (datetime), ip_address_list (list), start (datetime) OfficeActivity +Office365 list_activity_for_resource Lists Office/O365 Activity for a Resource (OfficeObjectId) end (datetime), resource_id (str), start (datetime) OfficeActivity SecurityAlert get_alert Retrieves a single alert by SystemAlertId system_alert_id (str) SecurityAlert -SecurityAlert list_alerts Retrieves list of alerts end (datetime), start (datetime) SecurityAlert -SecurityAlert list_alerts_counts Retrieves summary count of alerts by type end (datetime), start (datetime) SecurityAlert -SecurityAlert list_alerts_for_ip Retrieves list of alerts with a common IP Address end (datetime), source_ip_list (str), start (datetime) SecurityAlert -SecurityAlert list_related_alerts Retrieves list of alerts with a common host, account or process end (datetime), start (datetime) SecurityAlert -ThreatIntelligence list_indicators Retrieves list of all current indicators. end (datetime), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_domain Retrieves list of indicators by domain domain_list (list), end (datetime), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_email Retrieves list of indicators by email address end (datetime), observables (list), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_filepath Retrieves list of indicators by file path end (datetime), observables (list), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_hash Retrieves list of indicators by file hash end (datetime), file_hash_list (list), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_ip Retrieves list of indicators by IP Address end (datetime), ip_address_list (list), start (datetime) ThreatIntelligenceIndicator -ThreatIntelligence list_indicators_by_url Retrieves list of indicators by URL end (datetime), start (datetime), url_list (list) ThreatIntelligenceIndicator -WindowsSecurity account_change_events Gets events related to account changes end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity get_host_logon Retrieves the logon event for the session id on the host end (datetime), host_name (str), logon_session_id (str), start (datetime) SecurityEvent -WindowsSecurity get_parent_process Retrieves the parent process of a supplied process end (datetime), host_name (str), logon_session_id (str), process_id (str), process_name (str), start (datetime) SecurityEvent -WindowsSecurity get_process_tree Retrieves the process tree of a supplied process end (datetime), host_name (str), logon_session_id (str), process_id (str), process_name (str), start (datetime) SecurityEvent -WindowsSecurity list_all_logons_by_host account all failed or successful logons to a host end (datetime), host_name (str), start (datetime) SecurityEvent +SecurityAlert list_alerts Returns security alerts for a given time range end (datetime), start (datetime) SecurityAlert +SecurityAlert list_alerts_counts Returns summary count of alerts by type end (datetime), start (datetime) SecurityAlert +SecurityAlert list_alerts_for_ip Returns alerts with the specified IP Address or addresses. end (datetime), source_ip_list (str), start (datetime) SecurityAlert +SecurityAlert list_related_alerts Returns alerts with a host, account or process entity end (datetime), start (datetime) SecurityAlert +ThreatIntelligence list_indicators Returns list of all current indicators. end (datetime), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_domain Returns list of indicators by domain domain_list (list), end (datetime), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_email Returns list of indicators by email address end (datetime), observables (list), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_filepath Returns list of indicators by file path end (datetime), observables (list), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_hash Returns list of indicators by file hash end (datetime), file_hash_list (list), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_ip Returns list of indicators by IP Address end (datetime), ip_address_list (list), start (datetime) ThreatIntelligenceIndicator +ThreatIntelligence list_indicators_by_url Returns list of indicators by URL end (datetime), start (datetime), url_list (list) ThreatIntelligenceIndicator +WindowsSecurity account_change_events Returns events related to account changes end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity get_host_logon Returns the logon event for the logon session id on a host end (datetime), host_name (str), logon_session_id (str), start (datetime) SecurityEvent +WindowsSecurity get_parent_process Returns the parent process of process (process id, session id and host name) end (datetime), host_name (str), logon_session_id (str), process_id (str), process_name (str), start (datetime) SecurityEvent +WindowsSecurity get_process_tree Returns the process tree for process id, session id and host name. end (datetime), host_name (str), logon_session_id (str), process_id (str), process_name (str), start (datetime) SecurityEvent +WindowsSecurity list_all_logons_by_host Returns all failed or successful logons on a host end (datetime), host_name (str), start (datetime) SecurityEvent WindowsSecurity list_events Retrieves list of all events end (datetime), start (datetime) SecurityEvent -WindowsSecurity list_events_by_id Retrieves list of events on a host end (datetime), event_list (list), start (datetime) SecurityEvent -WindowsSecurity list_host_events Retrieves list of all events on a host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_host_events_by_id Retrieves list of events on a host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_host_logon_failures Retrieves the logon failure events on the host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_host_logons Retrieves the logon events on the host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_host_processes Retrieves list of processes on a host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_hosts_matching_commandline Retrieves processes on hosts with matching commandline commandline (str), end (datetime), process_name (str), start (datetime) SecurityEvent -WindowsSecurity list_logon_attempts_by_account Retrieves the logon events for an account account_name (str), end (datetime), start (datetime) SecurityEvent -WindowsSecurity list_logon_attempts_by_ip Retrieves the logon events for an IP Address end (datetime), ip_address (str), start (datetime) SecurityEvent -WindowsSecurity list_logon_failures_by_account Retrieves the logon failure events for an account account_name (str), end (datetime), start (datetime) SecurityEvent -WindowsSecurity list_logons_by_account Retrieves the logon events for an account account_name (str), end (datetime), start (datetime) SecurityEvent -WindowsSecurity list_matching_processes Retrieves list of processes matching process name end (datetime), process_name (str), start (datetime) SecurityEvent -WindowsSecurity list_other_events Retrieves list of events other than logon and process on a host end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity list_processes_in_session Retrieves all processes on the host for a logon session end (datetime), host_name (str), logon_session_id (str), process_id (str), process_name (str), start (datetime) SecurityEvent -WindowsSecurity notable_events Get notebable Windows events not returned in other queries end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity schdld_tasks_and_services Gets events related to scheduled tasks and services end (datetime), host_name (str), start (datetime) SecurityEvent -WindowsSecurity summarize_events Summarizes a the events on a host end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_events_by_id Returns list of events on a host by EventID end (datetime), event_list (list), start (datetime) SecurityEvent +WindowsSecurity list_host_events Returns list of all events on a host end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_host_events_by_id Returns list of specified event IDs on a host end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_host_logon_failures Returns the logon failure events on a host for time range end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_host_logons Returns the logon events on a host for time range end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_host_processes Returns list of processes on a host for a time range end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_hosts_matching_commandline Returns processes on hosts with matching command line commandline (str), end (datetime), process_name (str), start (datetime) SecurityEvent +WindowsSecurity list_logon_attempts_by_account Retrieves all logon events for an account (all hosts) account_name (str), end (datetime), start (datetime) SecurityEvent +WindowsSecurity list_logon_attempts_by_ip Returns the logon events for an IP Address (all hosts) end (datetime), ip_address (str), start (datetime) SecurityEvent +WindowsSecurity list_logon_failures_by_account Returns the logon failure events for an account (all hosts) account_name (str), end (datetime), start (datetime) SecurityEvent +WindowsSecurity list_logons_by_account Returns the logon success events for an account (all hosts) account_name (str), end (datetime), start (datetime) SecurityEvent +WindowsSecurity list_matching_processes Returns list of processes matching process name (all hosts) end (datetime), process_name (str), start (datetime) SecurityEvent +WindowsSecurity list_other_events Returns list of events other than logon and process on a host end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity list_processes_in_session Returns all processes on the host for a logon session end (datetime), host_name (str), logon_session_id (str), start (datetime) SecurityEvent +WindowsSecurity notable_events Return other significant Windows events not returned in other queries end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity schdld_tasks_and_services Returns scheduled tasks and services events (4698, 4700, 4697, 4702) end (datetime), host_name (str), start (datetime) SecurityEvent +WindowsSecurity summarize_events Summarize the events on a host by event type end (datetime), host_name (str), start (datetime) SecurityEvent ================== ================================ ================================================================================================================================== =============================================================================================================== =========================== @@ -209,95 +194,80 @@ Queries for Microsoft 365 Defender Data Environment identifier: M365D -============ ========================== ================================================================================================================================== ================================================================== =================== -QueryGroup Query Description Req-Params Table -============ ========================== ================================================================================================================================== ================================================================== =================== -M365D application_alerts Lists alerts associated with a cloud app or OAuth app app_name (str), end (datetime), start (datetime) AlertInfo -M365D host_alerts Lists alerts by for a specified hostname end (datetime), host_name (str), start (datetime) AlertInfo -M365D ip_alerts Lists alerts associated with a specified remote IP end (datetime), ip_address (str), start (datetime) AlertInfo -M365D list_alerts Retrieves list of alerts end (datetime), start (datetime) AlertInfo -M365D list_alerts_with_evidence Retrieves list of alerts with their evidence end (datetime), start (datetime) AlertInfo -M365D mail_message_alerts Lists alerts associated with a specified mail message end (datetime), message_id (str), start (datetime) AlertInfo -M365D mailbox_alerts Lists alerts associated with a specified mailbox end (datetime), mailbox (str), start (datetime) AlertInfo -M365D process_alerts Lists alerts associated with a specified process end (datetime), file_name (str), start (datetime) AlertInfo -M365D registry_key_alerts Lists alerts associated with a specified registry key end (datetime), key_name (str), start (datetime) AlertInfo -M365D sha1_alerts Lists alerts associated with a specified SHA1 hash end (datetime), file_hash (str), start (datetime) AlertInfo -M365D sha256_alerts Lists alerts associated with a specified SHA256 hash end (datetime), file_hash (str), start (datetime) AlertInfo -M365D url_alerts Lists alerts associated with a specified URL end (datetime), start (datetime), url (str) AlertInfo -M365D user_alerts Lists alerts associated with a specified user account_name (str), end (datetime), start (datetime) AlertInfo -MDATP file_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceProcessEvents -MDATP host_connections Lists connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents -MDATP ip_connections Lists network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents -MDATP list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents -MDATP list_filehash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceProcessEvents -MDATP list_files Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceProcessEvents -MDATP list_host_processes Lists all process creations for a host end (datetime), host_name (str), start (datetime) DeviceProcessEvents -MDATP process_cmd_line Lists all processes with a command line containing a string cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents -MDATP process_creations Lists all processes created by name or hash end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents -MDATP process_paths Lists all processes created from a path end (datetime), file_path (str), start (datetime) DeviceProcessEvents -MDATP protocol_connections Lists connections associated with a specified protocol end (datetime), protocol (str), start (datetime) DeviceNetworkEvents -MDATP url_connections Lists connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents -MDATP user_files Lists all files created by a user account_name (str), end (datetime), start (datetime) - -MDATP user_logons Lists all user logons by user account_name (str), end (datetime), start (datetime) - -MDATP user_network Lists all network connections associated with a user account_name (str), end (datetime), start (datetime) - -MDATP user_processes Lists all processes created by a user account_name (str), end (datetime), start (datetime) - -MDATPHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - -MDATPHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - -MDATPHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - -MDATPHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - -MDATPHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - -MDATPHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - -MDATPHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - -MDATPHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - -MDATPHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - -MDATPHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - -MDATPHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - -MDATPHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - -MDATPHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - -MDATPHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - -MDATPHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - -MDATPHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - -MDATPHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - -MDATPHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - -MDATPHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - -MDATPHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - -MDE accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - -MDE av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - -MDE b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - -MDE brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - -MDE cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - -MDE cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - -MDE cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - -MDE doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - -MDE dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - -MDE email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - -MDE email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - -MDE file_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceProcessEvents -MDE host_connections Lists connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents -MDE ip_connections Lists network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents -MDE list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents -MDE list_filehash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceProcessEvents -MDE list_files Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceProcessEvents -MDE list_host_processes Lists all process creations for a host end (datetime), host_name (str), start (datetime) DeviceProcessEvents -MDE malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - -MDE network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - -MDE powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - -MDE process_cmd_line Lists all processes with a command line containing a string cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents -MDE process_creations Lists all processes created by name or hash end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents -MDE process_paths Lists all processes created from a path end (datetime), file_path (str), start (datetime) DeviceProcessEvents -MDE protocol_connections Lists connections associated with a specified protocol end (datetime), protocol (str), start (datetime) DeviceNetworkEvents -MDE service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - -MDE smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - -MDE smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - -MDE tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - -MDE uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - -MDE url_connections Lists connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents -MDE user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - -MDE user_files Lists all files created by a user account_name (str), end (datetime), start (datetime) - -MDE user_logons Lists all user logons by user account_name (str), end (datetime), start (datetime) - -MDE user_network Lists all network connections associated with a user account_name (str), end (datetime), start (datetime) - -MDE user_processes Lists all processes created by a user account_name (str), end (datetime), start (datetime) - -============ ========================== ================================================================================================================================== ================================================================== =================== +============ ============================= ================================================================================================================================== ================================================================== =================== +QueryGroup Query Description Req-Params Table +============ ============================= ================================================================================================================================== ================================================================== =================== +M365D application_alerts Lists alerts associated with a cloud app or OAuth app app_name (str), end (datetime), start (datetime) AlertInfo +M365D host_alerts Lists alerts by for a specified hostname end (datetime), host_name (str), start (datetime) AlertInfo +M365D host_connections Returns connections by for a specified hostname end (datetime), host_name (str), start (datetime) DeviceNetworkEvents +M365D ip_alerts Lists alerts associated with a specified remote IP end (datetime), ip_address (str), start (datetime) AlertInfo +M365D ip_connections Returns network connections associated with a specified remote IP end (datetime), ip_address (str), start (datetime) DeviceNetworkEvents +M365D list_alerts Retrieves list of alerts end (datetime), start (datetime) AlertInfo +M365D list_alerts_with_evidence Retrieves list of alerts with their evidence end (datetime), start (datetime) AlertInfo +M365D list_connections Retrieves list of all network connections end (datetime), start (datetime) DeviceNetworkEvents +M365D list_file_events_for_filename Lists all file events by filename end (datetime), file_name (str), start (datetime) DeviceFileEvents +M365D list_file_events_for_hash Lists all file events by hash end (datetime), file_hash (str), start (datetime) DeviceFileEvents +M365D list_file_events_for_host Lists all file events for a host/device end (datetime), start (datetime) DeviceFileEvents +M365D list_file_events_for_path Lists all file events from files in a certain path end (datetime), path (str), start (datetime) DeviceFileEvents +M365D list_host_processes Return all process creations for a host for the specified time range end (datetime), host_name (str), start (datetime) DeviceProcessEvents +M365D mail_message_alerts Lists alerts associated with a specified mail message end (datetime), message_id (str), start (datetime) AlertInfo +M365D mailbox_alerts Lists alerts associated with a specified mailbox end (datetime), mailbox (str), start (datetime) AlertInfo +M365D process_alerts Lists alerts associated with a specified process end (datetime), file_name (str), start (datetime) AlertInfo +M365D process_cmd_line Lists all processes with a command line containing a string (all hosts) cmd_line (str), end (datetime), start (datetime) DeviceProcessEvents +M365D process_creations Return all processes with matching name or hash (all hosts) end (datetime), process_identifier (str), start (datetime) DeviceProcessEvents +M365D process_paths Return all processes with a matching path (part path) (all hosts) end (datetime), file_path (str), start (datetime) DeviceProcessEvents +M365D protocol_connections Returns connections associated with a specified protocol (port number) end (datetime), protocol (str), start (datetime) DeviceNetworkEvents +M365D registry_key_alerts Lists alerts associated with a specified registry key end (datetime), key_name (str), start (datetime) AlertInfo +M365D sha1_alerts Lists alerts associated with a specified SHA1 hash end (datetime), file_hash (str), start (datetime) AlertInfo +M365D sha256_alerts Lists alerts associated with a specified SHA256 hash end (datetime), file_hash (str), start (datetime) AlertInfo +M365D url_alerts Lists alerts associated with a specified URL end (datetime), start (datetime), url (str) AlertInfo +M365D url_connections Returns connections associated with a specified URL end (datetime), start (datetime), url (str) DeviceNetworkEvents +M365D user_alerts Lists alerts associated with a specified user account_name (str), end (datetime), start (datetime) AlertInfo +M365D user_files Return all files created by a user account_name (str), end (datetime), start (datetime) - +M365D user_logons Return all user logons for user name account_name (str), end (datetime), start (datetime) - +M365D user_network Return all network connections associated with a user account_name (str), end (datetime), start (datetime) - +M365D user_processes Return all processes created by a user account_name (str), end (datetime), start (datetime) - +M365DHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - +M365DHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - +M365DHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - +M365DHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - +M365DHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - +M365DHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - +M365DHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - +M365DHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - +M365DHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - +M365DHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - +M365DHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - +M365DHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - +M365DHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - +M365DHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - +M365DHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - +M365DHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - +M365DHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - +M365DHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - +M365DHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - +M365DHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - +MDEHunting accessibility_persistence This query looks for persistence or privilege escalation done using Windows Accessibility features. end (datetime), start (datetime) - +MDEHunting av_sites Pivot from downloads detected by Windows Defender Antivirus to other files downloaded from the same sites end (datetime), start (datetime) - +MDEHunting b64_pe Finding base64 encoded PE files header seen in the command line parameters end (datetime), start (datetime) - +MDEHunting brute_force Look for public IP addresses that failed to logon to a computer multiple times, using multiple accounts, and eventually succeeded. end (datetime), start (datetime) - +MDEHunting cve_2018_1000006l Looks for CVE-2018-1000006 exploitation end (datetime), start (datetime) - +MDEHunting cve_2018_1111 Looks for CVE-2018-1111 exploitation end (datetime), start (datetime) - +MDEHunting cve_2018_4878 This query checks for specific processes and domain TLD used in the CVE-2018-4878 end (datetime), start (datetime) - +MDEHunting doc_with_link Looks for a Word document attachment, from which a link was clicked, and after which there was a browser download. end (datetime), start (datetime) - +MDEHunting dropbox_link Looks for user content downloads from dropbox that originate from a link/redirect from a 3rd party site. end (datetime), start (datetime) - +MDEHunting email_link Look for links opened from mail apps – if a detection occurred right afterwards end (datetime), start (datetime) - +MDEHunting email_smartscreen Look for links opened from outlook.exe, followed by a browser download and then a SmartScreen app warning end (datetime), start (datetime) - +MDEHunting malware_recycle Finding attackers hiding malware in the recycle bin. end (datetime), start (datetime) - +MDEHunting network_scans Looking for high volume queries against a given RemoteIP, per ComputerName, RemotePort and Process end (datetime), start (datetime) - +MDEHunting powershell_downloads Finds PowerShell execution events that could involve a download. end (datetime), start (datetime) - +MDEHunting service_account_powershell Service Accounts Performing Remote PowerShell end (datetime), start (datetime) - +MDEHunting smartscreen_ignored Query for SmartScreen URL blocks, where the user has decided to run the malware nontheless. end (datetime), start (datetime) - +MDEHunting smb_discovery Query for processes that accessed more than 10 IP addresses over port 445 (SMB) - possibly scanning for network shares. end (datetime), start (datetime) - +MDEHunting tor Looks for Tor client, or for a common Tor plugin called Meek. end (datetime), start (datetime) - +MDEHunting uncommon_powershell Find which uncommon Powershell Cmdlets were executed on that machine in a certain time period. end (datetime), host_name (str), start (datetime), timestamp (str) - +MDEHunting user_enumeration The query finds attempts to list users or groups using Net commands end (datetime), start (datetime) - +============ ============================= ================================================================================================================================== ================================================================== =================== diff --git a/docs/source/data_acquisition/SentinelIncidents.rst b/docs/source/data_acquisition/SentinelIncidents.rst index da4d68610..1c876189a 100644 --- a/docs/source/data_acquisition/SentinelIncidents.rst +++ b/docs/source/data_acquisition/SentinelIncidents.rst @@ -17,11 +17,11 @@ See :py:meth:`list_incidents `_ +`Microsoft Sentinel API `__ and include the following key items: + - $top: this controls how many incidents are returned - - $filter: this accepts an OData query that filters the returned item. - (see `$filter parameter `_) + - $filter: this accepts an OData query that filters the returned item. https://learn.microsoft.com/graph/filter-query-parameter - $orderby: this allows for sorting results by a specific column .. code:: ipython3 diff --git a/docs/source/data_analysis/PivotFunctions.rst b/docs/source/data_analysis/PivotFunctions.rst index a2ec902c5..f3dcde6e3 100644 --- a/docs/source/data_analysis/PivotFunctions.rst +++ b/docs/source/data_analysis/PivotFunctions.rst @@ -1577,7 +1577,7 @@ as outputs, you can could imagine implementing this chain of operations as a series of calls to various pivot functions, taking the output from one and feeding it to the next, and so on. Pandas already supports stacking these kinds of operations in what is known as a -`fluent interface `_. +`fluent interface `__. Here is an example that chains three operations but without using any intermediate variables to store the results of each step. Each operation @@ -1787,7 +1787,7 @@ The name of the function to be run should be passed (as a string) in the The function **must** be a method of a pandas DataFrame - this includes built-in functions such as ``.plot``, ``.sort_values`` or a custom function added as a custom pd accessor function (see -`Extending pandas `_) +`Extending pandas `__) You can pass other named arguments to the ``tee_exec``. These will be passed to the ``df_func`` function. diff --git a/docs/source/extending/WritingDataProviders.rst b/docs/source/extending/WritingDataProviders.rst index a3b77f645..dd369f0cb 100644 --- a/docs/source/extending/WritingDataProviders.rst +++ b/docs/source/extending/WritingDataProviders.rst @@ -42,8 +42,8 @@ To implement a data provider you need to do the following: 1. Write the driver class ------------------------- -This must be derived from :py:class:`DriverBase` -(`DriverBase source `_). +This must be derived from :py:class:`DriverBase ` +(`DriverBase source `__). You should implement the following methods: - ``__init__`` @@ -86,7 +86,7 @@ section of your configuration settings from ``msticpyconfig.yaml``. Some existing drivers use an API key to authenticate, some use name/password and others use Azure Active Directory (AAD). See :py:class:`KqlDriver ` -(`KqlDriver source `_) +(`KqlDriver source `__) for an example of the latter.) @@ -206,7 +206,7 @@ follows: } See :py:class:`SplunkDriver ` -(`SplunkDriver source `_) +(`SplunkDriver source `__) for an example. Code: @@ -272,7 +272,7 @@ Add the provider as a DataEnvironment ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the enum :py:class:`DataEnvironment ` -(`DataEnvironments source `_) +(`DataEnvironments source `__) add an entry for your provider using the next available enum value. .. code-block:: Python3 @@ -307,7 +307,7 @@ Add an entry to the driver dynamic load table ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In the ``__init__.py`` module of data drivers -(`drivers sub-package __init__ source `_) +(`drivers sub-package __init__ source `__) .. code-block: Python3 :emphasize-lines: 10 diff --git a/docs/source/getting_started/Installing.rst b/docs/source/getting_started/Installing.rst index b9c079d41..cbbe4a094 100644 --- a/docs/source/getting_started/Installing.rst +++ b/docs/source/getting_started/Installing.rst @@ -206,21 +206,26 @@ Installing in Managed Spark compute in Azure Machine Learning Notebooks *MSTICPy* installation for Managed (Automatic) Spark Compute in Azure Machine Learning workspace requires different instructions since library installation is different. +.. note:: These notebook requires Azure ML Spark Compute. + If you are using it for the first time, follow the guidelines at + `Attach and manage a Synapse Spark pool in Azure Machine Learning (preview) + `__ -.. note:: These notebook requires Azure ML Spark Compute. If you are using - it for the first time, follow the guidelines mentioned here: - `Attach and manage a Synapse Spark pool in Azure Machine Learning (preview) `_ +Once you have completed the pre-requisites, you will see AzureML Spark Compute +in the dropdown menu for Compute. Select it and run any cell to start Spark Session. -Once you have completed the pre-requisites, you will see AzureML Spark Compute in the dropdown menu for Compute. Select it and run any cell to start Spark Session. Please refer to -`Managed (Automatic) Spark compute in Azure Machine Learning Notebooks `_ -for more detailed steps along with screenshots. - - - -In order to install any libraries in Spark compute, you need to use a conda file to configure a Spark session. -Please save below file as conda.yml , check the Upload conda file checkbox. You can modify the version number as needed. -Then, select Browse, and choose the conda file saved earlier with the Spark session configuration you want. +`Managed (Automatic) Spark compute in Azure Machine Learning Notebooks +`__ +for more guidance and screenshots. + +In order to install any libraries in Spark compute, you need to use a +conda file to configure a Spark session. +Please save below file as conda.yml , check the Upload conda file +checkbox. You can modify the version number as needed. +Then, select Browse, and choose the conda file saved earlier with +the Spark session configuration you want. +se, and choose the conda file saved earlier with the Spark session configuration you want. .. code-block:: yaml diff --git a/docs/source/visualization/FoliumMap.rst b/docs/source/visualization/FoliumMap.rst index 7549db94f..1a72881c9 100644 --- a/docs/source/visualization/FoliumMap.rst +++ b/docs/source/visualization/FoliumMap.rst @@ -4,7 +4,7 @@ Folium Map Plotting The :py:class:`FoliumMap` class is a wrapper around the Folium geo-mapping interactive mapping package. -See `Folium `_. +See `Folium `__. The MSTICPy Folium plotting can be used with DataFrames, IP addresses, locations, and geohashes as well as the MSTICPy diff --git a/msticpy/context/tilookup.py b/msticpy/context/tilookup.py index d2034cfa1..a584a61d5 100644 --- a/msticpy/context/tilookup.py +++ b/msticpy/context/tilookup.py @@ -58,7 +58,7 @@ def lookup_ioc( **kwargs, ) -> pd.DataFrame: """ - Lookup single IoC in active providers. + Lookup Threat Intelligence reports for a single IoC in active providers. Parameters ---------- @@ -88,6 +88,19 @@ def lookup_ioc( bool indicates whether a TI record was found in any provider list has an entry for each provider result + See Also + -------- + lookup_iocs : Lookup Threat Intelligence reports for a collection of IoCs. + + Notes + ----- + Queries active Threat Intelligence (TI) providers for a single + indicator of compromise (IoC). It returns results as a pandas + DataFrame. `ioc_type` can be used to specify the type (ipv4, + ipv6, dns, url, file_hash). If this is not supplied the + type is inferred using regular expressions. + By default, providers are queried asynchronously, in parallel. + """ ioc = ioc or kwargs.pop("observable", None) if ioc is None: @@ -114,7 +127,7 @@ def lookup_iocs( **kwargs, ) -> pd.DataFrame: """ - Lookup a collection of IoCs. + Lookup Threat Intelligence reports for a collection of IoCs in active providers. Parameters ---------- @@ -146,6 +159,21 @@ def lookup_iocs( pd.DataFrame DataFrame of results + See Also + -------- + lookup_ioc : Lookup Threat Intelligence reports for a single IoC. + + Notes + ----- + `lookup_iocs` queries active Threat Intelligence (TI) providers for + threat reports. It can accept input as a Python iterable or + a pandas dataframe. In the latter case, you also need to supply + the `ioc_col` parameter to indicate which column the IoC value can + be found. The `ioc_type_col` parameter is optional and can be used + to manually specify the IoC type for each row. If this is not supplied + the ioc types are inferred using regular expressions. + The results are returned as a pandas DataFrame. + """ return _make_sync( self._lookup_iocs_async( diff --git a/msticpy/context/vtlookupv3/vtlookupv3.py b/msticpy/context/vtlookupv3/vtlookupv3.py index 4c30e903e..783139ca8 100644 --- a/msticpy/context/vtlookupv3/vtlookupv3.py +++ b/msticpy/context/vtlookupv3/vtlookupv3.py @@ -386,7 +386,7 @@ async def _lookup_ioc_relationships_async( observable: str, vt_type: str, relationship: str, - limit: int = None, + limit: Optional[int] = None, all_props: bool = False, full_objects: bool = False, ): @@ -506,8 +506,9 @@ def lookup_ioc_relationships( observable: str, vt_type: str, relationship: str, - limit: int = None, + limit: Optional[int] = None, all_props: bool = False, + full_objects: bool = False, ) -> pd.DataFrame: """ Look up single IoC observable relationship links. @@ -524,6 +525,8 @@ def lookup_ioc_relationships( Relations limit all_props : bool, optional If True, return all properties, by default False + full_objects : bool, optional + If True, return the full object rather than just ID links. Returns ------- @@ -543,14 +546,23 @@ def lookup_ioc_relationships( try: return _make_sync( self._lookup_ioc_relationships_async( - observable, vt_type, relationship, limit, all_props=all_props + observable, + vt_type, + relationship, + limit, + all_props=all_props, + full_objects=full_objects, ) ) finally: self._vt_client.close() def lookup_ioc_related( - self, observable: str, vt_type: str, relationship: str, limit: int = None + self, + observable: str, + vt_type: str, + relationship: str, + limit: Optional[int] = None, ) -> pd.DataFrame: """ Look single IoC observable related items. @@ -675,7 +687,7 @@ def lookup_iocs_relationships( relationship: str, observable_column: str = ColumnNames.TARGET.value, observable_type_column: str = ColumnNames.TARGET_TYPE.value, - limit: int = None, + limit: Optional[int] = None, all_props: bool = False, ) -> pd.DataFrame: """ diff --git a/msticpy/data/core/data_providers.py b/msticpy/data/core/data_providers.py index 66a86157c..b0ac56abd 100644 --- a/msticpy/data/core/data_providers.py +++ b/msticpy/data/core/data_providers.py @@ -32,7 +32,7 @@ _COMPATIBLE_DRIVER_MAPPINGS = { "mssentinel": ["m365d"], "mde": ["m365d"], - "mssentinel_new": ["mssentinel"], + "mssentinel_new": ["mssentinel", "m365d"], "kusto_new": ["kusto"], } diff --git a/msticpy/data/core/query_provider_utils_mixin.py b/msticpy/data/core/query_provider_utils_mixin.py index 858d34cc9..18b202cd4 100644 --- a/msticpy/data/core/query_provider_utils_mixin.py +++ b/msticpy/data/core/query_provider_utils_mixin.py @@ -148,7 +148,11 @@ def list_data_environments(cls) -> List[str]: """ # pylint: disable=not-an-iterable - return [env for env in DataEnvironment.__members__ if env != "Unknown"] + return [ + de + for de in dir(DataEnvironment) + if de != "Unknown" and not de.startswith("_") + ] # pylint: enable=not-an-iterable def list_queries(self, substring: Optional[str] = None) -> List[str]: diff --git a/msticpy/data/queries/m365d/kql_mdatp_alerts.yaml b/msticpy/data/queries/m365d/kql_m365_alerts.yaml similarity index 100% rename from msticpy/data/queries/m365d/kql_mdatp_alerts.yaml rename to msticpy/data/queries/m365d/kql_m365_alerts.yaml diff --git a/msticpy/data/queries/m365d/kql_mdatp_file.yaml b/msticpy/data/queries/m365d/kql_m365_file.yaml similarity index 69% rename from msticpy/data/queries/m365d/kql_mdatp_file.yaml rename to msticpy/data/queries/m365d/kql_m365_file.yaml index 527223dcf..30e3083ad 100644 --- a/msticpy/data/queries/m365d/kql_mdatp_file.yaml +++ b/msticpy/data/queries/m365d/kql_m365_file.yaml @@ -1,9 +1,9 @@ metadata: version: 1 - description: MDATP Queries + description: M365D Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] - tags: ["file"] + data_families: [M365D] + tags: ["file", "process"] defaults: metadata: data_source: "file_events" @@ -11,7 +11,7 @@ defaults: table: description: Table name type: str - default: "DeviceProcessEvents" + default: "DeviceFileEvents" start: description: Query start time type: datetime @@ -27,7 +27,7 @@ defaults: type: str default: "Timestamp" sources: - list_files: + list_file_events_for_filename: description: Lists all file events by filename metadata: args: @@ -42,7 +42,7 @@ sources: file_name: description: Name of file type: str - file_path: + list_file_events_for_path: description: Lists all file events from files in a certain path metadata: args: @@ -56,7 +56,7 @@ sources: path: description: Full or partial path to search in type: str - list_filehash: + list_file_events_for_hash: description: Lists all file events by hash metadata: args: @@ -72,3 +72,23 @@ sources: description: Hash of file type: str aliases: hash + list_file_events_for_host: + description: Lists all file events for a host/device + metadata: + args: + query: ' + {table} + | where {time_column} >= datetime({start}) + | where {time_column} <= datetime({end}) + | where DeviceName == "{host_name}" or DeviceID == "{device_id}" + {add_query_items}' + uri: None + parameters: + host_name: + description: The name of the host/device + type: str + default: "" + device_id: + description: The device ID of the host/device + type: str + default: "" diff --git a/msticpy/data/queries/m365d/kql_mdatp_hunting.yaml b/msticpy/data/queries/m365d/kql_m365_hunting.yaml similarity index 99% rename from msticpy/data/queries/m365d/kql_mdatp_hunting.yaml rename to msticpy/data/queries/m365d/kql_m365_hunting.yaml index 4efd6e1b1..38fd73e7f 100644 --- a/msticpy/data/queries/m365d/kql_mdatp_hunting.yaml +++ b/msticpy/data/queries/m365d/kql_m365_hunting.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATPHunting, MDE] + data_families: [MDEHunting, M365DHunting] tags: ['user'] defaults: metadata: diff --git a/msticpy/data/queries/m365d/kql_mdatp_network.yaml b/msticpy/data/queries/m365d/kql_m365_network.yaml similarity index 86% rename from msticpy/data/queries/m365d/kql_mdatp_network.yaml rename to msticpy/data/queries/m365d/kql_m365_network.yaml index 31f8b18e2..182f5f93f 100644 --- a/msticpy/data/queries/m365d/kql_mdatp_network.yaml +++ b/msticpy/data/queries/m365d/kql_m365_network.yaml @@ -1,8 +1,8 @@ metadata: version: 1 - description: MDE Queries + description: M365D Network Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [M365D] tags: ["network"] defaults: metadata: @@ -39,7 +39,7 @@ sources: uri: None parameters: host_connections: - description: Lists connections by for a specified hostname + description: Returns connections by for a specified hostname metadata: args: query: ' @@ -55,7 +55,7 @@ sources: aliases: - hostname ip_connections: - description: Lists network connections associated with a specified remote IP + description: Returns network connections associated with a specified remote IP metadata: args: query: ' @@ -69,7 +69,7 @@ sources: description: Remote IP Address type: str url_connections: - description: Lists connections associated with a specified URL + description: Returns connections associated with a specified URL metadata: args: query: ' @@ -83,7 +83,7 @@ sources: description: Remote URL type: str protocol_connections: - description: Lists connections associated with a specified protocol + description: Returns connections associated with a specified protocol (port number) metadata: args: query: " diff --git a/msticpy/data/queries/m365d/kql_mdatp_process.yaml b/msticpy/data/queries/m365d/kql_m365_process.yaml similarity index 87% rename from msticpy/data/queries/m365d/kql_mdatp_process.yaml rename to msticpy/data/queries/m365d/kql_m365_process.yaml index 06bf98b65..08393a41a 100644 --- a/msticpy/data/queries/m365d/kql_mdatp_process.yaml +++ b/msticpy/data/queries/m365d/kql_m365_process.yaml @@ -1,8 +1,8 @@ metadata: version: 1 - description: MDATP Queries + description: M365D Process Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [M365D] tags: ["process"] defaults: metadata: @@ -28,7 +28,7 @@ defaults: default: "Timestamp" sources: list_host_processes: - description: Lists all process creations for a host + description: Return all process creations for a host for the specified time range metadata: args: query: ' @@ -43,7 +43,7 @@ sources: description: Name of host type: str process_creations: - description: Lists all processes created by name or hash + description: Return all processes with matching name or hash (all hosts) metadata: args: query: ' @@ -57,7 +57,7 @@ sources: description: Identifier for the process, filename, or hash type: str process_paths: - description: Lists all processes created from a path + description: Return all processes with a matching path (part path) (all hosts) metadata: args: query: ' @@ -71,7 +71,7 @@ sources: description: full or partial path type: str process_cmd_line: - description: Lists all processes with a command line containing a string + description: Lists all processes with a command line containing a string (all hosts) metadata: args: query: ' diff --git a/msticpy/data/queries/m365d/kql_mdatp_user.yaml b/msticpy/data/queries/m365d/kql_m365_user.yaml similarity index 86% rename from msticpy/data/queries/m365d/kql_mdatp_user.yaml rename to msticpy/data/queries/m365d/kql_m365_user.yaml index 0ba7d1b00..106b1eb8d 100644 --- a/msticpy/data/queries/m365d/kql_mdatp_user.yaml +++ b/msticpy/data/queries/m365d/kql_m365_user.yaml @@ -1,9 +1,9 @@ metadata: version: 1 - description: MDATP Queries + description: M365D User Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] - tags: ["user"] + data_families: [M365D] + tags: ["user", "account"] defaults: metadata: data_source: "user_events" @@ -24,7 +24,7 @@ defaults: default: "Timestamp" sources: user_logons: - description: Lists all user logons by user + description: Return all user logons for user name metadata: args: query: ' @@ -39,7 +39,7 @@ sources: description: Name of user type: str user_processes: - description: Lists all processes created by a user + description: Return all processes created by a user metadata: args: query: ' @@ -53,7 +53,7 @@ sources: description: Name of user type: str user_files: - description: Lists all files created by a user + description: Return all files created by a user metadata: args: query: ' @@ -67,7 +67,7 @@ sources: description: Name of user type: str user_network: - description: Lists all network connections associated with a user + description: Return all network connections associated with a user metadata: args: query: ' diff --git a/msticpy/data/queries/mde/kql_mdatp_alerts.yaml b/msticpy/data/queries/mde/kql_mdatp_alerts.yaml index bd2478585..50dafee4e 100644 --- a/msticpy/data/queries/mde/kql_mdatp_alerts.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_alerts.yaml @@ -1,8 +1,8 @@ metadata: version: 1 - description: MDATP Queries + description: MDE Alert Queries data_environments: [MDATP, MDE, M365D] - data_families: [MDATP] + data_families: [MDE, MDATP] tags: ["alert"] defaults: metadata: @@ -24,7 +24,7 @@ defaults: default: "" sources: list_alerts: - description: Retrieves list of alerts + description: Returns list of alerts for a specified time range metadata: args: query: " @@ -51,7 +51,7 @@ sources: description: Name of host type: str ip_alerts: - description: Lists alerts associated with a specified remote IP + description: Returns alerts associated with a specified remote IP metadata: pivot: short_name: alerts @@ -67,7 +67,7 @@ sources: description: Remote IP Address type: str url_alerts: - description: Lists alerts associated with a specified URL + description: Returns alerts associated with a specified URL metadata: pivot: short_name: alerts @@ -83,7 +83,7 @@ sources: description: Remote URL type: str sha1_alerts: - description: Lists alerts associated with a specified SHA1 hash + description: Returns alerts associated with a specified SHA1 hash metadata: pivot: short_name: alerts diff --git a/msticpy/data/queries/mde/kql_mdatp_file.yaml b/msticpy/data/queries/mde/kql_mdatp_file.yaml index 7e03480ab..7a6b94344 100644 --- a/msticpy/data/queries/mde/kql_mdatp_file.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_file.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [MDE, MDATP] tags: ["file"] defaults: metadata: @@ -11,7 +11,7 @@ defaults: table: description: Table name type: str - default: "DeviceProcessEvents" + default: "DeviceFileEvents" start: description: Query start time type: datetime @@ -22,9 +22,13 @@ defaults: description: Additional query clauses type: str default: "" + time_column: + description: The name of the column detailing the time the event was generated. + type: str + default: "Timestamp" sources: list_files: - description: Lists all file events by filename + description: Returns all file events by filename metadata: pivot: short_name: file_events_filename @@ -41,7 +45,7 @@ sources: description: Name of file type: str file_path: - description: Lists all file events from files in a certain path + description: Returns all file events from files in a path metadata: pivot: short_name: file_events_path @@ -57,7 +61,7 @@ sources: description: Full or partial path to search in type: str list_filehash: - description: Lists all file events by hash + description: Returns all file events by hash metadata: pivot: short_name: file_events_hash @@ -74,3 +78,68 @@ sources: description: Hash of file type: str aliases: hash + list_file_events_for_filename: + description: Lists all file events by filename + metadata: + args: + query: ' + {table} + | where {time_column} >= datetime({start}) + | where {time_column} <= datetime({end}) + | where FileName has "{file_name}" + {add_query_items}' + uri: None + parameters: + file_name: + description: Name of file + type: str + list_file_events_for_path: + description: Lists all file events from files in a certain path + metadata: + args: + query: ' + {table} + | where {time_column} >= datetime({start}) + | where {time_column} <= datetime({end}) + | where FolderPath contains "{path}" + {add_query_items}' + parameters: + path: + description: Full or partial path to search in + type: str + list_file_events_for_hash: + description: Lists all file events by hash + metadata: + args: + query: ' + {table} + | where {time_column} >= datetime({start}) + | where {time_column} <= datetime({end}) + | where SHA1 == "{file_hash}" or SHA256 == "{file_hash}" or MD5 == "{file_hash}" + {add_query_items}' + uri: None + parameters: + file_hash: + description: Hash of file + type: str + aliases: hash + list_file_events_for_host: + description: Lists all file events for a host/device + metadata: + args: + query: ' + {table} + | where {time_column} >= datetime({start}) + | where {time_column} <= datetime({end}) + | where DeviceName == "{host_name}" or DeviceID == "{device_id}" + {add_query_items}' + uri: None + parameters: + host_name: + description: The name of the host/device + type: str + default: "" + device_id: + description: The device ID of the host/device + type: str + default: "" \ No newline at end of file diff --git a/msticpy/data/queries/mde/kql_mdatp_hunting.yaml b/msticpy/data/queries/mde/kql_mdatp_hunting.yaml index 7e0af9743..66e0a34cb 100644 --- a/msticpy/data/queries/mde/kql_mdatp_hunting.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_hunting.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [MDEHunting, M365DHunting] tags: ['user'] defaults: metadata: diff --git a/msticpy/data/queries/mde/kql_mdatp_network.yaml b/msticpy/data/queries/mde/kql_mdatp_network.yaml index 31402ff83..06234cd80 100644 --- a/msticpy/data/queries/mde/kql_mdatp_network.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_network.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [MDE, MDATP] tags: ["network"] defaults: metadata: @@ -24,7 +24,7 @@ defaults: default: "" sources: list_connections: - description: Retrieves list of network connections + description: Returns list of network connections for a specified time range metadata: args: query: " @@ -35,7 +35,7 @@ sources: uri: None parameters: host_connections: - description: Lists alerts by for a specified hostname + description: Returns network connections a specified hostname/device name metadata: pivot: short_name: net_connections @@ -53,7 +53,7 @@ sources: aliases: - hostname ip_connections: - description: Lists alerts associated with a specified remote IP + description: Returns network connections associated with a specified remote IP metadata: pivot: short_name: net_connections @@ -69,7 +69,7 @@ sources: description: Remote IP Address type: str url_connections: - description: Lists alerts associated with a specified URL + description: Returns network connections associated with a specified URL metadata: pivot: short_name: net_connections @@ -85,7 +85,7 @@ sources: description: Remote URL type: str protocol_connections: - description: Lists alerts associated with a specified protocol + description: Returns network connections associated with a specified protocol (port number) metadata: args: query: " diff --git a/msticpy/data/queries/mde/kql_mdatp_process.yaml b/msticpy/data/queries/mde/kql_mdatp_process.yaml index b3a8c4d27..4c0e97060 100644 --- a/msticpy/data/queries/mde/kql_mdatp_process.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_process.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [MDE, MDATP] tags: ["process"] defaults: metadata: @@ -24,7 +24,7 @@ defaults: default: "" sources: list_host_processes: - description: Lists all process creations for a host + description: Return all process creations for a host for the specified time range metadata: pivot: short_name: processes @@ -41,7 +41,7 @@ sources: description: Name of host type: str process_creations: - description: Lists all processes created by name or hash + description: Return all processes with matching name or hash (all hosts) metadata: args: query: ' @@ -55,7 +55,7 @@ sources: description: Identifier for the process, filename, or hash type: str process_for_hash: - description: Lists all processes created for a file hash + description: Return all processes with a matching hash (all hosts) metadata: pivot: short_name: processes_by_hash @@ -71,7 +71,7 @@ sources: description: File hash of the process type: str process_paths: - description: Lists all processes created from a path + description: Return all processes with a matching path (part path) (all hosts) metadata: pivot: short_name: processes_by_path @@ -87,7 +87,7 @@ sources: description: full or partial path type: str process_cmd_line: - description: Lists all processes with a command line containing a string + description: Lists all processes with a command line containing a string (all hosts) metadata: pivot: short_name: processes_by_cmdline diff --git a/msticpy/data/queries/mde/kql_mdatp_user.yaml b/msticpy/data/queries/mde/kql_mdatp_user.yaml index a8af2d8b1..b8c8b2731 100644 --- a/msticpy/data/queries/mde/kql_mdatp_user.yaml +++ b/msticpy/data/queries/mde/kql_mdatp_user.yaml @@ -2,7 +2,7 @@ metadata: version: 1 description: MDATP Queries data_environments: [MDATP, MDE, M365D, LogAnalytics] - data_families: [MDATP, MDE] + data_families: [MDE, MDATP] tags: ["user"] defaults: metadata: @@ -20,7 +20,7 @@ defaults: default: "" sources: user_logons: - description: Lists all user logons by user + description: Return all user logons for user name metadata: pivot: short_name: logon_events @@ -41,7 +41,7 @@ sources: description: Name of user type: str user_processes: - description: Lists all processes created by a user + description: Return all processes created by a user metadata: pivot: short_name: process_events @@ -61,7 +61,7 @@ sources: description: Name of user type: str user_files: - description: Lists all files created by a user + description: Return all files created by a user metadata: pivot: short_name: file_events @@ -81,7 +81,7 @@ sources: description: Name of user type: str user_network: - description: Lists all network connections associated with a user + description: Return all network connections associated with a user metadata: pivot: short_name: network_events diff --git a/msticpy/data/queries/mssentinel/kql_sent_alert.yaml b/msticpy/data/queries/mssentinel/kql_sent_alert.yaml index 7f44e0443..7e125c650 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_alert.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_alert.yaml @@ -25,7 +25,7 @@ defaults: default: '' sources: list_alerts: - description: Retrieves list of alerts + description: Returns security alerts for a given time range metadata: args: query: ' @@ -46,7 +46,7 @@ sources: description: Query end time type: datetime list_alerts_counts: - description: Retrieves summary count of alerts by type + description: Returns summary count of alerts by type metadata: args: query: ' @@ -82,7 +82,7 @@ sources: description: 'The ID of the alert' type: str list_related_alerts: - description: Retrieves list of alerts with a common host, account or process + description: Returns alerts with a host, account or process entity metadata: pivot: short_name: alerts @@ -144,7 +144,7 @@ sources: type: str default: '\\' list_alerts_for_ip: - description: Retrieves list of alerts with a common IP Address + description: Returns alerts with the specified IP Address or addresses. metadata: args: query: ' diff --git a/msticpy/data/queries/mssentinel/kql_sent_az_dns.yaml b/msticpy/data/queries/mssentinel/kql_sent_az_dns.yaml index cfb154ad6..dee1bd00b 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_az_dns.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_az_dns.yaml @@ -24,7 +24,7 @@ defaults: default: '' sources: dns_lookups_for_domain: - description: Dns queries for a domain + description: Returns DNS query events for a specified domain metadata: pivot: short_name: queries @@ -43,7 +43,7 @@ sources: description: Domain to query for type: str dns_lookups_for_ip: - description: Dns queries for a domain + description: Returns Dns query events that contain a resolved IP address metadata: pivot: short_name: queries @@ -62,7 +62,7 @@ sources: description: IP lookup result to query for type: str dns_lookups_from_ip: - description: Dns queries for a domain + description: Returns Dns queries originating from a specified IP address metadata: pivot: short_name: queries_from_ip diff --git a/msticpy/data/queries/mssentinel/kql_sent_az_network.yaml b/msticpy/data/queries/mssentinel/kql_sent_az_network.yaml index 3f5186f05..d52c2379f 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_az_network.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_az_network.yaml @@ -54,7 +54,7 @@ defaults: VMRegion = Region_s' sources: az_net_analytics: - description: All Azure Network Analytics Data + description: Returns all Azure Network Flow (NSG) Data for a given host metadata: pivot: short_name: net_flows_depr diff --git a/msticpy/data/queries/mssentinel/kql_sent_azure.yaml b/msticpy/data/queries/mssentinel/kql_sent_azure.yaml index fc3f916d2..ea1628b01 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_azure.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_azure.yaml @@ -20,7 +20,7 @@ defaults: default: "" sources: list_aad_signins_for_account: - description: Lists Azure AD Signins for Account + description: Returns Azure AD Signins for Account metadata: pivot: short_name: signins @@ -57,7 +57,7 @@ sources: type: str default: "!!DEFAULT!!" list_aad_signins_for_ip: - description: Lists Azure AD Signins for an IP Address + description: Returns Azure AD Signins for an IP Address metadata: pivot: short_name: signins @@ -79,7 +79,7 @@ sources: description: The IP Address or list of Addresses type: list list_azure_activity_for_account: - description: Lists Azure Activity for Account + description: Returns Azure Activity for Account metadata: pivot: short_name: activity @@ -107,7 +107,7 @@ sources: description: The account name to find type: str list_azure_activity_for_ip: - description: Lists Azure Activity for Caller IP Address(es) + description: Returns Azure Activity for Caller IP Address(es) metadata: pivot: short_name: activity @@ -129,7 +129,7 @@ sources: description: The IP Address or list of Addresses type: list list_azure_activity_for_resource: - description: Lists Azure Activity for a Resource + description: Returns Azure Activity for an Azure Resource ID metadata: pivot: short_name: activity @@ -170,7 +170,7 @@ sources: type: str default: SigninLogs list_storage_ops_for_ip: - description: + description: Returns Storage Operations for an IP Address metadata: pivot: short_name: storage_ops @@ -203,7 +203,7 @@ sources: description: Client IP Address type: str list_storage_ops_for_hash: - description: + description: Returns Azure Storage Operations for an MD5 file hash args: query: ' union @@ -233,7 +233,7 @@ sources: description: MD5 hash of file type: str get_vmcomputer_for_ip: - description: Gets latest VMComputer record for IPAddress + description: Returns most recent VMComputer record for IPAddress metadata: pivot: short_name: vmcomputer @@ -254,7 +254,7 @@ sources: description: The IP Address of the VM type: str get_vmcomputer_for_host: - description: Gets latest VMComputer record for Host + description: Returns most recent VMComputer record for Host metadata: pivot: short_name: vmcomputer diff --git a/msticpy/data/queries/mssentinel/kql_sent_azuresentinel.yaml b/msticpy/data/queries/mssentinel/kql_sent_azuresentinel.yaml index 9784f87a0..85bb12d33 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_azuresentinel.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_azuresentinel.yaml @@ -28,7 +28,7 @@ defaults: default: "" sources: list_bookmarks: - description: Retrieves list of bookmarks + description: Retrieves list of bookmarks for a time range metadata: args: query: ' @@ -45,7 +45,7 @@ sources: {add_query_items}' parameters: list_bookmarks_for_entity: - description: Retrieves bookmarks for entity string + description: Retrieves bookmarks for a host, account, ip address, domain, url or other entity identifier metadata: pivot: short_name: bookmarks @@ -104,7 +104,7 @@ sources: type: str default: na get_bookmark_by_id: - description: Retrieves a single Bookmark by BookmarkId + description: Returns a single Bookmark by BookmarkId metadata: args: query: ' @@ -143,7 +143,7 @@ sources: description: Name or part name of Bookmark type: str list_bookmarks_for_tags: - description: Retrieves Bookmark by one or mare Tags + description: Returns Bookmark by one or more Tags metadata: args: query: ' @@ -166,7 +166,7 @@ sources: description: Bookmark tags type: list get_dynamic_summary_by_id: - description: Retrieves Dynamic Summary by SummaryId + description: Returns a Dynamic Summary by SummaryId metadata: args: query: ' @@ -187,7 +187,7 @@ sources: description: Dynamic Summary ID type: str get_dynamic_summary_by_name: - description: Retrieves Dynamic Summary by Name + description: Returns a Dynamic Summary by Name metadata: args: query: ' @@ -214,7 +214,7 @@ sources: description: Dynamic Summary Name type: str list_dynamic_summaries: - description: Retrieves Dynamic Summaries by date range + description: Returns all Dynamic Summaries by time range metadata: args: query: ' diff --git a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_activity.yaml b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_activity.yaml index b9de423d5..52ac087d2 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_activity.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_activity.yaml @@ -55,7 +55,7 @@ defaults: | extend TimeCreatedUtc=TimeGenerated' sources: sudo_activity: - description: All sudo activity + description: Returns all sudo activity for a host and account name args: query: ' {table} @@ -75,7 +75,7 @@ sources: type: str default: '' cron_activity: - description: All cron activity + description: Returns all cron activity for a host args: query: ' {table} @@ -88,7 +88,7 @@ sources: {add_query_items}' parameters: user_group_activity: - description: All user/group additions, deletions, and modifications + description: Returns all user/group additions, deletions, and modifications for a host args: query: ' {table} @@ -118,7 +118,7 @@ sources: {add_query_items}' parameters: summarize_events: - description: Returns all syslog activity for a host + description: Returns summarized syslog activity for a host args: query: ' {table} @@ -130,7 +130,7 @@ sources: {add_query_items}' parameters: notable_events: - description: Returns all syslog activity for a host + description: Returns all 'alert' and 'crit' syslog activity for a host args: query: ' {table} diff --git a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_apps.yaml b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_apps.yaml index ec4c9904f..14f52c945 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_apps.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_apps.yaml @@ -44,7 +44,7 @@ defaults: | extend TimeCreatedUtc=TimeGenerated' sources: squid_activity: - description: All squid proxy activity + description: Returns all squid proxy activity for a host args: query: ' {table} diff --git a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_logon.yaml b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_logon.yaml index fed5eabb0..7d2d72b4f 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_logon.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_logon.yaml @@ -71,7 +71,7 @@ defaults: | extend TimeCreatedUtc=TimeGenerated" sources: user_logon: - description: All user logon events on a host + description: User logon events on a host args: query: ' {table} @@ -115,7 +115,7 @@ sources: description: Hostname to query for type: str list_logons_for_account: - description: All successful user logon events for account (all hosts) + description: Successful user logon events for account name (all hosts) metadata: pivot: short_name: logons @@ -168,7 +168,7 @@ sources: description: The account name to search on type: str list_logons_for_source_ip: - description: All successful user logon events for source IP (all hosts) + description: Successful user logon events for source IP (all hosts) metadata: pivot: short_name: logons @@ -251,7 +251,7 @@ sources: description: Hostname to query for type: str list_host_logon_failures: - description: All failed user logon events on a host + description: Failed user logon events on a host metadata: pivot: short_name: logon_failures @@ -268,7 +268,7 @@ sources: description: Hostname where clause value: '| where Computer has "{host_name}"' list_ip_logon_failures: - description: All failed user logon events from an IP address + description: Failed user logon events from an IP address metadata: pivot: short_name: logon_failures @@ -285,7 +285,7 @@ sources: description: IP Address where clause value: '| where SyslogMessage has "{ip_address}"' list_account_logon_failures: - description: All failed user logon events from an IP address + description: All failed user logon events for account name metadata: pivot: short_name: logon_failures diff --git a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_sysmon.yaml b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_sysmon.yaml index 8fd1b90db..7d620d994 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_sysmon.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_lxsyslog_sysmon.yaml @@ -24,7 +24,7 @@ defaults: default: "true" sources: sysmon_process_events: - description: Get Process Events from a specified host + description: Sysmon Process Events on host metadata: args: query: | diff --git a/msticpy/data/queries/mssentinel/kql_sent_net.yaml b/msticpy/data/queries/mssentinel/kql_sent_net.yaml index e1dd46e0c..78eb42d6b 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_net.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_net.yaml @@ -69,7 +69,7 @@ defaults: default: "DestinationIP" sources: get_ips_for_host: - description: Gets the latest AzureNetworkAnalytics interface event for a host. + description: Returns the most recent Azure Network NSG Interface event for a host. metadata: pivot: short_name: interface @@ -93,7 +93,7 @@ sources: type: str default: AzureNetworkAnalytics_CL get_host_for_ip: - description: Gets the latest AzureNetworkAnalytics interface event for a host. + description: Returns the most recent Azure NSG Interface event for an IP Address. metadata: pivot: short_name: interface @@ -119,7 +119,7 @@ sources: type: str default: AzureNetworkAnalytics_CL get_heartbeat_for_host: - description: Retrieves latest OMS Heartbeat event for host. + description: Returns latest OMS Heartbeat event for host. metadata: data_families: [AzureNetwork, Network, Heartbeat] pivot: @@ -141,7 +141,7 @@ sources: type: str default: Heartbeat get_heartbeat_for_ip: - description: Retrieves latest OMS Heartbeat event for ip address. + description: Returns latest OMS Heartbeat event for ip address. metadata: data_families: [AzureNetwork, Network, Heartbeat] pivot: @@ -163,7 +163,7 @@ sources: type: str default: Heartbeat list_azure_network_flows_by_ip: - description: Retrieves Azure network analytics flow events. + description: Returns Azure NSG flow events for an IP Address. metadata: pivot: short_name: net_flows @@ -197,7 +197,7 @@ sources: or DestIP_s in ({ip_address_list}) )' list_azure_network_flows_by_host: - description: Retrieves Azure network analytics flow events. + description: Returns Azure NSG flow events for a host. metadata: pivot: short_name: net_flows @@ -228,7 +228,7 @@ sources: description: Query-specific where clause value: '| where VM_s has "{host_name}"' network_connections_to_url: - description: List of network connections to a URL + description: Returns connections to a URL or domain (CommonSecurityLog) metadata: pivot: short_name: url_connections @@ -249,10 +249,10 @@ sources: type: str default: "CommonSecurityLog" url: - description: URL to search for + description: URL or partial URL to search for type: str host_network_connections_csl: - descritpion: List network connections to and from a host using CommonSecurityLog + description: Returns network connections to and from a host (CommonSecurityLog) metadata: pivot: short_name: host_connections_csl @@ -277,7 +277,7 @@ sources: type: str default: "" ip_network_connections_csl: - descritpion: List network connections to and from an IP address using CommonSecurityLog + descritpion: Returns network connections to and from an IP address (CommonSecurityLog) metadata: pivot: short_name: ip_connections_csl @@ -302,7 +302,7 @@ sources: type: str default: "" all_network_connections_csl: - descritpion: List network connections to and from an IP address using CommonSecurityLog + descritpion: Returns all network connections for a time range (CommonSecurityLog) args: query: ' {table} @@ -318,7 +318,7 @@ sources: type: str default: "CommonSecurityLog" ips_by_host_csl: - descritpion: All IP addresses associated with a host using CommonSecurityLog + descritpion: Returns all IP addresses associated with a host (CommonSecurityLog) metadata: pivot: short_name: ips_csl @@ -344,7 +344,7 @@ sources: type: str default: "" hosts_by_ip_csl: - descritpion: All hosts associated with a IP addresses using CommonSecurityLog + descritpion: Returns hosts associated with a IP addresses (CommonSecurityLog) metadata: pivot: short_name: hosts_csl diff --git a/msticpy/data/queries/mssentinel/kql_sent_o365.yaml b/msticpy/data/queries/mssentinel/kql_sent_o365.yaml index eca1da6b6..97d7e15e1 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_o365.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_o365.yaml @@ -3,7 +3,7 @@ metadata: description: Kql Sentinel Azure data_environments: [LogAnalytics] data_families: [Office365] - tags: ['office', 'o365'] + tags: ['office', 'o365', 'sharepoint', 'exchange', 'onedrive', 'teams'] defaults: metadata: data_source: 'OfficeActivity' @@ -20,7 +20,7 @@ defaults: default: '' sources: list_activity_for_account: - description: Lists Office Activity for Account + description: Lists Office/O365 Activity for Account metadata: pivot: short_name: activity @@ -48,7 +48,7 @@ sources: description: The account name to find type: str list_activity_for_ip: - description: Lists Office Activity for Caller IP Address(es) + description: Lists Office/O365 Activity for Caller IP Address(es) metadata: pivot: short_name: activity @@ -70,7 +70,7 @@ sources: description: The IP Address or list of Addresses type: list list_activity_for_resource: - description: Lists Office Activity for a Resource + description: Lists Office/O365 Activity for a Resource (OfficeObjectId) metadata: pivot: short_name: activity diff --git a/msticpy/data/queries/mssentinel/kql_sent_threatintel.yaml b/msticpy/data/queries/mssentinel/kql_sent_threatintel.yaml index 652599fb2..d542ae167 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_threatintel.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_threatintel.yaml @@ -28,7 +28,7 @@ defaults: default: '' sources: list_indicators: - description: Retrieves list of all current indicators. + description: Returns list of all current indicators. metadata: args: query: ' @@ -41,7 +41,7 @@ sources: uri: None parameters: list_indicators_by_ip: - description: Retrieves list of indicators by IP Address + description: Returns list of indicators by IP Address metadata: args: query: ' @@ -71,7 +71,7 @@ sources: aliases: - observables list_indicators_by_hash: - description: Retrieves list of indicators by file hash + description: Returns list of indicators by file hash metadata: args: query: ' @@ -90,7 +90,7 @@ sources: aliases: - observables list_indicators_by_filepath: - description: Retrieves list of indicators by file path + description: Returns list of indicators by file path metadata: args: query: ' @@ -107,7 +107,7 @@ sources: description: List of observables type: list list_indicators_by_domain: - description: Retrieves list of indicators by domain + description: Returns list of indicators by domain metadata: args: query: ' @@ -131,7 +131,7 @@ sources: aliases: - observables list_indicators_by_email: - description: Retrieves list of indicators by email address + description: Returns list of indicators by email address metadata: args: query: ' @@ -153,7 +153,7 @@ sources: description: List of observables type: list list_indicators_by_url: - description: Retrieves list of indicators by URL + description: Returns list of indicators by URL metadata: args: query: ' diff --git a/msticpy/data/queries/mssentinel/kql_sent_timeseries.yaml b/msticpy/data/queries/mssentinel/kql_sent_timeseries.yaml index ebcf07463..08c7fa406 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_timeseries.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_timeseries.yaml @@ -49,7 +49,7 @@ defaults: default: 'count()' sources: get_timeseries_data: - description: Retrieves TimeSeriesData prepared to use with built-in KQL time series functions + description: Generic query to return TimeSeriesData for use with native KQL time series functions args: query: ' {table} @@ -60,7 +60,7 @@ sources: | make-series {aggregatecolumn}={aggregatefunction} on {timestampcolumn} from datetime({start}) to datetime({end}) step {timeframe} by {groupbycolumn} {add_query_items}' get_timeseries_decompose: - description: Time Series decomposition and anomalies generated using built-in KQL time series function- series_decompose + description: Generic Time Series decomposition using native KQL analysis (series_decompose) args: query: ' {table} @@ -73,7 +73,7 @@ sources: | mv-expand {aggregatecolumn} to typeof(double), {timestampcolumn} to typeof(datetime), baseline to typeof(double), seasonal to typeof(double), trend to typeof(long), residual to typeof(long) {add_query_items}' get_timeseries_anomalies: - description: Time Series filtered anomalies detected using built-in KQL time series function-series_decompose_anomalies + description: Time Series filtered anomalies using native KQL analysis (series_decompose_anomalies) args: query: ' {table} @@ -87,7 +87,7 @@ sources: | extend score = round(score,2) {add_query_items}' plot_timeseries_datawithbaseline: - description: Plot timeseries data using built-in KQL time series decomposition using built-in KQL render method + description: Plot of Time Series data using native KQL analysis and plot rendering (KQLMagic only) args: query: ' {table} @@ -100,9 +100,9 @@ sources: | mv-expand {aggregatecolumn} to typeof(double), {timestampcolumn} to typeof(datetime), baseline to typeof(long), seasonal to typeof(long), trend to typeof(long), residual to typeof(long) | project {timestampcolumn}, {aggregatecolumn}, baseline | render timechart with (title="Time Series Decomposition - Baseline vs Observed TimeChart") - {add_query_items}' + ' plot_timeseries_scoreanomolies: - description: Plot timeseries anomaly score using built-in KQL render method + description: Plot Time Series anomaly score using native KQL render (KQLMagic only) args: query: ' {table} diff --git a/msticpy/data/queries/mssentinel/kql_sent_winevent.yaml b/msticpy/data/queries/mssentinel/kql_sent_winevent.yaml index 7e3ab9c6e..36537a96b 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_winevent.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_winevent.yaml @@ -28,7 +28,7 @@ defaults: default: '' sources: list_host_events: - description: Retrieves list of all events on a host + description: Returns list of all events on a host metadata: pivot: short_name: all_events @@ -51,7 +51,7 @@ sources: type: str default: has list_host_events_by_id: - description: Retrieves list of events on a host + description: Returns list of specified event IDs on a host metadata: pivot: short_name: events_by_id @@ -79,7 +79,7 @@ sources: type: list default: has list_other_events: - description: Retrieves list of events other than logon and process on a host + description: Returns list of events other than logon and process on a host args: query: ' {table} @@ -108,7 +108,7 @@ sources: {add_query_items}' parameters: list_events_by_id: - description: Retrieves list of events on a host + description: Returns list of events on a host by EventID args: query: ' {table} @@ -122,7 +122,7 @@ sources: description: List of event IDs to match type: list summarize_events: - description: Summarizes a the events on a host + description: Summarize the events on a host by event type args: query: ' {table} @@ -141,7 +141,7 @@ sources: type: str default: has schdld_tasks_and_services: - description: Gets events related to scheduled tasks and services + description: Returns scheduled tasks and services events (4698, 4700, 4697, 4702) args: query: ' let account_evnt_ids = dynamic([4698, 4700, 4697, 4702]); @@ -171,7 +171,7 @@ sources: type: str default: has account_change_events: - description: Gets events related to account changes + description: Returns events related to account changes args: query: ' let account_evnt_ids = dynamic([4720, 4725, 4727, 4728, 4731, 4732, 4740, 4754, 4756, 4767]); @@ -190,7 +190,7 @@ sources: type: str default: has notable_events: - description: Get notebable Windows events not returned in other queries + description: Return other significant Windows events not returned in other queries args: query: ' let evnt_ids = dynamic([1102, 4657]); diff --git a/msticpy/data/queries/mssentinel/kql_sent_winevent_logon.yaml b/msticpy/data/queries/mssentinel/kql_sent_winevent_logon.yaml index 76994a99e..04fdd7cb7 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_winevent_logon.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_winevent_logon.yaml @@ -58,7 +58,7 @@ defaults: default: "true" sources: get_host_logon: - description: Retrieves the logon event for the session id on the host + description: Returns the logon event for the logon session id on a host metadata: pivot: short_name: logon_session @@ -81,7 +81,7 @@ sources: description: The logon session ID of the source process type: str list_host_logons: - description: Retrieves the logon events on the host + description: Returns the logon events on a host for time range metadata: pivot: short_name: logons @@ -102,7 +102,7 @@ sources: description: Name of host type: str list_host_logon_failures: - description: Retrieves the logon failure events on the host + description: Returns the logon failure events on a host for time range metadata: pivot: short_name: logon_failures @@ -127,7 +127,7 @@ sources: type: str default: "| where EventID == 4625" list_logons_by_account: - description: Retrieves the logon events for an account + description: Returns the logon success events for an account (all hosts) metadata: pivot: short_name: logons @@ -148,7 +148,7 @@ sources: description: The account name to find type: str list_logon_attempts_by_account: - description: Retrieves the logon events for an account + description: Retrieves all logon events for an account (all hosts) metadata: pivot: short_name: logon_attempts @@ -173,7 +173,7 @@ sources: type: str default: "| where EventID in (4624, 4625)" list_logon_failures_by_account: - description: Retrieves the logon failure events for an account + description: Returns the logon failure events for an account (all hosts) metadata: pivot: short_name: logon_failures @@ -198,7 +198,7 @@ sources: type: str default: "| where EventID == 4625" list_all_logons_by_host: - description: account all failed or successful logons to a host + description: Returns all failed or successful logons on a host metadata: pivot: short_name: logon_attempts @@ -225,7 +225,7 @@ sources: type: str default: "| where EventID in (4624, 4625)" list_logon_attempts_by_ip: - description: Retrieves the logon events for an IP Address + description: Returns the logon events for an IP Address (all hosts) metadata: pivot: short_name: logon_attempts diff --git a/msticpy/data/queries/mssentinel/kql_sent_winevent_proc.yaml b/msticpy/data/queries/mssentinel/kql_sent_winevent_proc.yaml index 352dd0b3f..0bf49575d 100644 --- a/msticpy/data/queries/mssentinel/kql_sent_winevent_proc.yaml +++ b/msticpy/data/queries/mssentinel/kql_sent_winevent_proc.yaml @@ -59,7 +59,7 @@ defaults: default: '\\' sources: list_host_processes: - description: Retrieves list of processes on a host + description: Returns list of processes on a host for a time range metadata: pivot: short_name: processes @@ -86,7 +86,7 @@ sources: type: str default: has list_matching_processes: - description: Retrieves list of processes matching process name + description: Returns list of processes matching process name (all hosts) metadata: pivot: short_name: similar_processes @@ -113,7 +113,7 @@ sources: type: str default: has get_process_tree: - description: Retrieves the process tree of a supplied process + description: Returns the process tree for process id, session id and host name. args: query: ' let start = datetime({start}); @@ -226,7 +226,7 @@ sources: description: The logon session ID of the source process type: str get_parent_process: - description: Retrieves the parent process of a supplied process + description: Returns the parent process of process (process id, session id and host name) metadata: pivot: short_name: parent_process @@ -286,7 +286,7 @@ sources: type: int default: 2 list_hosts_matching_commandline: - description: Retrieves processes on hosts with matching commandline + description: Returns processes on hosts with matching command line metadata: pivot: short_name: processes_with_same_commandline @@ -311,7 +311,7 @@ sources: description: The command line of the source process type: str list_processes_in_session: - description: Retrieves all processes on the host for a logon session + description: Returns all processes on the host for a logon session metadata: pivot: short_name: process_session @@ -336,12 +336,6 @@ sources: host_name: description: Name of host type: str - process_name: - description: Name of process - type: str - process_id: - description: The process ID of the source process - type: str logon_session_id: description: The logon session ID of the source process type: str \ No newline at end of file diff --git a/msticpy/init/pivot_init/vt_pivot.py b/msticpy/init/pivot_init/vt_pivot.py index 4a07756d2..dd3a8211b 100644 --- a/msticpy/init/pivot_init/vt_pivot.py +++ b/msticpy/init/pivot_init/vt_pivot.py @@ -98,6 +98,26 @@ class VTAPIScope(Flag): "Url": "Url", } +_FUNC_DOC = """ +{description} + +Parameters +---------- +observable: str + The observable value +limit: int + Relations limit +all_props : bool, optional + If True, return all properties, by default False +full_objects : bool, optional + If True, return the full object rather than just ID links. + +Returns +------- + Relationship Pandas DataFrame with the relationships of the entity + +""" + def init(): """Load VT3 Pivots if vt library is available.""" @@ -166,7 +186,8 @@ def _create_pivots(api_scope: Union[str, VTAPIScope, None]): vt_type=vt_type, relationship=relationship, ) - func_dict[_create_func_name(relationship)] = f_part + f_part.__doc__ = _create_func_doc(entity, relationship) + func_dict[relationship] = f_part ent_funcs[entity] = func_dict return ent_funcs @@ -174,8 +195,11 @@ def _create_pivots(api_scope: Union[str, VTAPIScope, None]): # pylint: enable=no-member -def _create_func_name(relationship): - return f"vt_{relationship}" +def _create_func_doc(entity, relationship): + """Create the relationship docstring.""" + fmt_name = " ".join(rel.capitalize() for rel in relationship.split("_")) + description = f"Lookup VirusTotal {fmt_name} for {entity}." + return _FUNC_DOC.format(description=description) def _get_relationships(vt_client, entity_id, vt_type, relationship): From a8a2de42ef5d77dda1de7d1aae1d5c0645a1f61b Mon Sep 17 00:00:00 2001 From: Ian Hellen Date: Mon, 24 Jul 2023 17:49:06 -0700 Subject: [PATCH 24/26] Update _version.py to 2.6.0 --- msticpy/_version.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/msticpy/_version.py b/msticpy/_version.py index 4344fd749..9fe71cdf3 100644 --- a/msticpy/_version.py +++ b/msticpy/_version.py @@ -1,2 +1,2 @@ """Version file.""" -VERSION = "2.5.3" +VERSION = "2.6.0" From f76650df76fc770d5ff9ea081cacbbc77d23a6f6 Mon Sep 17 00:00:00 2001 From: Shivam Sandbhor Date: Tue, 25 Jul 2023 21:20:46 +0530 Subject: [PATCH 25/26] Add CrowdSec TIProvider (#673) * Add CrowdSec TIProvider Signed-off-by: Shivam Sandbhor * Add user agent for crowdsec tiprovider Signed-off-by: Shivam Sandbhor * Implement review suggestions Signed-off-by: Shivam Sandbhor * Fix import error in tests Signed-off-by: Shivam Sandbhor * Extraneous braces in test data in test, unneeded ioc_param item in test data for CrowdSec Added CrowdSec settings entry to test msticpyconfig.yaml and msticpyconfig-test.yaml * Adding docstring to crowdsec.py parse_results --------- Signed-off-by: Shivam Sandbhor Co-authored-by: Ian Hellen --- docs/source/data_acquisition/TIProviders.rst | 14 +- docs/source/getting_started/msticpyconfig.rst | 6 + msticpy/context/tiproviders/__init__.py | 1 + msticpy/context/tiproviders/crowdsec.py | 75 ++++++++++ tests/context/test_tiproviders.py | 132 +++++++++++++++++- tests/msticpyconfig-test.yaml | 5 + tests/testdata/msticpyconfig.yaml | 5 + 7 files changed, 234 insertions(+), 4 deletions(-) create mode 100644 msticpy/context/tiproviders/crowdsec.py diff --git a/docs/source/data_acquisition/TIProviders.rst b/docs/source/data_acquisition/TIProviders.rst index 03d266986..d3c9f957d 100644 --- a/docs/source/data_acquisition/TIProviders.rst +++ b/docs/source/data_acquisition/TIProviders.rst @@ -30,6 +30,7 @@ Features - **IBM XForce** - **MS Sentinel TI** - **GreyNoise** + - **CrowdSec** - Other pseudo-TI providers are also included: @@ -268,6 +269,11 @@ fictitious - the format of the keys may differ from what is shown TenantID: 228d7b5f-4920-4f8e-872f-52072b92b651 Primary: True Provider: "AzSTI" + CrowdSec: + Args: + AuthKey: [PLACEHOLDER] + Primary: True + Provider: "CrowdSec" You need to tell `TILookup` to refresh its configuration. @@ -281,12 +287,13 @@ of providers loaded. .. parsed-literal:: + ['OTX - AlientVault OTX Lookup. (primary)', 'VirusTotal - VirusTotal Lookup. (primary)', 'XForce - IBM XForce Lookup. (primary)', 'GreyNoise - GreyNoise Lookup. (primary)', - 'AzSTI - Azure Sentinel TI provider class. (primary)', - 'OPR - Open PageRank Lookup. (secondary)'] + 'AzSTI - Microsoft Sentinel TI provider class. (primary)', + 'CrowdSec - CrowdSec CTI Smoke Lookup. (primary)'] .. warning:: Depending on the type of account that you have with a provider, they will typically impose a limit @@ -459,6 +466,7 @@ class method shows the current set of providers. VirusTotal XForce Intsights + CrowdSec You can view the list of supported query types for each provider with the ``show_query_types=True`` parameter. @@ -650,6 +658,8 @@ TILookup syntax +-------------+--------------+----------+---------------+---------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+---------+ | GreyNoise | 38.75.137.9 | ipv4 | None | False | Not found. | | https://api.greynoise.io/v3/community/38.75.137.9 | 404 | +-------------+--------------+----------+---------------+---------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+---------+ +| CrowdSec | 38.75.137.9 | ipv4 | None | False | {'Background Noise': 0, 'Overall Score': 0, 'First Seen': '2021-12-26T18:45:00+00:00', 'Last See... | {'ip_range_score': 0, 'ip': '38.75.137.9', 'ip_range': '38.75.136.0/23', 'as_name': 'AS-GLOBALTE... | https://cti.api.crowdsec.net/v2/smoke/38.75.137.9 | 200 | ++-------------+--------------+----------+---------------+---------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------+---------+ Pivot function syntax diff --git a/docs/source/getting_started/msticpyconfig.rst b/docs/source/getting_started/msticpyconfig.rst index 4244628d3..0184243dd 100644 --- a/docs/source/getting_started/msticpyconfig.rst +++ b/docs/source/getting_started/msticpyconfig.rst @@ -115,6 +115,7 @@ Currently supported provider names are: - IntSights - TorExitNodes - OpenPageRank +- CrowdSec .. code:: yaml @@ -136,6 +137,11 @@ Currently supported provider names are: KeyVault: Primary: False Provider: "OPR" + CrowdSec: + Args: + AuthKey: [PLACEHOLDER] + Primary: True + Provider: "CrowdSec" .. note:: You store values in the ``Args`` section as simple strings, as names of environment variables containing the value, or diff --git a/msticpy/context/tiproviders/__init__.py b/msticpy/context/tiproviders/__init__.py index 84c2675ba..42c977c34 100644 --- a/msticpy/context/tiproviders/__init__.py +++ b/msticpy/context/tiproviders/__init__.py @@ -14,6 +14,7 @@ __version__ = VERSION TI_PROVIDERS: Dict[str, Tuple[str, str]] = { + "CrowdSec": ("crowdsec", "CrowdSec"), "OTX": ("alienvault_otx", "OTX"), "AzSTI": ("azure_sent_byoti", "AzSTI"), "GreyNoise": ("greynoise", "GreyNoise"), diff --git a/msticpy/context/tiproviders/crowdsec.py b/msticpy/context/tiproviders/crowdsec.py new file mode 100644 index 000000000..f80dcdd4f --- /dev/null +++ b/msticpy/context/tiproviders/crowdsec.py @@ -0,0 +1,75 @@ +# ------------------------------------------------------------------------- +# Copyright (c) Microsoft Corporation. All rights reserved. +# Licensed under the MIT License. See License.txt in the project root for +# license information. +# -------------------------------------------------------------------------- +""" +CrowdSec Provider. + +Input can be a single IoC observable or a pandas DataFrame containing +multiple observables. Processing may require an API key and +processing performance may be limited to a specific number of +requests per minute for the account type that you have. + +""" +from typing import Any, Dict, Tuple + +from ..._version import VERSION +from ..http_provider import APILookupParams +from .ti_http_provider import HttpTIProvider +from .ti_provider_base import ResultSeverity + +__version__ = VERSION +__author__ = "Shivam Sandbhor" + + +class CrowdSec(HttpTIProvider): + """CrowdSec CTI Smoke Lookup.""" + + _BASE_URL = "https://cti.api.crowdsec.net" + + _QUERIES = { + "ipv4": APILookupParams( + path="/v2/smoke/{observable}", + headers={ + "x-api-key": "{AuthKey}", + "User-Agent": "crowdsec-msticpy-tiprovider/v1.0.0", + }, + ), + } + _QUERIES["ipv6"] = _QUERIES["ipv4"] + + def parse_results(self, response: Dict) -> Tuple[bool, ResultSeverity, Any]: + """Return the details of the response.""" + if self._failed_response(response): + return False, ResultSeverity.information, response["RawResult"]["message"] + + if response["RawResult"]["scores"]["overall"]["total"] <= 2: + result_severity = ResultSeverity.information + elif response["RawResult"]["scores"]["overall"]["total"] <= 3: + result_severity = ResultSeverity.warning + else: + result_severity = ResultSeverity.high + + return ( + True, + result_severity, + { + "Background Noise": response["RawResult"]["background_noise_score"], + "Overall Score": response["RawResult"]["scores"]["overall"]["total"], + "First Seen": response["RawResult"]["history"]["first_seen"], + "Last Seen": response["RawResult"]["history"]["last_seen"], + "Attack Details": ",".join( + [ + attack_detail["label"] + for attack_detail in response["RawResult"]["attack_details"] + ] + ), + "Behaviors": ",".join( + [ + behavior["name"] + for behavior in response["RawResult"]["behaviors"] + ] + ), + }, + ) diff --git a/tests/context/test_tiproviders.py b/tests/context/test_tiproviders.py index 5e76f9457..5927b11af 100644 --- a/tests/context/test_tiproviders.py +++ b/tests/context/test_tiproviders.py @@ -157,7 +157,15 @@ def _format_json_response(resp_data, **kwargs): "www.microsoft.com": ("hostname", "whois"), } -_TI_PROVIDER_TESTS = ["XForce", "OTX", "VirusTotal", "GreyNoise", "RiskIQ", "IntSights"] +_TI_PROVIDER_TESTS = [ + "XForce", + "OTX", + "VirusTotal", + "GreyNoise", + "RiskIQ", + "IntSights", + "CrowdSec", +] @pytest.mark.parametrize("provider_name", _TI_PROVIDER_TESTS) @@ -218,7 +226,15 @@ def verify_result(result, ti_lookup): for lu_result in result.to_dict(orient="records"): check.is_in( lu_result["Provider"], - ["OTX", "XForce", "VirusTotal", "GreyNoise", "RiskIQ", "IntSights"], + [ + "OTX", + "XForce", + "VirusTotal", + "GreyNoise", + "RiskIQ", + "IntSights", + "CrowdSec", + ], ) check.is_not_none(lu_result["Ioc"]) check.is_not_none(lu_result["IocType"]) @@ -887,6 +903,118 @@ def _get_riskiq_classification(): "Tags": ["tag"], }, }, + "https://cti.api.crowdsec.net": { + "response": { + "ip_range_score": 1, + "ip": "167.248.133.133", + "ip_range": "167.248.133.0/24", + "as_name": "CENSYS-ARIN-03", + "as_num": 398722, + "location": { + "country": "US", + "city": None, + "latitude": 1.751, + "longitude": -97.822, + }, + "reverse_dns": "scanner-03.ch1.censys-scanner.com", + "behaviors": [ + { + "name": "sip:bruteforce", + "label": "SIP Bruteforce", + "description": "IP has been reported for performing a SIP (VOIP) brute force attack.", + }, + { + "name": "tcp:scan", + "label": "TCP Scan", + "description": "IP has been reported for performing TCP port scanning.", + }, + ], + "history": { + "first_seen": f"{dt.datetime.now().isoformat(timespec='seconds')}+00:00", + "last_seen": f"{dt.datetime.now().isoformat(timespec='seconds')}+00:00", + "full_age": 490, + "days_age": 489, + }, + "classifications": { + "false_positives": [], + "classifications": [ + { + "name": "scanner:legit", + "label": "Legit scanner", + "description": "IP belongs to a company that scans the internet", + }, + { + "name": "scanner:censys", + "label": "Known Security Company", + "description": "IP belongs to a company that scans the internet: Censys.", + }, + { + "name": "community-blocklist", + "label": "CrowdSec Community Blocklist", + "description": "IP belongs to the CrowdSec Community Blocklist", + }, + ], + }, + "attack_details": [ + { + "name": "crowdsecurity/opensips-request", + "label": "SIP Bruteforce", + "description": "Detect brute force on VOIP/SIP services", + "references": [], + }, + { + "name": "firewallservices/pf-scan-multi_ports", + "label": "Port Scanner", + "description": "Detect tcp port scan", + "references": [], + }, + ], + "target_countries": { + "DE": 27, + "FR": 19, + "US": 16, + "EE": 16, + "HK": 5, + "DK": 2, + "GB": 2, + "FI": 2, + "KR": 2, + "SG": 2, + }, + "background_noise_score": 10, + "scores": { + "overall": { + "aggressiveness": 1, + "threat": 4, + "trust": 5, + "anomaly": 0, + "total": 3, + }, + "last_day": { + "aggressiveness": 0, + "threat": 0, + "trust": 0, + "anomaly": 0, + "total": 0, + }, + "last_week": { + "aggressiveness": 0, + "threat": 4, + "trust": 5, + "anomaly": 0, + "total": 3, + }, + "last_month": { + "aggressiveness": 0, + "threat": 4, + "trust": 5, + "anomaly": 0, + "total": 3, + }, + }, + "references": [], + }, + }, } _TOR_NODES = [ diff --git a/tests/msticpyconfig-test.yaml b/tests/msticpyconfig-test.yaml index a6bd04c78..715c650e6 100644 --- a/tests/msticpyconfig-test.yaml +++ b/tests/msticpyconfig-test.yaml @@ -90,6 +90,11 @@ TIProviders: AuthKey: "[PLACEHOLDER]" Primary: True Provider: Pulsedive + CrowdSec: + Args: + AuthKey: "[PLACEHOLDER]" + Primary: True + Provider: CrowdSec ContextProviders: ServiceNow: Primary: True diff --git a/tests/testdata/msticpyconfig.yaml b/tests/testdata/msticpyconfig.yaml index 35b7b01db..b9833b84a 100644 --- a/tests/testdata/msticpyconfig.yaml +++ b/tests/testdata/msticpyconfig.yaml @@ -67,6 +67,11 @@ TIProviders: AuthKey: "INTSIGHTS_KEY" Primary: False Provider: "IntSights" + CrowdSec: + Args: + AuthKey: "[PLACEHOLDER]" + Primary: True + Provider: CrowdSec OtherProviders: GeoIPLite: Args: From d854d9ed14aa0a1a7d058be0df9b7583a31c9947 Mon Sep 17 00:00:00 2001 From: Micah Babinski <63474467+mbabinski@users.noreply.github.com> Date: Wed, 26 Jul 2023 19:39:32 -0700 Subject: [PATCH 26/26] Added delete_watchlist_item method (#682) * Added delete_watchlist_item method * Black format sentinel_watchlists.py --------- Co-authored-by: Ian Hellen Co-authored-by: Pete Bryan --- msticpy/context/azure/sentinel_watchlists.py | 39 ++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/msticpy/context/azure/sentinel_watchlists.py b/msticpy/context/azure/sentinel_watchlists.py index e6445a7b3..48ccaf0ee 100644 --- a/msticpy/context/azure/sentinel_watchlists.py +++ b/msticpy/context/azure/sentinel_watchlists.py @@ -268,6 +268,45 @@ def delete_watchlist( raise CloudError(response=response) print(f"Watchlist {watchlist_name} deleted") + def delete_watchlist_item(self, watchlist_name: str, watchlist_item_id: str): + """ + Delete a Watchlist item. + + Parameters + ---------- + watchlist_name : str + The name of the watchlist with the item to be deleted + watchlist_item_id : str + The watchlist item ID to delete + + Raises + ------ + MsticpyUserError + If the specified Watchlist does not exist. + CloudError + If the API returns an error. + + """ + self.check_connected() # type: ignore + # Check requested watchlist actually exists + if not self._check_watchlist_exists(watchlist_name): + raise MsticpyUserError(f"Watchlist {watchlist_name} does not exist.") + + watchlist_url = ( + self.sent_urls["watchlists"] # type: ignore + + f"/{watchlist_name}/watchlistItems/{watchlist_item_id}" + ) + response = httpx.delete( + watchlist_url, + headers=get_api_headers(self.token), # type: ignore + params={"api-version": "2023-02-01"}, + timeout=get_http_timeout(), + ) + if response.status_code != 200: + raise CloudError(response=response) + + print(f"Item deleted from {watchlist_name}") + def _check_watchlist_exists( self, watchlist_name: str,