v2.7.0 - changing new kql/sentinel drivers to be defaults

conda/conda-reqs.txt

@@ -4,12 +4,12 @@ azure-core>=1.24.0
 azure-mgmt-core>=1.2.1
 azure-identity>=1.10.0
 azure-keyvault-secrets>=4.0.0
-azure-kusto-data>=4.0.0
+azure-kusto-data>=4.0.0, <=5.0.0
 azure-mgmt-compute>=4.6.2
 azure-mgmt-keyvault>=2.0.0
 azure-mgmt-network>=2.7.0
 azure-mgmt-resource>=16.1.0
-azure-monitor-query>=1.0.0
+azure-monitor-query>=1.0.0, <=2.0.0
 azure-storage-blob>=12.5.0
 beautifulsoup4>=4.0.0
 bokeh>=1.4.0, <4.0.0
@@ -19,7 +19,7 @@ dnspython>=2.0.0, <3.0.0
 folium>=0.9.0
 geoip2>=2.9.0
 html5lib
-httpx==0.24.1
+httpx>=0.23.0, <1.0.0
 ipython>=7.23.1
 ipywidgets>=7.4.2, <9.0.0
 keyring>=13.2.1

docs/source/DataAcquisition.rst

@@ -17,7 +17,6 @@ Individual Data Environments
    :maxdepth: 2
 
    data_acquisition/DataProv-MSSentinel
-   data_acquisition/DataProv-MSSentinel-New
    data_acquisition/DataProv-MSDefender
    data_acquisition/DataProv-MSGraph
    data_acquisition/DataProv-LocalData
@@ -26,10 +25,11 @@ Individual Data Environments
    data_acquisition/MordorData
    data_acquisition/DataProv-Sumologic
    data_acquisition/DataProv-Kusto
-   data_acquisition/DataProv-Kusto-New
    data_acquisition/DataProv-Cybereason
    data_acquisition/DataProv-OSQuery
    data_acquisition/DataProv-Velociraptor
+   data_acquisition/DataProv-MSSentinel-Legacy
+   data_acquisition/DataProv-Kusto-Legacy
 
 
 Built-in Data Queries

docs/source/data_acquisition/DataProv-Kusto-Legacy.rst

@@ -0,0 +1,302 @@
+Azure Data Explorer/Kusto Provider - Legacy Version
+===================================================
+
+Kusto Configuration
+-------------------
+
+Kusto Configuration in MSTICPy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can store your connection details in *msticpyconfig.yaml*.
+
+For more information on using and configuring *msticpyconfig.yaml* see
+:doc:`msticpy Package Configuration <../getting_started/msticpyconfig>`
+and :doc:`MSTICPy Settings Editor<../getting_started/SettingsEditor>`
+
+The settings in the file should look like the following two examples:
+
+.. code:: yaml
+
+    DataProviders:
+      ...
+      Kusto:
+        Args:
+          Cluster: https://mstic.kusto.windows.net
+          IntegratedAuth: True
+
+.. code:: yaml
+
+    DataProviders:
+      ...
+      Kusto:
+        Args:
+          Cluster: https://msticapp.kusto.windows.net
+          ClientId: 69d28fd7-42a5-48bc-a619-af56397b1111
+          TenantId: 69d28fd7-42a5-48bc-a619-af56397b9f28
+          ClientSecret: "[PLACEHOLDER]"
+
+
+We strongly recommend storing the client secret value
+in Azure Key Vault. You can replace the text value with a referenced
+to a Key Vault secret using the MSTICPy configuration editor.
+
+Your configuration when using Key Vault should look like the following:
+
+.. code:: yaml
+
+        Kusto:
+          Args:
+            Cluster: https://msticapp.kusto.windows.net
+            ClientId: 69d28fd7-42a5-48bc-a619-af56397b1111
+            TenantId: 69d28fd7-42a5-48bc-a619-af56397b9f28
+            ClientSecret:
+              KeyVault:
+
+You can create multiple instances of the Kusto settings for
+multiple clusters by adding
+an instance string to the "Kusto" section names
+
+.. code:: yaml
+
+    DataProviders:
+      ...
+      Kusto-mstic:
+        Args:
+          Cluster: https://mstic.kusto.windows.net
+          IntegratedAuth: True
+      Kusto-mstic2:
+        Args:
+          Cluster: https://mstic2.kusto.windows.net
+          IntegratedAuth: True
+      Kusto-msticapp:
+        Args:
+          Cluster: https://msticapp.kusto.windows.net
+          ClientId: 69d28fd7-42a5-48bc-a619-af56397b1111
+          TenantId: 69d28fd7-42a5-48bc-a619-af56397b9f28
+          ClientSecret:
+            KeyVault:
+
+
+Data Query Format for Kusto clusters
+------------------------------------
+
+The query template format for Kusto queries should look like
+the following.
+
+.. code:: yaml
+
+    metadata:
+        version: 1
+        description: Kusto Queries
+        data_environments: [Kusto]
+        data_families: [DeviceEvents.hostdata]
+        cluster: https://msticapp.kusto.windows.net
+        database: hostdata
+        tags: ["user"]
+    defaults:
+      parameters:
+        table:
+            description: Table name
+            type: str
+            default: "DeviceProcessEvents"
+        start:
+            description: Query start time
+            type: datetime
+            default: -30
+        end:
+            description: Query end time
+            type: datetime
+            default: 0
+        add_query_items:
+            description: Additional query clauses
+            type: str
+            default: ""
+    sources:
+        list_host_processes:
+            description: Lists all process creations for a host
+            metadata:
+            args:
+            query: '
+                {table}
+                | where Timestamp >= datetime({start})
+                | where Timestamp <= datetime({end})
+                | where DeviceName has "{host_name}"
+                {add_query_items}'
+            uri: None
+            parameters:
+            host_name:
+                description: Name of host
+                type: str
+
+Most of the query file is identical to queries for other drivers.
+However, the metadata section has additional items: ``cluster`` and
+``database``.
+
+.. code-block:: yaml
+   :emphasize-lines: 4, 5, 6
+
+    metadata:
+        version: 1
+        description: Kusto Queries
+        data_environments: [Kusto]
+        data_families: [ALIAS[.DATABASE]]
+        cluster: KUSTO_CLUSTER_URI
+        database: DATABASE
+
+
+The ``data_environments`` item must include "Kusto" in the list of
+applicable environments.
+
+You can specify the Kusto database to use in one of two ways:
+
+1. Use the ``database`` key.
+   Add the name of the database to connect to. The ``data_families`` key
+   is used as a container name when adding attributes. Whatever string
+   you specify here will be added as a prefix to the query name before attaching
+   the query to the query provider.
+
+2. Encode the database in the ``data_families`` item. If you do not
+   specify a database key explicitly, you should use a dot-separated string
+   for the data_families item:
+
+   - the first part (before the dot) is an alias that will be used as a prefix
+     when the queries are added to the query provider.
+   - the second part is the Kusto database containing the data to be queried.
+
+The ``cluster`` item in the query template file must match the ``Cluster``
+setting in the *msticpyconfig* setting described in the previous section.
+
+Here is are two examples.
+
+.. code-block:: yaml
+
+    metadata:
+        version: 1
+        description: Kusto Queries
+        data_environments: [Kusto]
+        data_families: [DeviceEvents]
+        database: hostdata
+        cluster: https://msticapp.kusto.windows.net
+
+.. code-block:: yaml
+
+    metadata:
+        version: 1
+        description: Kusto Queries
+        data_environments: [Kusto]
+        data_families: [DeviceEvents.hostdata]
+        cluster: https://msticapp.kusto.windows.net
+
+Queries using either of these metadata sections would be accessed and run as follows:
+
+.. code:: ipython3
+
+    kql_prov.DeviceEvents.list_host_processes(host_name="my_host", ...)
+
+The file-level ``metadata`` section applies to all queries in the file by
+default. You can specify a metadata section for individual queries. Any
+settings here will override the file-level settings.
+
+The example below shows overriding the ``data_families`` and ``cluster``
+entries for an individual query.
+
+.. code:: yaml
+
+    metadata:
+        version: 1
+        description: Kusto Queries
+        data_environments: [Kusto]
+        data_families: [DeviceEvents.hostdata]
+        cluster: https://msticapp.kusto.windows.net
+        tags: ["user"]
+    defaults:
+      parameters:
+        table:
+            description: Table name
+            type: str
+            default: "DeviceProcessEvents"
+        # ...
+    sources:
+        list_host_processes:
+            description: Lists all process creations for a host
+            metadata:
+                data_families: [DeviceEvents.scrubbeddata]
+                cluster: https://msticapp.kusto.windows.net
+            args:
+            query: '
+                {table}
+                | where Timestamp >= datetime({start})
+                | where Timestamp <= datetime({end})
+                | where DeviceName has "{host_name}"
+                {add_query_items}'
+            uri: None
+            parameters:
+            host_name:
+                description: Name of host
+                type: str
+
+Loading a QueryProvider for Kusto
+---------------------------------
+
+.. code:: ipython3
+
+        kql_prov = QueryProvider("Kusto")
+
+
+
+Connecting to a Kusto cluster
+-----------------------------
+
+If you are using query files (as described above) you do not need to explicitly
+connect - the connection will be made dynamically using the parameters in the
+query definition.
+
+To run add-hoc queries however, you need to explicitly connect to a cluster and
+database. The parameters required for connection to a Kusto cluster can be passed in
+a number of ways. You can provide a full connection string or parameters
+for ``cluster`` and ``database``. In the latter case, you must have configured
+settings for the cluster defined in your msticpyconfig.yaml.
+
+The ``cluster`` name can be either the actual cluster name or the alias
+that you used in your settings (i.e. the ``INSTANCE`` value in ``Kusto-INSTANCE``
+configuration key). To connect, you must also specify a valid database
+name in the cluster.
+
+
+.. code:: ipython3
+
+        kql_prov.connect(cluster="msticapp", database="hostdata")
+
+
+If you have queries defined (in template files) for multiple clusters
+and databases, you do not need to connect explicitly to each one.
+You can call these queries by name - the driver will dynamically
+read the connection parameters from the query file and attempt
+to authenticate to the cluster.
+
+Additional Kusto query parameters
+---------------------------------
+
+You can override the cluster and database for an individual
+query by supply the ``cluster`` and/or ``database`` parameters
+as query parameters.
+
+
+.. code:: ipython3
+
+        kql_prov.DeviceEvents.list_host_processes(
+            host_name="my_host",
+            cluster="https://somecluster.kusto.windows.net",
+            database="archive"
+            ...
+        )
+
+
+Other Kusto Documentation
+-----------------------------------
+
+For examples of using the Kusto provider, see the samples
+`Kusto Analysis Notebook<https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Kusto-Analysis.ipynb>`
+and `Kusto Ingest Notebook<https://github.com/microsoft/msticpy/blob/master/docs/notebooks/Kusto-Ingest.ipynb>`
+
+:py:mod:`Kusto driver API documentation<msticpy.data.drivers.kusto_driver>`