microsoft · ianhelle · Jul 6, 2023 · Jun 30, 2023 · Jul 3, 2023 · Jul 3, 2023
diff --git a/docs/source/Development.rst b/docs/source/Development.rst
@@ -16,8 +16,8 @@ an improvement to an existing feature that you have thought
 about such as a new data connector or threat intelligence provider,
 or a completely new feature category.
 
-If you don’t have a specific idea in mind take a look at the
-Issues page on GitHub: `https://github.com/microsoft/msticpy/issues`__
+If you don't have a specific idea in mind take a look at the
+`Issues page on GitHub <https://github.com/microsoft/msticpy/issues>`__
 
 This page tracks a range of issues, enhancements, and features that
 members of the community have thought of. The MSTICPy team uses these

diff --git a/docs/source/api/msticpy.common.utility.rst b/docs/source/api/msticpy.common.utility.rst
@@ -16,4 +16,3 @@ Submodules
    msticpy.common.utility.ipython
    msticpy.common.utility.package
    msticpy.common.utility.types
-   msticpy.common.utility.yaml_loader
diff --git a/docs/source/api/msticpy.common.utility.yaml_loader.rst b/docs/source/api/msticpy.common.utility.yaml_loader.rst
diff --git a/docs/source/api/msticpy.data.drivers.rst b/docs/source/api/msticpy.data.drivers.rst
@@ -27,5 +27,6 @@ Submodules
    msticpy.data.drivers.odata_driver
    msticpy.data.drivers.resource_graph_driver
    msticpy.data.drivers.security_graph_driver
+   msticpy.data.drivers.sentinel_query_reader
    msticpy.data.drivers.splunk_driver
    msticpy.data.drivers.sumologic_driver
diff --git a/docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst b/docs/source/api/msticpy.data.drivers.sentinel_query_reader.rst
@@ -0,0 +1,7 @@
+msticpy.data.drivers.sentinel\_query\_reader module
+===================================================
+
+.. automodule:: msticpy.data.drivers.sentinel_query_reader
+   :members:
+   :undoc-members:
+   :show-inheritance:
diff --git a/docs/source/data_acquisition/AzureBlobStorage.rst b/docs/source/data_acquisition/AzureBlobStorage.rst
@@ -129,7 +129,7 @@ See :py:mod:`delete_blob<msticpy.data.storage.azure_blob_storage.AzureBlobStorag
 Generate a SAS Token for a Blob
 -------------------------------
 
-``get_sas_token`` generates a `SAS token <https://docs.microsoft.com/azure/storage/common/storage-sas-overview>`_ for the specified blob.
+``get_sas_token`` generates a `SAS token <https://docs.microsoft.com/azure/storage/common/storage-sas-overview>`__ for the specified blob.
 By default the token generated is valid for read access for 7 days but permissions can be modified with the
 ``permission`` keyword, and validity time-frame with the ``start`` and ``end`` keywords.
 The returned string is a full URI for the blob, with the SAS token appended.

diff --git a/docs/source/data_acquisition/DataProviders.rst b/docs/source/data_acquisition/DataProviders.rst
@@ -96,7 +96,7 @@ details) with:
     )
 
 
-For more details see :py:class:`QueryProvider API<msticpy.data.data_providers.QueryProvider>`.
+For more details see :py:class:`QueryProvider API <msticpy.data.data_providers.QueryProvider>`.
 
 
 Connecting to a Data Environment

diff --git a/docs/source/data_acquisition/DataQueries.rst b/docs/source/data_acquisition/DataQueries.rst
diff --git a/docs/source/data_acquisition/SentinelIncidents.rst b/docs/source/data_acquisition/SentinelIncidents.rst
@@ -17,11 +17,11 @@ See :py:meth:`list_incidents <msticpy.context.azure_sentinel.MicrosoftSentinel.l
 This returns a DataFrame with details of incidents. By default this will return the 50 latest incidents.
 It is possible to pass a set of parameters to `.list_incidents` to adjust the incidents returned via the `params` parameter.
 These parameters follow the format of the
-`Microsoft Sentinel API <https://learn.microsoft.com/rest/api/securityinsights/stable/incidents/list>`_
+`Microsoft Sentinel API <https://learn.microsoft.com/rest/api/securityinsights/stable/incidents/list>`__
 and include the following key items:
+
  - $top: this controls how many incidents are returned
- - $filter: this accepts an OData query that filters the returned item.
-   (see `$filter parameter <https://learn.microsoft.com/graph/filter-query-parameter>`_)
+ - $filter: this accepts an OData query that filters the returned item. https://learn.microsoft.com/graph/filter-query-parameter
  - $orderby: this allows for sorting results by a specific column
 
 .. code:: ipython3

diff --git a/docs/source/data_analysis/PivotFunctions.rst b/docs/source/data_analysis/PivotFunctions.rst
@@ -1577,7 +1577,7 @@ as outputs, you can could imagine implementing this chain of operations as
 a series of calls to various pivot functions, taking the output from one
 and feeding it to the next, and so on. Pandas already supports
 stacking these kinds of operations in what is known as a
-`fluent interface <https://en.wikipedia.org/wiki/Fluent_interface>`_.
+`fluent interface <https://en.wikipedia.org/wiki/Fluent_interface>`__.
 
 Here is an example that chains three operations but without using
 any intermediate variables to store the results of each step. Each operation
@@ -1787,7 +1787,7 @@ The name of the function to be run should be passed (as a string) in the
 The function **must** be a method of a pandas DataFrame - this includes
 built-in functions such as ``.plot``, ``.sort_values`` or a custom function
 added as a custom pd accessor function (see
-`Extending pandas <https://pandas.pydata.org/pandas-docs/stable/development/extending.html?highlight=accessor>`_)
+`Extending pandas <https://pandas.pydata.org/pandas-docs/stable/development/extending.html?highlight=accessor>`__)
 
 You can pass other named arguments to the ``tee_exec``. These
 will be passed to the ``df_func`` function.

diff --git a/docs/source/extending/WritingDataProviders.rst b/docs/source/extending/WritingDataProviders.rst
@@ -42,8 +42,8 @@ To implement a data provider you need to do the following:
 1. Write the driver class
 -------------------------
 
-This must be derived from :py:class:`DriverBase<msticpy.data.drivers.driver_base.DriverBase>`
-(`DriverBase source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/driver_base.py>`_).
+This must be derived from :py:class:`DriverBase <msticpy.data.drivers.driver_base.DriverBase>`
+(`DriverBase source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/driver_base.py>`__).
 You should implement the following methods:
 
 - ``__init__``
@@ -86,7 +86,7 @@ section of your configuration settings from ``msticpyconfig.yaml``.
 
 Some existing drivers use an API key to authenticate, some use name/password and others
 use Azure Active Directory (AAD). See :py:class:`KqlDriver <msticpy.data.drivers.kql_driver.KqlDriver>`
-(`KqlDriver source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/kql_driver.py>`_)
+(`KqlDriver source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/kql_driver.py>`__)
 for an example
 of the latter.)
 
@@ -206,7 +206,7 @@ follows:
         }
 
 See :py:class:`SplunkDriver <msticpy.data.drivers.splunk_driver.SplunkDriver>`
-(`SplunkDriver source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/splunk_driver.py>`_)
+(`SplunkDriver source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/splunk_driver.py>`__)
 for an example.
 
 Code:
@@ -272,7 +272,7 @@ Add the provider as a DataEnvironment
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 In the enum :py:class:`DataEnvironment <msticpy.data.query_defns.DataEnvironment>`
-(`DataEnvironments source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/query_defns.py>`_)
+(`DataEnvironments source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/query_defns.py>`__)
 add an entry for your provider using the next available enum value.
 
 .. code-block:: Python3
@@ -307,7 +307,7 @@ Add an entry to the driver dynamic load table
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 In the ``__init__.py`` module of data drivers
-(`drivers sub-package __init__ source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/__init__.py>`_)
+(`drivers sub-package __init__ source <https://github.com/microsoft/msticpy/tree/main/msticpy/data/drivers/__init__.py>`__)
 
 .. code-block: Python3
   :emphasize-lines: 10

diff --git a/docs/source/getting_started/Installing.rst b/docs/source/getting_started/Installing.rst
@@ -206,21 +206,26 @@ Installing in Managed Spark compute in Azure Machine Learning Notebooks
 *MSTICPy* installation for Managed (Automatic) Spark Compute in Azure Machine Learning workspace requires
 different instructions since library installation is different.
 
+.. note:: These notebook requires Azure ML Spark Compute.
+   If you are using it for the first time, follow the guidelines at
+   `Attach and manage a Synapse Spark pool in Azure Machine Learning (preview)
+   <https://learn.microsoft.com/azure/machine-learning/how-to-manage-synapse-spark-pool?tabs=studio-ui>`__
 
-.. note:: These notebook requires Azure ML Spark Compute. If you are using
-   it for the first time, follow the guidelines mentioned here:
-   `Attach and manage a Synapse Spark pool in Azure Machine Learning (preview) <https://learn.microsoft.com/azure/machine-learning/how-to-manage-synapse-spark-pool?tabs=studio-ui>`_
+Once you have completed the pre-requisites, you will see AzureML Spark Compute
+in the dropdown menu for Compute. Select it and run any cell to start Spark Session.
 
-Once you have completed the pre-requisites, you will see AzureML Spark Compute in the dropdown menu for Compute. Select it and run any cell to start Spark Session.
 Please refer to
-`Managed (Automatic) Spark compute in Azure Machine Learning Notebooks <https://learn.microsoft.com/azure/machine-learning/interactive-data-wrangling-with-apache-spark-azure-ml>`_
-for more detailed steps along with screenshots.
-
-
-
-In order to install any libraries in Spark compute, you need to use a conda file to configure a Spark session.
-Please save below file as conda.yml , check the Upload conda file checkbox. You can modify the version number as needed.
-Then, select Browse, and choose the conda file saved earlier with the Spark session configuration you want.
+`Managed (Automatic) Spark compute in Azure Machine Learning Notebooks
+<https://learn.microsoft.com/azure/machine-learning/interactive-data-wrangling-with-apache-spark-azure-ml>`__
+for more guidance and screenshots.
+
+In order to install any libraries in Spark compute, you need to use a
+conda file to configure a Spark session.
+Please save below file as conda.yml , check the Upload conda file
+checkbox. You can modify the version number as needed.
+Then, select Browse, and choose the conda file saved earlier with
+the Spark session configuration you want.
+se, and choose the conda file saved earlier with the Spark session configuration you want.
 
 .. code-block:: yaml
 

diff --git a/docs/source/visualization/FoliumMap.rst b/docs/source/visualization/FoliumMap.rst
@@ -4,7 +4,7 @@ Folium Map Plotting
 The :py:class:`FoliumMap<msticpy.vis.foliummap.FoliumMap>`
 class is a wrapper around the Folium geo-mapping interactive
 mapping package.
-See `Folium <https://python-visualization.github.io/folium/>`_.
+See `Folium <https://python-visualization.github.io/folium/>`__.
 
 The MSTICPy Folium plotting can be used with DataFrames,
 IP addresses, locations, and geohashes as well as the MSTICPy

diff --git a/msticpy/context/tilookup.py b/msticpy/context/tilookup.py
@@ -58,7 +58,7 @@ def lookup_ioc(
         **kwargs,
     ) -> pd.DataFrame:
         """
-        Lookup single IoC in active providers.
+        Lookup Threat Intelligence reports for a single IoC in active providers.
 
         Parameters
         ----------
@@ -88,6 +88,19 @@ def lookup_ioc(
             bool indicates whether a TI record was found in any provider
             list has an entry for each provider result
 
+        See Also
+        --------
+        lookup_iocs : Lookup Threat Intelligence reports for a collection of IoCs.
+
+        Notes
+        -----
+        Queries active Threat Intelligence (TI) providers for a single
+        indicator of compromise (IoC). It returns results as a pandas
+        DataFrame. `ioc_type` can be used to specify the type (ipv4,
+        ipv6, dns, url, file_hash). If this is not supplied the
+        type is inferred using regular expressions.
+        By default, providers are queried asynchronously, in parallel.
+
         """
         ioc = ioc or kwargs.pop("observable", None)
         if ioc is None:
@@ -114,7 +127,7 @@ def lookup_iocs(
         **kwargs,
     ) -> pd.DataFrame:
         """
-        Lookup a collection of IoCs.
+        Lookup Threat Intelligence reports for a collection of IoCs in active providers.
 
         Parameters
         ----------
@@ -146,6 +159,21 @@ def lookup_iocs(
         pd.DataFrame
             DataFrame of results
 
+        See Also
+        --------
+        lookup_ioc : Lookup Threat Intelligence reports for a single IoC.
+
+        Notes
+        -----
+        `lookup_iocs` queries active Threat Intelligence (TI) providers for
+        threat reports. It can accept input as a Python iterable or
+        a pandas dataframe. In the latter case, you also need to supply
+        the `ioc_col` parameter to indicate which column the IoC value can
+        be found. The `ioc_type_col` parameter is optional and can be used
+        to manually specify the IoC type for each row. If this is not supplied
+        the ioc types are inferred using regular expressions.
+        The results are returned as a pandas DataFrame.
+
         """
         return _make_sync(
             self._lookup_iocs_async(

diff --git a/msticpy/context/vtlookupv3/vtlookupv3.py b/msticpy/context/vtlookupv3/vtlookupv3.py
@@ -386,7 +386,7 @@ async def _lookup_ioc_relationships_async(
         observable: str,
         vt_type: str,
         relationship: str,
-        limit: int = None,
+        limit: Optional[int] = None,
         all_props: bool = False,
         full_objects: bool = False,
     ):
@@ -506,8 +506,9 @@ def lookup_ioc_relationships(
         observable: str,
         vt_type: str,
         relationship: str,
-        limit: int = None,
+        limit: Optional[int] = None,
         all_props: bool = False,
+        full_objects: bool = False,
     ) -> pd.DataFrame:
         """
         Look up single IoC observable relationship links.
@@ -524,6 +525,8 @@ def lookup_ioc_relationships(
             Relations limit
         all_props : bool, optional
             If True, return all properties, by default False
+        full_objects : bool, optional
+            If True, return the full object rather than just ID links.
 
         Returns
         -------
@@ -543,14 +546,23 @@ def lookup_ioc_relationships(
         try:
             return _make_sync(
                 self._lookup_ioc_relationships_async(
-                    observable, vt_type, relationship, limit, all_props=all_props
+                    observable,
+                    vt_type,
+                    relationship,
+                    limit,
+                    all_props=all_props,
+                    full_objects=full_objects,
                 )
             )
         finally:
             self._vt_client.close()
 
     def lookup_ioc_related(
-        self, observable: str, vt_type: str, relationship: str, limit: int = None
+        self,
+        observable: str,
+        vt_type: str,
+        relationship: str,
+        limit: Optional[int] = None,
     ) -> pd.DataFrame:
         """
         Look single IoC observable related items.
@@ -675,7 +687,7 @@ def lookup_iocs_relationships(
         relationship: str,
         observable_column: str = ColumnNames.TARGET.value,
         observable_type_column: str = ColumnNames.TARGET_TYPE.value,
-        limit: int = None,
+        limit: Optional[int] = None,
         all_props: bool = False,
     ) -> pd.DataFrame:
         """

diff --git a/msticpy/data/core/data_providers.py b/msticpy/data/core/data_providers.py
@@ -32,7 +32,7 @@
 _COMPATIBLE_DRIVER_MAPPINGS = {
     "mssentinel": ["m365d"],
     "mde": ["m365d"],
-    "mssentinel_new": ["mssentinel"],
+    "mssentinel_new": ["mssentinel", "m365d"],
     "kusto_new": ["kusto"],
 }
 

diff --git a/msticpy/data/core/query_provider_utils_mixin.py b/msticpy/data/core/query_provider_utils_mixin.py
@@ -148,7 +148,11 @@ def list_data_environments(cls) -> List[str]:
 
         """
         # pylint: disable=not-an-iterable
-        return [env for env in DataEnvironment.__members__ if env != "Unknown"]
+        return [
+            de
+            for de in dir(DataEnvironment)
+            if de != "Unknown" and not de.startswith("_")
+        ]
         # pylint: enable=not-an-iterable
 
     def list_queries(self, substring: Optional[str] = None) -> List[str]:

diff --git a/.../data/queries/m365d/kql_mdatp_alerts.yaml → ...y/data/queries/m365d/kql_m365_alerts.yaml b/.../data/queries/m365d/kql_mdatp_alerts.yaml → ...y/data/queries/m365d/kql_m365_alerts.yaml