Added multithreading support for additional connections (+fixes)

[d3vzer0]

Support for running a query across multiple connections (with optional async operation)

It is common for data services to be spread across multiple tenants or workloads. E.g., multiple Sentinel workspaces,
Microsoft Defender subscriptions or Splunk instances. You can use the QueryProvider to run a query across multiple connections and return the results in a single DataFrame.

To create a multi-instance provider, create an instance of a QueryProvider for your data source and execute the connect() method to connect to the first instance of your data service.

Then use the add_connection() method. This takes the same parameters as the connect() method (the parameters for this method vary by data provider).

add_connection() also supports an alias parameter to allow you to refer to the connection by a friendly name.

    qry_prov = QueryProvider("MSSentinel")
    qry_prov.connect(workspace="Workspace1")
    qry_prov.add_connection(workspace="Workspace2, alias="Workspace2")
    qry_prov.list_connections()

When you now run a query for this provider, the query will be run on all of the connections and the results will be returned as a single dataframe.

    test_query = '''
        SecurityAlert
        | take 5
        '''

    query_test = qry_prov.exec_query(query=test_query)
    query_test.head()

Some of the MSTICPy drivers support asynchronous execution of queries against multiple instances, so that the time taken to run the query is much reduced compared to running the queries sequentially. Drivers that support asynchronous queries will use this automatically. The initial set of multi-threaded drivers are:

• MSSentinel_New (the new version of the MSSentinel driver)
• Kusto_New (the new version of the Kusto/Azure Data Explorer driver)

By default, the queries will use at most 4 concurrent threads. You can override this by initializing the QueryProvider with the
max_threads parameter to set it to the number of threads you want.

    qry_prov = QueryProvider("MSSentinel", max_threads=10)

Multi-threaded support for split/shared queries

MSTICPy has supported splitting large queries by time-slice for a while. This is documented here Splitting a Query into time chunks. With this release, we've added asynchronous support for this (if the driver supports threaded/async operation) so that multiple chunks of the query will run in parallel.

    qry_prov.SecurityAlert.list_alerts(start=start, end=end, split_by="1d")

Use the parameter split_query_by or split_by to specify a time range (the time unit uses the same syntax as pandas time intervals - e.g. "1D", "4h", etc. - the the pandas documentation for more details on this).

In this release sharding is also supported for ad hoc queries as long as you add "start" and "end" parameters to the query (this is still experimental, so let us know if you have issues with this).

[ianhelle]

I think the only change that would be good is changing MsticpyInstace to DataProviderInstance.
Looks great though!!!

Needs some further review

[ianhelle]

@d3vzer0 - I'm going to make some additions to this to add the same capability for splitting queries by time.
I've made some refactoring changes in PR #656 - I want to put all of the multi-connection functionality in a separate mixin class. So I'll get back to this when this PR is merged and synced with this one.
Awesome stuff though - thanks for the work.

[ianhelle]

Hey @d3vzer0,
Inspired by the PR, I implemented this for both multiple connections and for split (by time interval) queries.
Since the QueryProvider class was getting a bit out of hand, I moved all of the connections/threading code into its own mixin class. I haven't full tested this yet but it seems to work for both scenarios.
I've also added this all to the docs so that it's visible to more than just you and me. :-D
Let me know what you think.

[ianhelle]

/azpipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ianhelle]

/azpipelines run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).