Better Error Reporting and Timeout for KQL Query Provider

[ashwin-patil]

Describe the bug
Currently Msticpy allows to connect to Data Environment such as Sentinel and once connected it can execute KQL queries. In some cases when KQL query returns large set of records the friendly error returned is not informative about and needs more details for user to troubleshoot.

To Reproduce
Steps to reproduce the behavior:

1. Import msticpy and initialize the query provider
2. Connect to Sentinel workspace.
3. Execute query which can return large set of records preferably with set notruncation option from a table such as SecurityEvents, CommonSecurityLog etc.
4. You may see an error as shown in the below screenshots.
5. Try to display the dataframe in another cell - optional.

Expected behavior
The Error returned should have clear description what went wrong executing the query.

Screenshots

Desktop (please complete the following information):

• OS: Windows, Linux
• Browser : Chrome, Edge
• Version: Edge 102

[FlorianBracq]

Hello,

I've been facing this issue too in the past, and the quick and dirty fix I found was:
msticpy\data\drivers\kql_driver.py (l430-431)
update from:

            else:
                err_contents = [f"Unknown error type: {q_info}"]

to

            else:
                if hasattr(result, "dataSetCompletion"):
                    ds_completion = result.dataSetCompletion
                    for completion in ds_completion:
                        if completion.get("HasErrors", False):
                            for one_api_error in completion.get("OneApiErrors", []):
                                error = one_api_error.get("error", {})
                                err_contents.extend(
                                    [
                                        f"StatusDescription {error.get('@message')}",
                                        f"(err_code: {error.get('code')})",
                                    ]
                                )
                else:
                    err_contents = [f"Unknown error type: {q_info}"]

There are 2 types of errors that can be raised: Either the result has more than 500k hits, or is bigger than 64MB https://docs.microsoft.com/en-us/azure/azure-monitor/service-limits

I need to run more checks, but the above code should cover both case.

The error that I don't know how to handle yet is the timeout: if the query runs for more than 10min, the client gets disconnected and kqlmagic does not seem to raise a clear error for that (or I completely missed it, which is highly likely too).

[ianhelle]

Should implement Florian's code or defer to #510

[ianhelle]

Added ability for use to set custom timeout.