[Bug]: Sentinel provider connection(fail_on_partial=True) fails to raise an exception

[JPvRiel]

Describe the bug

fail_on_partial=True paramter for the Sentinel connect method does not work as documented at https://msticpy.readthedocs.io/en/stable/data_acquisition/DataProv-MSSentinel.html.

fail_on_partial: bool - if True, raise an exception if only a partial result is returned. If False, return the partial result. The default is False.

To Reproduce

Config:

Azure:
  auth_methods:
  - env
  - vscode
  - cli
  - devicecode
  - interactive
  cloud: global
MSSentinel:
  Workspaces:
    Default:
      TenantId: <Tenant ID>
      WorkspaceId: <sentinel LA workspace ID>

Code:

import msticpy as mp

qp_sentinel = mp.QueryProvider('MSSentinel')
qp_sentinel.connect(fail_on_partial=True)

q_win_sec_large = '''
SigninLogs
| take 10000
'''
captured_exception = None
try:
    result_large_query = qp_sentinel.exec_query(q_win_sec_large)
except mp.common.exceptions.MsticpyDataQueryError as e:
    captured_exception = e
    print('Exception captured')

Output:

[/home/<somehome>/.venv/secnb/lib/python3.11/site-packages/msticpy/data/drivers/azure_monitor_driver.py:369](https://file+.vscode-resource.vscode-cdn.net/home/<somehome>/.venv/secnb/lib/python3.11/site-packages/msticpy/data/drivers/azure_monitor_driver.py:369): RuntimeWarning: Partial results returned. This may indicate a query timeout.
  warnings.warn(

Expected behavior

Calling code wanted to catch an exception, but no exception was raised. Only the default behavior of logging a warning occurred. Calling code cannot do anything useful with just a warning.

Arguably, a safer default would be to raise an an exception and users should need to opt into unsafe 'warn' only failure modes. E.g. if I were running a query to hunt for an IoC, concluding there's no IoC is a risk given partial query results and just a small one-line subtle warning.

Screenshots and/or Traceback

[/home/<somehome>/.venv/secnb/lib/python3.11/site-packages/msticpy/data/drivers/azure_monitor_driver.py:369](https://file+.vscode-resource.vscode-cdn.net/home/<somehome>/.venv/secnb/lib/python3.11/site-packages/msticpy/data/drivers/azure_monitor_driver.py:369): RuntimeWarning: Partial results returned. This may indicate a query timeout.
  warnings.warn(

Environment (please complete the following information):

• Python Version: 3.11
• OS: Ubuntu
• Python environment: venv
• MSTICPy Version: 2.15.0

[ianhelle]

Good points.
I'll fix the fail-on-partial flag. The original intent was not to fail by default - because throwing an exception would result in no data being returned. My thinking was that if it had failed because it was a really gnarly long-running query a user might be happier to get partial results - as long as they were warned that they were partial.
I'll see if I can add a config property so that you can change its behavior to always fail on partial without having to specify for each query.

[ianhelle]

Actually, looking at the code, the fail_on_partial parameter is a parameter to the exec_query method, not the connect method.
I could change the code so that you can also supply it as a connect parameter.
Just checking how easy it would be to add it as a configuration setting to the workspace definitions

[ianhelle]

I've added the ability to specify the fail_on_partial parameter in either connect() or when you create the query provider.
See the linked PR.