updates to latest #2
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Bumps [httpx](https://github.com/encode/httpx) from 0.23.3 to 0.24.0. - [Release notes](https://github.com/encode/httpx/releases) - [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md) - [Commits](encode/httpx@0.23.3...0.24.0) --- updated-dependencies: - dependency-name: httpx dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
* Initial code for az_monitor_driver.py Added provider_settings.py for global proxy config Add settings.py as a facade module for common setttings functions Added lazy_import function and generic functions for implementing dynamic __getattr__ and __dir__ functions for __init__ modules Added ability to create (partial) WorkspaceConfig from connection string. Moved MpConfigEdit and MpConfigFile to dynamic imports in config/__init__.py Changed MSSentinel import in ce_azure_sentinel.py to be imported on demand Extended _execute_query to extract and supply timespan parameters to driver query functions Removed deprecated imports or nbtools and sectools from nbinit.py Importing get_config to msticpy/__init__.py * Removing use of multiple workspace IDs from az_monitor_driver * Adding workaround protection for process tree NA values for Bokeh 3.0 Moving bokeh 3.0 requirements back to 2.4.3 for compat with panel. * Initial code for azure_monitor_driver * Refactoring methods from QueryProvider into mixin classes * Azure kusto driver based on azure-data-kusto * Azure Kusto driver and Azure monitor updates: - Adding to DataEnvironments and drivers/__init__.py Adding raw Kusto response test data * Finished unit tests for test_azure_monitor_driver.py / azure_monitor_driver.py Implemented driver properties dictionary and use of DriverProps class to normalize naming in multiple drivers. Added code to allow driver to override environment name for reading queries. * Added documentation and fixed unit tests. Updated config documentation Fixed keyring_client.py test for valid backend Merged several global settings (proxies, http timeout) into "msticpy" section of config Added settings editor support for msticpy global settings in ce_msticpy.py Some Mypy fixes in file_browser.py Removed some commented-out code from query_source.py Added filtering for queries to query_source.py and data_providers.py - this allows individual kusto providers to show only queries that are relevant for the connected cluster. Additional test files and unit tests to support this. Added consistent support for proxy settings and timeout in azure_kusto_driver.py and azure_monitor_driver.py Updated setup.py, requirements-all.txt and conda-reqs.txt to add azure-kusto-data and azure-monitor-query packages. Added alternative to custom_mp_config test utility. This patches get_config calls in specified modules - more complex to set up but does not rely on a lock file, so allows more unit tests to run in parallel. * Fix for sentinel and azurekusto test failures * Fixing test and lint failures * Fixing settings source for test_sentinel_core.py Change folder name for test_azure_monitor_driver.py (not sure why but seems to have an effect on linux) * isort fixes in polling_detection.py pylint check name change in azure_auth_core.py adding validate_config to settings.py added ability to get workspace using case-insensitive ws name, ID or key name in wsconfig.py removed some debugging lines from sentinel_utils.py suppressed pylint error in data_providers.py fixed field initializer in query_template.py adding more logging to azure_monitor_driver.py adding more logging to azure_kusto_driver.py removing matplotlib import in nbinit, simplifying config loading code if config is already loaded. Moving matplotlib to extra in requirements.txt, setup.py Black re-format of test_periodogram_polling_detector.py Added tests for WorkspaceConfig to test_wsconfig.py Fixed test_azure_monitor_driver.py to use consistent settings. Fixed test_nbinit.py to avoid using default loaded settings. * Fix for 0.14.0 of statsmodels
* fix a critical bug of splunk result reader * typo pagenate -> paginate * Refactored code and reformatted long lines. Updated failing tests for new code. --------- Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
Ensure that the driver is loaded so that queries can be executed.
Add sphinx jquery extension
Add sphinxcontrib-jquery to RTD requirements
Fixing typo in RTD conf.py
* second stage adding extension Co-authored-by: Ian Hellen <Ian.Hellen@microsoft.com> * Working plugin code with tests. Still needs documentation * Fixing some mypy typing annotation errors * - Added documentation for PluginFramework - Added new document on createing TI providers - WritingTIAndContextProviders - Added docs for Development section in RTD - Adding ExtendingMsticpy section to RTD - moved sections for Queries, PivotFunctions, Creating data providers to this section - Have changed the internal _REQUIRED_PARAMS to use the same strings as in config and other places: - bulk edit of this http_provider, servicenow, alienvault_otx, greynoise, ibm_xforce, open_page_rank, virustotal Added ImportException trapping for mp_plugins.py Removing dev notebook - MSTICPyExtensions.ipynb * Moving some updates from Main into new extending/Queries.rst document * A couple of additions to docs - clarity and grammar * Test break due to merge * Addressing comments from Florian and Ryan's reviews. * Fixing issue with unit_test_lib not properly isolating temporary settings changes * Fixing bug in mp_plugins handling plugin paths * Adding locking around pivot data providers loader to fix config file for pivot tests. Changing test_nbinit.py to avoid using config locking and just use monkeypatch.setenv * Cleaned up ambiguity between DataEnvironment and environment_name in data_providers.py Removing unneeded comments in test_nbinit.py * Fixed error referencing "driver" variable in data_providers.py * Adding enviroment property for backward compatibility --------- Co-authored-by: Ian Hellen <Ian.Hellen@microsoft.com>
…ses to allow for old and new format. (#667) Build-breaking in test cases - no change to production code.
* add code to write yaml files * PR comment changes * replace requests with httpx * remove separate download * add docs and parameter types * remove unnecessary comment * add tests and headers * Remove old code file * rename files * remove unused imports * black reformatting * isort imports * pylint adjustments * update yaml loader for bandit checks * try with no prospector in requirements-dev.txt * test changes * fix pytest errors * test github pytest print * test pytest again * test pytest with file path changes * test if directories are being listed out in pytest * see what files are in the github runner * check directory contents again * list data available in testdata * see contents of test data folder * fix typo * try different base_dir_test_folder * check yaml_files * test _import_sentinel_query output * change split for _import_sentinel_query * test import_sentinel_query * see yaml file order * check sample query in list instead of equal * fix typo * sort lists for equivalence in organize test * try sort again * test with no datetime.now() in metadata * fix value unpacking error * switch set to list to fix issue * test getting_test_data * pylint fixes * check disabling certain pylint errors * try pylint again * fix mypy * additional linting fixes * fix flake8 trailing whitespace * add author and version * requested changes * fix linting issues * fix typerror * fix linting pt 3 * test mypy * test mypy pt 2 * try mypy pt 4 * mypy fix --------- Co-authored-by: Jannie Li <jannieli@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
Add more defense against import errors - in msticpy.__init__.py - this causes failures when help(msticpy) is used, causing loading of all dynamic attributes Better exception message on import error in azure_data.py Moving ResourceGraph query provider to only instantiate the provider when needed. Made data_query_reader.py produce warnings rather throw exceptions when encountering a bad query file
* Changed Bokeh requirements to work with panel 1.x Moved code from nbinit that checked CLI credentials to be run only in AML - avoiding non-core import. * Adding back MicrosoftSentinel as mp attribute Making the attribute failure more robust/informative. * Adding documentation URL to ignored links "https://username:password@proxy_host:port" appears in docstrings and documentation and is being checked for URL validity. * Checked in test version of readthedocs conf.py by mistake - reverting
* Azure monitor endpoint URL has changed format in v1.2.0 Unfortunately, older versions break with new format - so need a version-specific code branch. * Bug for missing attribute in kusto_driver - due to code change in kql_driver.py. Added documentation of need for additional packages to DataProv-Kusto-New.rst and DataProv-MSSentinel-New.rst * Changing ipwidgets requirement to <9.0.0 * Update DataProv-Kusto-New.rst Fixing name of `azure-kusto-data`
Updating version to 2.5.3
* Adding Velociraptor provider for local logs * Format of cluster name has changed in new KustoClient. Fixing test cases to allow for old and new format. * Minor updates for DataProv-Velociraptor.rst * Fixing comments in PR. Fixed bug in azure_kusto_driver and test_azure_kusto_driver Fixed some doc references. * Adding acknowledgement of Blue Team Village data
* Multithreading support when using multiple connections * Renamed additional connection column in results df * Fix flake warning * Adding threaded execution for both multiple instances and split queries for drivers that support multi-threading in query_provider_connections_mixin.py Added unit tests for threading code in test_async_queries.py Added driver properties to azure_kusto_driver.py, azure_monitor_driver.py and odata.py (mdatp_driver and security_graph_driver) Fixed test in test_azure_kusto_driver.py Some doc fixes to docstring in DataProv-Kusto-New.rst, DataProv-MSSentinel-New.rst, DataProviders.rst Unrelated doc fixes in polling_detection.py, Installing.rst, SentinelIncidents.rst * Fixing issue with unit_test_lib not properly isolating temporary settings changes * Adding locking around pivot data providers loader to fix config file for pivot tests. Changing test_nbinit.py to avoid using config locking and just use monkeypatch.setenv * Fixing some bugs in multi-threaded code - ensuring that loop is available if nested threading is happening. Converting pd.Timestamps to datetimes to allow serialization in Azure-azure_monitor_driver (in AZmon SDK) Fixing some logger info outputs in nbinit.py - that normally have no output. * Fxing handling of datetime/pd.Timestamp in query_provider_connections_mixin. Add more logging to data_providers.QueryProvider and azure_monitor_driver.py * Typo in data_providers (self.logger instead of logger) * Typo calling logger.info in data_providers.py Format of cluster name has changed in new KustoClient. Fixing test cases to allow for old and new format. * Cleaned up and refactored code in query_provider_connections_mixin.py * Typo in type annotation in query_provider_connections_mixin Reverted change to pop start and end parameters Fixed failing test in test_dataqueries.py::test_split_query_err * Removing redundant line in mdatp_driver * Bug in commit from merge - missing self._connection_str attribute in azure_kusto_driver.py --------- Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
Bumps [readthedocs-sphinx-ext](https://github.com/readthedocs/readthedocs-sphinx-ext) from 2.2.0 to 2.2.2. - [Commits](readthedocs/readthedocs-sphinx-ext@2.2.0...2.2.2) --- updated-dependencies: - dependency-name: readthedocs-sphinx-ext dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
Bumps [sphinx-rtd-theme](https://github.com/readthedocs/sphinx_rtd_theme) from 1.2.0 to 1.2.2. - [Changelog](https://github.com/readthedocs/sphinx_rtd_theme/blob/master/docs/changelog.rst) - [Commits](readthedocs/sphinx_rtd_theme@1.2.0...1.2.2) --- updated-dependencies: - dependency-name: sphinx-rtd-theme dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
Bumps [httpx](https://github.com/encode/httpx) from 0.24.0 to 0.24.1. - [Release notes](https://github.com/encode/httpx/releases) - [Changelog](https://github.com/encode/httpx/blob/master/CHANGELOG.md) - [Commits](encode/httpx@0.24.0...0.24.1) --- updated-dependencies: - dependency-name: httpx dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
* Changes to doc strings and queries to improve descriptions of pivot functions and queries. Renamed m365 query files Some corrections to kql_mdatp_file.yaml and kql_m365_file.yaml to reference DeviceFileEvents instead of DeviceProcessEvents * Missing column from query in kql_mdatp_file.yaml * Bunch of minor typo/URL link fixes in documents * Some missing doc files from module generation
* Add CrowdSec TIProvider Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> * Add user agent for crowdsec tiprovider Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> * Implement review suggestions Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> * Fix import error in tests Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> * Extraneous braces in test data in test, unneeded ioc_param item in test data for CrowdSec Added CrowdSec settings entry to test msticpyconfig.yaml and msticpyconfig-test.yaml * Adding docstring to crowdsec.py parse_results --------- Signed-off-by: Shivam Sandbhor <shivam.sandbhor@gmail.com> Co-authored-by: Ian Hellen <ianhelle@microsoft.com>
* Added delete_watchlist_item method * Black format sentinel_watchlists.py --------- Co-authored-by: Ian Hellen <ianhelle@microsoft.com> Co-authored-by: Pete Bryan <peter.bryan@microsoft.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+212
to
+214
| lambda driver: driver.client._kusto_cluster.startswith( | ||
| "https://test.kusto.windows.net" | ||
| ) |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Comment on lines
+327
to
+329
| lambda driver: driver.client._kusto_cluster.startswith( | ||
| "https://test.kusto.windows.net" | ||
| ), |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Comment on lines
+341
to
+343
| lambda driver: driver.client._kusto_cluster.startswith( | ||
| "https://random.kusto.windows.net" | ||
| ) |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.