Compatibility and test cases

Some fixes for url_summary, host_network_summary and miscellaneous shared code.
Adding more unit tests.
Ensuring unit tests do not call online services.
Fixing pandas and pkg_resources deprecation warnings

What's Changed

• Ianhelle/url-summary-fixes-2024-09-23 by @ianhelle in #47

Full Changelog: v1.2.2...v1.2.3

Compatibility fixes

What's Changed

• Updates for msticpy compatibility by @ianhelle in #46

Full Changelog: v1.2.1...v1.2.2

v1.2.1 - UrlSummary fix

What's Changed

• Adding new azure pipelines to repo by @ianhelle in #43
• Misc build and linter fixes by @ianhelle in #44
• Ianhelle/tldextract fix 2024 07 22 by @ianhelle in #45

Full Changelog: v.1.2.0...v1.2.1

v.1.2.0

What's Changed

• Test and doc updates by @ianhelle in #39
• [Fix] Added MANIFEST.in to include YAML files by @rjaakke in #38
• Fixes for MSTICPy 2.9 compatibility by @ianhelle in #41
• Ianhelle/fix build errors 2024 02 11 by @ianhelle in #42
• Adding patch to mock GeoIP provider in Folium map by @ianhelle in #40

New Contributors

• @rjaakke made their first contribution in #38

Full Changelog: v1.0.1...v.1.2.0

Version 1.0.1

This release contains several fixes and enhancements to individual notebooklets.
Also in this release a function to generate template notebooklets so you can build your own and import them into the package.

    import msticnb as nb

    nb.create_template(nb_name="MyNotebooklet", folder="mynotebooklet")

What's Changed

• Updated and fixed host notebooks by @petebryan in #35
• Fix import custom Nblts and added create_template by @ianhelle in #34
• Fixed Linting Issues by @petebryan in #36

Full Changelog: v1.0.0...v1.1.0

Release 1.0 of MSTIC Notebooklets

MSTIC Notebooklets has been around for a while but we are about to start using in production notebooks so decided it was time to make the transition to v 1.0.0.

New Features and major changes

New Notebooklets

Host Network Summary

Retrieves Azure NSG and/or MDE network connection summary for hosts retrieving TI, GeoIP and Whois information for the remote IPs. Also displays folium map of IP locations. See HostNetworkSummary for more details

URL Summary

Retrieves summary information about URLs and domains including the following:

• Threat intelligence reports
• WhoIs records
• Certificate information
• Optional screenshot of target web page
• Relate Sentinel alerts and bookmarks
• Host activity related to the endpoint using MDE connection data, syslog entries, Windows process command lines.
• Network connection activity related to the endpoints

Updated notebooklets

Host Summary

Update to work with either Windows or Linux hosts. The data retrieved obviously varies a little between the two platforms but you can now invoke the summary on a host without having to worry about which OS is installed.

See HostSummary for more details

IP Summary

Updated to add several data sources such as CommonSecurityLog and MDE data. New data categories returned include accounts and hosts related to the IP (i.e. have connections to or from the IP).

See IPSummary for more details.

Support modules

msticnb.nblib.ti and msticnb.nblib.iptools

The TI module is new and provides high-level functions for using MSTICPy's TILookup functionality in notebooks.
Additional functionality added the iptools module.

Global changes

• The notebooklets have been updated to work with MSTICPy 2.0 and should continue to work with recent prior releases.
• Notebooklet base class has two new methods - data_properties and vis_properties - making it easy to list the different major categories of result data from the notebook results class.

Feature Refresher

Notebooklets via Pivots

Most of the notebooklets have Pivot version. When you import msticnb and run nb.init() pivot functions are created on the entities corresponding to the notebooklet functionality. E.g.

IpAddress.nblt.ip_summary()
Host.nblt.host_network_summary()

### Access Notebooklet properties and methods from the Notebooklet results class.

Something that we implemented a while ago that is worth remembering. The notebooklet results class returned from the notebooklets `run` method, contains data sets such as pandas dataframes, Bokeh visualizations and other data. The results class maintains a link to the Notebooklet that created it and lets you access the notebooklet attributes as if they were attributes of the results class.

```python
    result = my_notebooklet.run(value="my_host"....)
    # get_additional_data() is a method of the my_notebooklet class
    result.get_additional_data()

    # this also works
    result.data_properties()

This is especially helpful if you are invoking a notebooklet via a pivot function, since you never get to see the notebooklet class directly. Having everything accessible from the result class (which is returned from the pivot function) means that you can use it as a proxy for the notebooklet class instance attributes..

What's Changed

• Updating iPython requirement to avoid MSTICPy clash by @petebryan in #24
• Ianhelle/notebooklet fixes 2022 04 12 by @ianhelle in #26
• Updating docs for notebooklets by @ianhelle in #27
• URL Summary by @petebryan in #29
• Added HostNetworkSummary documentation by @petebryan in #30
• 1.0.0 release by @petebryan in #31

Full Changelog: v0.2.3...v1.0.0

Minor fixes to host and account notebooklets

Fixes

b5b3598@Updating version to 0.2.3
5ce81a0@Reverting change to calling SelectAlert since it fails on MSTICPy 1.4.5 and earlier
d06e23d@Fixing error caused by msticpy bug in ti_enrich
4bdf7b6@Fixing test breaks in ti_enrich.py and account_summary.pyAdding additional McCabe suppressions to deal with diff versions (sometimes McCabe IDs the start of decorated function as the decorator line, in newer versions, it uses the def line)

34af0da@Addressing McCabe and Prospector warnings
bdae992@Fix failing test in test_metadata.py
79c8ae8@Linting errors
e10a429@Updating azure-pipelines to python 3.8type hints in data_providers

a124c8f@Removing azure_data requirement from host_summary notebooklet for test
bb908cd@Merge pull request #20 from microsoft/pebryan/2021-7-2MinorFixesMinor fixes to account and ip nblts
9b2b549@Minor fixes to account and ip nblts
cc391c9@Updating version

Fixed bugs in ipsummary, accountsummary, hostsummary

Fixes

1c3c72b@Bug fixes for ipsummary, account summary and host summary

aea7603@

• Fix when only one account matched in account_summary.py
• More defensive dictionary access for Azure Data in host_summary.py
• Fixed using IP address rather than string in iptools.py
• Skipping some tests when running in Linux/Mac CI

Account Summary, IP Summary, Logon Session rarity notebooklets

The second release of Notebooklets has been a long time coming but finally here.
It includes 3 new notebooklets:

• Account summary - explore an account (Azure/Office, Windows or Linux)
  Logon activity, Azure office activity, alerts, etc.
• IP Address Summary - explore IP address:
  Threat intel, geolocation, whois plus checks for presence of IP in multiple Azure Sentinel logs
• Logon session rarity
  Using clustering of processes to estimate the relative unusualness of individial logon sessions.
  Browse the sessions with unusual activity using event timelines or process trees.

Also support for MSTICPy pivot functions - loading Notebooklets package will add
notebooklet run functions as pivots to the appropriate entity (e.g. Host, Account, IP)

Updates

86c0865@Automated ReadtheDocs documentation for notebooklets
e3bc125@Logon session rarity notebooklet.
58c8e60@Adding print_options function to notebooklet.py
49e05a6@

• Add data_viewers.py module for simple event browsing
• Added Pivot initialization to the package init.py so that notebooklets are added as pivot functions
• Created local version of convert_to_ip_entities that accepts geoip provider in args. This is used by ti_enrich, host.py, host_logons_summary.py and, indirectly, by ip_summary and network_flow_summary.
• Added map_ips function to ip_tools - generic Folium map for list of IPs
• Change notebooklet_result.py so that it only displays first 5 rows of DF and has explanatory text why not everything is showing.
  3d619cb@
• Added some utility functions to common.py and notebooklet.py
  • check_valid_result_data
  • check_table_exists
  • get_methods/list_methods (lists only methods defined on subclasses, not Notebooklet class)
• Split NotebooketResult into separate module notebooklet_result.py
• Added ability to invoke notebooklet functions from results class
• Added alert.py alert browser

Fixes

27db47e@Changed requirements for msticpy to be >=1.0.0
631a57d@Fixing docstring in ip_summary
d330b22@

• Better formatting of options in help- added options doc string to notebooklet init.
• Refactored large init function in notebooklet
  419cce1@Fixing tests that depend on GeoLiteLookup - replace with mock class.Temporary workaround for convert_to_ip_entities in host.py
  1d2cf20@Update azure-pipelines.yml for Azure PipelinesAdd maxmind auth key
  256f6ec@Fixing setup.py to read from requirements.txt
  e2e48e6@Update azure-pipelines.yml for Azure PipelinesAdd install of pytest-check
  0a88c16@Some test and linter fixes
  3d619cb@
• Fixes/regularization to host.host.py and iptools.py. Added VPS lookup
• Added several test data sets such as azure_activity_df, az_net_df (interface), vmcomputer_df, host_hb_df
• Added mock classes for TILookup and GeoIP for testing
• Switched several test modules to native pytest format.
  184a2af@
• Update to add extra method to notebooklet base class for wrapped run method
• Some fixes due to pandas TZ-specific changes and some of the test data.
• Removed TimeSpan from common.py - now imported from msticpy.
  029753e@Add pivot support for notebooklets run method.
  ee1125e@
• Update to add extra method to notebooklet base class for wrapped run method
• Some fixes due to pandas TZ-specific changes and some of the test data.
• Removed TimeStamp from common.py - now imported from msticpy.
  159d63f@Create CONTRIBUTING.md
  617ce20@Changing image sizes in readme. Spelling corrections
  99a3441@Documentation addition and update to README.md
  906da10@Update README.mdAdded link to the notebook and introductory text.

Initial release

Features

• Notebooklet infrastructure:
  • Dataproviders (automating load of msticpy providers)
  • Notebooklet and NotebookletResult base classes
  • Notebooklet importer - handling classes and yaml metadata
  • Classdoc - self-documentation of notebooklets
• Notebooklets:
  The initial set of notebooklets are specific to Azure Sentinel
  • HostSummary (Linux and Windows) - basic details about a host from Azure Sentinel and Azure APIs
  • HostLogonsSummary (Linux and Windows) - analysis of logons to the host
  • WinHostEvents (Windows) analysis of security events on a Windows host (esp Account management events)
  • NetworkFlowSummary - analysis of network traffic for a specific host/IP address
  • Alert enrichment - additional enrichment (e.g. ThreatIntel) for alert triage
  • AccountSummary (Windows, Linux, Azure AD, Office) - analysis of logon activity for an account.