Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ready for Review] Add Morphues dfp #5

Merged
merged 64 commits into from
Nov 18, 2022

Conversation

AjayThorve
Copy link
Contributor

@AjayThorve AjayThorve commented Nov 4, 2022

This PR aims to create a directory structure for the various morhpheus visualization applications

cc @efajardo-nv @exactlyallan

Closes #3
Closes #4

@AjayThorve AjayThorve changed the title [WIP] Add Morphues dfp [Ready for Review] Add Morphues dfp Nov 8, 2022
@AjayThorve
Copy link
Contributor Author

@efajardo-nv @exactlyallan this is pretty much ready now. Feel free to try it out.

@AjayThorve
Copy link
Contributor Author

I'd say this is ready to be merged, barring improvements to the Readme file, which always has scope to be better :)

@exactlyallan
Copy link

Please add this section the the README under Configuration File:

## Usage Guide
The visualization app can be used as a 'real-time' dashboard to monitor for anomalous events, or as research tool for finding patterns in historical anomalous behaviors.

The visualization app has four main components:

- **Area chart** of total network events vs anomalous events
- **Hexagon grid** of events grouped by user
- **Event details** menu on the right
- **Settings menu** on the left

There is also a bottom status bar with information on last action, update time, and more.

Note: **hovering over** the circle **i** icons will give you information about the given view or setting.


### Area Chart

The chart represents the full view of the loaded dataset. The white area represents **all** network traffic by event count. The orange represents anomalous events, as set by the lower bound of *anomalous color threshold* in the settings menu.


### Hexagon Grid

The hexagon grid represents users, with anomalous events binned, colored, and sorted by row and time. The grid view can be reformatted in the settings menu. The **most recent** time stamp starts on the left. Note, the hexagon grid and area chart axis are not synced. You can navigate the hex grid by **left click and dragging** the view. You can **zoom in and out** with the mouse wheel. To reset the view **double click** anywhere on the grid.

The **color of the hexagon** is based on the **maximum anomalous event** if there are multiple events binned in the hexagon. Clicking on a a colored hexagon opens the details menu. If the menu is open, click anywhere or the menu's X to close. A selected hexagon will have a black border. Hexagons colored **gray** signify no recorded events.

Using **SHIFT + left click and dragging** will tilt into a **3D view** of the hexagons, with the height representing the number of events. To reset the view back into 2D, **double click** anywhere on the grid. Note, it is possible to **right click and drag** to tilt into a 3D view, but is not recommended.


### Event Details

Clicking on a colored hexagon will open the event details menu. Clicking the X or anywhere on the grid will close it. **Selected Events** is a dropdown of all the anomalous events binned within the hexagon, ordered from most anomalous to least. Note, the number of events can vary depending on *Time Bin Per Hexagon* in the settings menu.

The **Anomalous Scale** is the hexagon color legend for **scaled** anomaly scores between 0 and 1. This does not re-scale the score, just the range and threshold of the color pallette. The scale will effect what the area chart counts as an anomalous event.

The **Attributes** list consists of the **overall scaled anomaly score**, which the hexagon colors are based, and its unscaled raw score. Below, are the individual **contributing attributes** scaled score and raw score. The scaled **mean score** is calculated based on the values of the loaded dataset. A larger difference between the mean and attribute anomaly scores will be an indicator contributing to its overall anomaly score.


### Settings Menu
These are settings that alter the behavior and views of the app. Note, settings above the break line must be **applied** by clicking the **apply button**. Those below the break line will update in real time. 

The settings menu can be opened by clicking the **menu icon** in the upper left side of the app. Clicking the X will close it.

The **Current Dataset** drop down menu shows the currently selected dataset located in the data in the data directory specified by the `.env` file. Clicking the *reload icon* will update the drop down if new files have been added.

The **Sort By..** drop down menu shows the different ways to order the users, based on the anomalous events.

The **Anomalous Color Threshold** range slider sets at what anomaly score the hexagons are colored, and the range between what is considered slightly anomalous to critical.

The **Visible Users (Rows)** slider sets how many user ID rows to show from top to bottom, based on the currently selected sort option. The max is automatically set based on the number of user IDs in the dataset to the limit set in the `.env` file. If the hexagon interaction performance is slow, decreasing the visible users can help.

The **Time Bin Per Hexagon** slider sets how many seconds of time each hexagon represents. A larger value will show less columns of hexagons and have more events per hexagon, while a smaller value will show more hexagons. If the hexagon interaction performance is slow, increasing the time bin can help.

The **Look Back Time** represents the amount of time shown in seconds, starting from the most recent time in the dataset. The maximum value is based on the full time of the dataset. Setting the look back time to anything less than the maximum will generate a status warning noting that the full dataset has not been visualized. If the hexagon interaction performance is slow, decreasing the look back time can help.

The **Update Frequency** represents the time in seconds between checking for a newer dataset file in the data directory. If a new file is present, the app will **automatically load that file** and visualize it. Note, the 'newness' of the dataset is based on the preconfigured **timestamp based name** of the file.

The **Live Updates** toggles if the app will **automatically update** the loaded dataset based on the update frequency setting. Note, if a user has selected an older dataset file and live updates is on, the **latest dataset will load** even if they are still interacting with the file. 

The **Hexagon Height** range slider scales the heights of the hexagons when in a **3D view**.

The **3d Perspective Lock** toggles if the user is able to rotate and pan while in a **3D view**.

@exactlyallan
Copy link

@AjayThorve found some issues:

README fixes:

  • Add a section about the expected data format (headers etc) in requirments
  • Add readme to root folder with single sentence description of each viz folder (currently empty)
  • In Readme screenshots, anomalous and network traffic legend colors are switched (ok in live app)
  • Rename Readme Screenshots to 'Settings Meu' and 'Event Details'

App fixes

  • (!)Clicking the refresh mini-icon in Current Dataset no longer refreshes folder?
  • Gray out the Settings Menu Apply button if the current selection has already been applied / is active (A non-grayed out button will show they have un-applied options selected)
  • Set MAX and MIN hexagon zoom-in levels so user wont get lost
  • Clicking on previously selected hexagon (menu closed) 'unselects' it
  • Status bar:" "Not all data is visible at the moment (toggle the following settings: Look Back Time, Time Bin Per Hexagon" <- add ) bracket

@AjayThorve
Copy link
Contributor Author

AjayThorve commented Nov 16, 2022

@AjayThorve found some issues:

README fixes:

  • Add a section about the expected data format (headers etc) in requirments
  • Add readme to root folder with single sentence description of each viz folder (currently empty)
  • In Readme screenshots, anomalous and network traffic legend colors are switched (ok in live app)
  • Rename Readme Screenshots to 'Settings Meu' and 'Event Details'

App fixes

  • (!)Clicking the refresh mini-icon in Current Dataset no longer refreshes folder?
  • Gray out the Settings Menu Apply button if the current selection has already been applied / is active (A non-grayed out button will show they have un-applied options selected)
  • Set MAX and MIN hexagon zoom-in levels so user wont get lost
  • Clicking on previously selected hexagon (menu closed) 'unselects' it
  • Status bar:" "Not all data is visible at the moment (toggle the following settings: Look Back Time, Time Bin Per Hexagon" <- add ) bracket

@exactlyallan, I fixed/implemented all of the above.

Copy link
Collaborator

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only have a couple of comments:

  1. Can we convert this to a VS Code multi-root workspace? Will help with organization as more visualizations are added over time. This way we can have global as well as local settings for each viz
  2. Why are the main folders prefixed with morpheus-? Seems a bit redundant to have morpheus-visualizations/morpheus-DFP and morpheus-visualizations/morpheus-GraphVis. Can we just make these DFP and GraphVis?

morpheus-DFP/.vscode/launch.json Outdated Show resolved Hide resolved
morpheus-DFP/.vscode/settings.json Outdated Show resolved Hide resolved
Copy link
Collaborator

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now. Approved.

@mdemoret-nv mdemoret-nv changed the base branch from branch-22.09 to branch-22.11 November 18, 2022 17:24
@mdemoret-nv mdemoret-nv added the enhancement Additional functionality added to an existing feature label Nov 18, 2022
@mdemoret-nv mdemoret-nv merged commit 27efc4f into nv-morpheus:branch-22.11 Nov 18, 2022
ghost pushed a commit to nv-morpheus/Morpheus that referenced this pull request Nov 18, 2022
Production DFP pipeline that uses new post-processing stage to generate input for DFP Visualization UI:
- `DFPVizPostprocStage` that selects columns from DFP output required by UI. Post-processing output is saved to multiple files where each corresponds to a specified time period (default is day).
- Python scripts for running viz pipeline for Azure and Duo.
- README with instructions for setting up environment and running DFP viz pipelines and UI for Azure and Duo.

Depends on nv-morpheus/morpheus-visualizations#5
Closes #321

Authors:
  - Eli Fajardo (https://github.com/efajardo-nv)

Approvers:
  - Michael Demoret (https://github.com/mdemoret-nv)

URL: #439
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Additional functionality added to an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] DFP Visualization [FEA] SID Viz Folder
3 participants