Skip to content

VAST 2020.10.29
Compare
Choose a tag to compare
@tobim tobim released this 29 Oct 09:02
2020.10.29
This tag was signed with the committer’s verified signature.
tobim
GPG key ID: F8657E90819A1298
Learn about vigilant mode.
b1f7367
This commit was signed with the committer’s verified signature.
tobim
GPG key ID: F8657E90819A1298
Learn about vigilant mode.

We're happy to announce the monthly release 2020.10.29 of VAST.

Taxonomies

This release includes an exciting experimental feature to deliver a scalable user experience as the number of different data formats in VAST keep increasing: taxonomies. With this feature, you can now define your own unified access layer to consolidate syntactic differences of the various data formats at play. By using a taxonomy, you establish a semantic frame over the domain you analyze. Thereafter you can write queries "in your own bubble" without having to juggle the various naming schemes of each individual data source.

Consider the scenario of having two types of network security logs: Sysmon from the endpoint and Suricata on the network side. When you want to query all flow events to a particular destination in both formats, you would combine two predicates as follows:

suricata.flow.src_ip == 6.6.6.6 || sysmon.NetworkConnection.SourceIp == 6.6.6.6

But what you actually want to write is:

source_ip == 6.6.6.6

Thanks to taxonomies, this is now possible. While the expert knows the semantics of the format-specific field names, memorizing the mapping from the meaning of a field to its name is not something we want to burden the user with. It’s inefficient, error-prone, and does not scale with dozens of different data formats.

We developed two building blocks: this release introduces concepts and models will follow next. Together, they enable abstraction and composition of data semantics. Please consult our documentation of taxonomies to learn more.

Experimental Feature: Age Rotation of Old Data

As more and more data is ingested into VAST, more and more disk space will be required to store this data. As a telemetry database, typically new data is imported into VAST continuously leading to a linear increase in disk space usage over time.

To help control this tide of data, we introduced a new experimental age rotation feature to VAST: Operators can now specify a disk budget, and when the size of the database exceeds the budget old data will be deleted.

With this, operators are able to decide on the desired retention period for their data and allocate the appropriate amount of disk space once without having to permanently clean up their disks.

The main user interface for the new age rotation are the vast.start.disk-budget-high and -low config options, which can also be specified as command-line flags to vast start. These define a corridor for the amount of disk space to be used for the database directory.

Increased Source Responsiveness

VAST ingest path has to process up to hundreds of thousands log lines per second. To cope with this volume, we use both batching and backpressure. However, the sheer number of messages that VAST has to process could result in overload, which manifested simply as an unresponsive component. For example, when an input source under high load needs to reply to a vast status request, the source often fails to come back with a reply in the given timeout (10s by default). Interestingly, the other extreme could also cause a timeout: when a source did not receive enough events, it was unable to yield back to the scheduler and thereby 'got stuck" on the inbound path, unable to handle any other form of interaction.

We fixed the issue by rewriting the control logic that handles CAF actor streams. The input sources can now handle large message volumes just fine while also remaining responsive and resource-efficient when the data path idles. To control the new behavior, the new option import.read-timeout can set an input timeout for low-volume sources. Reaching the timeout forwards the current batch immediately. Previously, the option import.batch-timeout controlled this behavior, which now only controls the maximum buffer time before the source forwards batches to the server.

Changelog Highlights

As always, you can find the full technical scoop of what changed in our changelog.

🧬 Experimental Features

  • The query language now comes with support for concepts, the first part of taxonomies. Concepts is a mechanism to unify the various naming schemes of different data formats into a single, coherent nomenclature. #1102
  • A new disk monitor component can now monitor the database size and delete data that exceeds a specified threshold. Once VAST reaches the maximum amount of disk space, the disk monitor deletes the oldest data. The command-line options --disk-quota-high, --disk-quota-low, and --disk-quota-check-interval control the rotation behavior. #1103

🎁 Features

  • The new options vast.segments and vast.max-segment-size control how the archive generates segments. #1103
  • When running VAST under systemd supervision, it is now possible to use the Type=notify directive in the unit file to let VAST notify the service manager when it becomes ready. #1091
  • The new script splunk-to-vast converts a splunk CIM model file in JSON to a VAST taxonomy. For example, splunk-to-vast < Network_Traffic.json renders the concept definitions for the Network Traffic datamodel. The generated taxonomy does not include field definitions, which users should add separately according to their data formats. #1121

⚠️ Changes

  • The new option import.read-timeout allows for setting an input timeout for low volume sources. Reaching the timeout causes the current batch to be forwarded immediately. This behavior was previously controlled by import.batch-timeout, which now only controls the maximum buffer time before the source forwards batches to the server. #1096
  • VAST will now warn if a client command connects to a server that runs on a different version of the vast binary #1098
  • The default database directory moved to /var/lib/vast for Linux deployments. #1116
  • Log files are now less verbose because class and function names are not printed on every line. #1107

🐞 Bug Fixes

  • The vast status --detailed command now correctly shows the status of all sources, i.e., vast import or vast spawn source commands. #1109
  • Sources that receive no or very little input do not block vast status any longer. #1096
  • VAST no longer opens a random public port, which used to be enabled in the experimental VAST cluster mode in order to transparently establish a full mesh. #1110
  • The lookup for schema directories now happens in a fixed order. #1086
  • The lsvast tool failed to print FlatBuffers schemas correctly. The output now renders correctly. #1123
Assets 6
Loading