Skip to content

VAST 2021.03.25
Compare
Choose a tag to compare
@dominiklohmann dominiklohmann released this 25 Mar 13:00
2021.03.25
b8ed41c
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We’re happy to announce our release 2021.03.25 of VAST. Modifying types is now easier than ever, and the plugin framework is able to build plugins into VAST statically.

Type Algebra

VAST’s schema language now supports operations on record types to make it easier to adapt to the dynamic natures of events. In particular, it is now possible to add and remove fields from records. The documentation on Type Algebra explains this feature in detail.

One of our customers had an interesting use case for this feature: they are using VAST to store and analyze Suricata logs, but they enrich their Eve JSON with a custom field. Before the type algebra feature, they modified the bundled Suricata schema in-place, which they had to manually maintain with every update to VAST they rolled out. Now they can simply add a "delta" file to their deployment:

type suricata.customizations = record {
  _internal_id: count,
  _internal_name: string,
}

// Enhance Suricata types with the fields of the “customizations” type
type suricata.alert = suricata.alert + suricata.customizations
type suricata.anomaly = suricata.anomaly + suricata.customizations
// repeat for other Suricata types

Plugin Framework Improvements

The plugin build scaffolding is becoming more powerful: It is now possible to bake plugins directly into VAST, which enables static binaries with zero dependencies with features from plugins. This is the first step towards a modularization of VAST into more easily maintainable pieces, which helps speed up development. We are now able to generate custom VAST builds with selected plugins enabled on-demand.

Workflow Change: Release Candidate Testing

For the last week before the release, we’ve experimented with a feature freeze period, in which we solely focused on fixing bugs and avoided merging features. We want to encourage users to test our release candidates and report back their findings. The feature freeze period turned out to be effective this month, resulting in the most merged bug fixes in a single week.

We’re announcing releases and release candidates in our Matrix chat and publish them directly to our GitHub repository. The releases come with a static binary for Linux attached that has zero external dependencies, which makes deploying as easy as downloading and unpacking the archive.

Acknowledgements

We want to thank our open-source community for numerous contributions to VAST. This month, we received the following external contributions:

  • Joe Loser (@JoeLoser): many PRs to modernize our code base and switch to the filesystem implementation of the C++ standard library, and a subtle bug fix to our PCAP import and export that led to already free’d resources being free’d again
  • Sascha Steinbiss (@satta): new Suricata schemas for MQTT and anomaly and testing
  • Andi Herz(@norg): extensive testing and invaluable operational feedback.

Changelog Highlights

As always, you can find the full scoop in our changelog.

⚡️ Breaking Changes

  • Plugins can now be linked statically against VAST. A new VASTRegisterPlugin CMake function enables easy setup of the build scaffolding required for plugins. Configure with --with-static-plugins or build a static binary to link all plugins built alongside VAST statically. All plugin build scaffoldings must be adapted, older plugins do no longer work. #1445 #1452
  • The previously deprecated #timestamp extractor has been removed from the query language entirely. Use :timestamp instead. #1399

⚠️ Changes

  • The type extractor in the expression language now works with type aliases. For example, given the type definition for port from the base schema type port = count, a search for :count will also consider fields of type port. #1446
  • Query latency for expressions that contain concept names has improved substantially. For DB sizes in the TB region, and with a large variety of event types, queries with a high selectivity experience speedups of up to 5x. #1433
  • The option vast.no-default-schema is deprecated, as it is no longer needed to override types from bundled schemas. #1409
  • VAST now ships with schema record types for Suricata's mqtt and anomaly event types. #1408 @satta

🎁 Features

  • VAST now supports nested records in Arrow table slices and in the JSON import, e.g., data of type list<record<name: string, age: count>. While nested record fields are not yet queryable, ingesting such data will no longer cause VAST to crash. MessagePack table slices don't support records in lists yet. #1429
  • The schema language now supports 4 operations on record types: + combines the fields of 2 records into a new record. <+ and +> are variations of + that give precedence to the left and right operand respectively. - creates a record with the field specified as its right operand removed. #1407 #1487 #1490

🐞 Bug Fixes

  • A race condition during server shutdown could lead to an invariant violation, resulting in a firing assertion. Streamlining the shutdown logic resolved the issue. #1473 #1485
  • A query for a field or field name suffix that matches multiple fields of different types would erroneously return no results. #1447
  • The disk monitor now correctly erases partition synopses from the meta index. #1450
  • The archive, index, source, and sink components now report metrics when idle instead of omitting them entirely. This allows for distinguishing between idle and not running components from the metrics. #1451
  • The JSON parser now accepts data with numerical or boolean values in fields that expect strings according to the schema. VAST converts these values into string representations. #1439
  • Data that was ingested before the deprecation of the #timestamp attribute wasn't exported correctly with newer versions. This is now corrected. #1432
Assets 7
Loading