Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Fix bugs in Netflow input #13821

Merged
merged 5 commits into from
Sep 30, 2019

Conversation

adriansr
Copy link
Contributor

@adriansr adriansr commented Sep 27, 2019

This PR fixes a couple bugs in the Netflow input causing flow loss in Netflow V9 and IPFIX:

  • Bad expiration of templates caused all templates to be removed after each expiration_timeout elapsed.
  • Incorrect handling of sequence numbers caused all templates to be removed if a device exported packets from different Observation Domains.

It's easier to review by looking at the individual commits.

Closes #13699

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.
The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.
@adriansr adriansr added bug Filebeat Filebeat Team:SIEM x-pack Issues and pull requests for X-Pack features. labels Sep 27, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem

@adriansr adriansr added the needs_backport PR is waiting to be backported to other branches. label Sep 27, 2019
@adriansr adriansr requested a review from a team September 27, 2019 15:49
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lots of good fixes here. This should also close #13699 IIUC.

@adriansr adriansr merged commit 62d0e87 into elastic:master Sep 30, 2019
adriansr added a commit to adriansr/beats that referenced this pull request Sep 30, 2019
This commit fixes a couple bugs in the Netflow input causing flow loss
in Netflow V9 and IPFIX:

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.

The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.

(cherry picked from commit 62d0e87)
@adriansr adriansr added v7.4.0 and removed needs_backport PR is waiting to be backported to other branches. labels Sep 30, 2019
adriansr added a commit that referenced this pull request Sep 30, 2019
This commit fixes a couple bugs in the Netflow input causing flow loss
in Netflow V9 and IPFIX:

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.

The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.

(cherry picked from commit 62d0e87)
adriansr added a commit to adriansr/beats that referenced this pull request Oct 14, 2019
This commit fixes a couple bugs in the Netflow input causing flow loss
in Netflow V9 and IPFIX:

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.

The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.

(cherry picked from commit 62d0e87)
adriansr added a commit that referenced this pull request Oct 14, 2019
This commit fixes a couple bugs in the Netflow input causing flow loss
in Netflow V9 and IPFIX:

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.

The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.

(cherry picked from commit 62d0e87)
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
elastic#13846)

This commit fixes a couple bugs in the Netflow input causing flow loss
in Netflow V9 and IPFIX:

Due to an error the expiration loop was expiring the sessions twice in
each iteration. This caused all the sessions to be expired wether active
or not, every time that the expiration interval elapsed.

The NetFlow input was mistakenly comparing sequence numbers (v9 and
IPFIX) from the same source but different Observation Domain (aka Source
ID). This caused the session to be reset when packets from different
source IDs where interweaved.

This refactors the source ID out of session and into the session key, so
that now a different source ID from the same source creates a new
session.

(cherry picked from commit 2da3e27)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Filebeat Filebeat v6.8.4 v7.4.0 x-pack Issues and pull requests for X-Pack features.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Make it possible to enable logging from netflow decoders
4 participants