feat: IP Flow Information eXport (IPFIX) plugin #1649

rewenset · 2020-04-15T07:42:15Z

IPFIX plugin: support of VPP IPFIX and Flowprobe plugin.

Add binapi for VPP versions 2001, 1908 and 1904.
Create plugin.
Define proto messages.
Generate adapters. Create descriptors.
Add vppcalls.
Add IPFIX plugin to Agent.
Add integration tests.

Problems:

with IPFIX:
- [EDIT: Solved with creating ticket in VPP Jira] VPP uses two types of IPFIX messages: template message and data message. Template is a message which specifies the structure for data records in data message. To configure export in VPP, user can set among other parameters also a collector port. But when port is set other than default, data records will still go to the default 4739 port, and only templates will go to specified one. This is probably a bug in VPP. I'm only 99% sure about this, because I'm using my own very basic IPFIX protocol parser.
with Flowprobe:
- [EDIT: Solved with creating ticket in VPP Jira] There is no ability to get info on which interfaces Flowprobe feature is currently enabled. And also, there is no ability to dump Flowprobe params.
- [EDIT: Solved with using metadata] It is forbidden to enable Flowprobe feature before setting Flowprobe params, but if Flowprobe params were configured and at least one interface have Flowprobe feature enabled, then sending either new or the same Flowprobe params again will result in error returned from API:
```
VPPApiError: Unspecified Error (-1)
```

- Add binapi for VPP versions 2001, 1908 and 1904. - Create plugin. - Define proto messages. - Generate adapters. Create descriptors. - Add vppcalls. - Add IPFIX plugin to Agent. - Add integration tests.

codecov · 2020-04-15T10:40:21Z

Codecov Report

Merging #1649 into master will increase coverage by 0.47%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1649      +/-   ##
==========================================
+ Coverage   57.64%   58.11%   +0.47%     
==========================================
  Files         495      293     -202     
  Lines       40639    23850   -16789     
==========================================
- Hits        23425    13861    -9564     
+ Misses      14755     8858    -5897     
+ Partials     2459     1131    -1328

Flag	Coverage Δ
#e2e	`?`
#unittests	`58.11% <ø> (+<0.01%)`	⬆️

rastislavs · 2020-04-17T09:53:37Z

LGTM. For missing flowprobe dumps, you may open an issue in fd.io Jira as well.

…hods in FP API. FP - Flowprobe :)

golangcibot · 2020-04-17T18:06:33Z

plugins/vpp/ipfixplugin/descriptor/flowprobe_params.go

+// IsRetriableFailure returns false if error is one of errors
+// defined at the top of this file as non-retriable.
+func (d *FPParamsDescriptor) IsRetriableFailure(err error) bool {
+	if errors.Is(err, ErrFeatureEnabled) {


S1008: should use 'return ' instead of 'if { return }; return ' (from gosimple)

rewenset · 2020-04-17T18:08:55Z

How did I dismiss the review? Is it because of that new commit I've added? Anyway, somehow I didn't have time to create issue about missing dump API. Will do it later. Hope not on weekends :)

rewenset · 2020-04-22T10:56:27Z

"Restart Agent without restarting VPP" test

When Agent was running I've created two TAP interfaces, configured IPFIX and Flowprobe params and enabled Flowprobe feature on one of the interfaces.

Show FullResync transaction

+======================================================================================================================+
| #0 - NBTransaction                                                                                        FullResync |
+======================================================================================================================+
  * transaction arguments:
      - seqNum: 0
      - type: NB Transaction, Full Resync
      - values:
          - key: config/linux/interfaces/v2/interface/linuxtap1
            val: { name:"linuxtap1" type:TAP_TO_VPP host_if_name:"host-tap1" enabled:true ip_addresses:"10.10.10.2/24" tap:<vpp_tap_if_name:"vpptap1" >  }
          - key: config/linux/interfaces/v2/interface/linuxtap2
            val: { name:"linuxtap2" type:TAP_TO_VPP host_if_name:"host-tap2" enabled:true ip_addresses:"20.20.20.2/24" tap:<vpp_tap_if_name:"vpptap2" >  }
          - key: config/vpp/ipfix/v2/flowprobe-feature/vpptap2
            val: { interface:"vpptap2" l2:true  }
          - key: config/vpp/ipfix/v2/flowprobe-params
            val: { active_timer:2 passive_timer:10 record_l2:true record_l3:true record_l4:true  }
          - key: config/vpp/ipfix/v2/ipfix
            val: { collector:<address:"10.10.10.2" port:9374 > source_address:"10.10.10.1"  }
          - key: config/vpp/v2/interfaces/vpptap1
            val: { name:"vpptap1" type:TAP enabled:true ip_addresses:"10.10.10.1/24" tap:<version:2 host_if_name:"host-tap1" >  }
          - key: config/vpp/v2/interfaces/vpptap2
            val: { name:"vpptap2" type:TAP enabled:true ip_addresses:"20.20.20.1/24" tap:<version:2 host_if_name:"host-tap2" >  }

o----------------------------------------------------------------------------------------------------------------------o
  * executed operations (2020-04-22 10:54:45.262 +0000 UTC -> 2020-04-22 10:54:45.292 +0000 UTC, dur: 31ms):
      1. UPDATE [NOOP WAS-OBTAINED]:
          - key: config/vpp/ipfix/v2/ipfix
          - prev-value: { collector:<address:"10.10.10.2" port:9374 > source_address:"10.10.10.1" path_mtu:512 template_interval:20  }
          - new-value: { collector:<address:"10.10.10.2" port:9374 > source_address:"10.10.10.1"  }
x----------------------------------------------------------------------------------------------------------------------x
| #0 - NBTransaction                                                                                       took 30.8ms |
x----------------------------------------------------------------------------------------------------------------------x

ondrej-fabry · 2020-05-09T15:33:26Z

@milanlenco @rewenset is this ready for merge?

rewenset · 2020-05-09T15:56:03Z

@ondrej-fabry

@milanlenco @rewenset is this ready for merge?

I think so. But would be nice to get a review from @milanlenco too 😇

plugins/vpp/ipfixplugin/descriptor/flowprobe_feature.go

…hod.

sonarqubecloud · 2020-05-11T09:02:55Z

SonarCloud Quality Gate failed.

0 Bugs
14 Vulnerabilities (and 0 Security Hotspots to review)
31 Code Smells

No Coverage information
39.3% Duplication

IPFIX plugin: support of VPP IPFIX and Flowprobe plugin.

3543523

- Add binapi for VPP versions 2001, 1908 and 1904. - Create plugin. - Define proto messages. - Generate adapters. Create descriptors. - Add vppcalls. - Add IPFIX plugin to Agent. - Add integration tests.

rewenset added ⚡️ feature 🚧 WIP do not merge! work in progress! labels Apr 15, 2020

rewenset requested review from rastislavs, milanlenco and ondrej-fabry April 15, 2020 07:42

rastislavs previously approved these changes Apr 17, 2020

View reviewed changes

Connect FP Params and FP Feautre. Add workaround for missing dump met…

3e880f5

…hods in FP API. FP - Flowprobe :)

rewenset dismissed rastislavs’s stale review via 3e880f5 April 17, 2020 18:02

golangcibot reviewed Apr 17, 2020

View reviewed changes

rewenset added 2 commits April 22, 2020 10:58

Add IPFIX and Flowprobe to clientv2

519b513

Add custom equivalent check for IPFIX configurations

edaa01a

rewenset changed the title ~~feat: IP Flow Information eXport (IPFIX) plugin [WIP]~~ feat: IP Flow Information eXport (IPFIX) plugin Apr 22, 2020

rewenset removed the 🚧 WIP do not merge! work in progress! label Apr 22, 2020

milanlenco suggested changes May 11, 2020

View reviewed changes

plugins/vpp/ipfixplugin/descriptor/flowprobe_feature.go Show resolved Hide resolved

plugins/vpp/ipfixplugin/descriptor/flowprobe_feature.go Outdated Show resolved Hide resolved

Add dependency on interface for FP feature. Fix changing input in met…

6b1d418

…hod.

milanlenco approved these changes May 11, 2020

View reviewed changes

ondrej-fabry merged commit d9b6dd4 into ligato:master May 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: IP Flow Information eXport (IPFIX) plugin #1649

feat: IP Flow Information eXport (IPFIX) plugin #1649

rewenset commented Apr 15, 2020 •

edited

Loading

codecov bot commented Apr 15, 2020 •

edited

Loading

rastislavs commented Apr 17, 2020

golangcibot Apr 17, 2020

rewenset commented Apr 17, 2020 •

edited

Loading

rewenset commented Apr 22, 2020

ondrej-fabry commented May 9, 2020

rewenset commented May 9, 2020

sonarqubecloud bot commented May 11, 2020

feat: IP Flow Information eXport (IPFIX) plugin #1649

feat: IP Flow Information eXport (IPFIX) plugin #1649

Conversation

rewenset commented Apr 15, 2020 • edited Loading

codecov bot commented Apr 15, 2020 • edited Loading

Codecov Report

rastislavs commented Apr 17, 2020

golangcibot Apr 17, 2020

Choose a reason for hiding this comment

rewenset commented Apr 17, 2020 • edited Loading

rewenset commented Apr 22, 2020

"Restart Agent without restarting VPP" test

ondrej-fabry commented May 9, 2020

rewenset commented May 9, 2020

sonarqubecloud bot commented May 11, 2020

rewenset commented Apr 15, 2020 •

edited

Loading

codecov bot commented Apr 15, 2020 •

edited

Loading

rewenset commented Apr 17, 2020 •

edited

Loading