cel: new generic integration #5539

efd6 · 2023-03-15T02:34:19Z

What does this PR do?

This makes the Filebeat CEL input available as an integration package.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Fix regexp docs - ref
Resolve mechanism to get program source though kibana/agent - ref

How to test this PR locally

Related issues

Depends x-pack/filebeat/input/cel: improve user docs beats#34831

Screenshots

elasticmachine · 2023-03-15T02:37:48Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-04-02T21:51:13.995+0000
Duration: 15 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	4
Skipped	0
Total	4

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

packages/cel/data_stream/generic/manifest.yml

elasticmachine · 2023-03-15T03:23:28Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`0/0`)	💚
Files	100.0% (`0/0`)	💚
Classes	100.0% (`0/0`)	💚
Methods	100.0% (`3/3`)	💚
Lines	100.0% (`0/0`)	💚
Conditionals	100.0% (`0/0`)	💚

elasticmachine · 2023-03-15T03:49:11Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

This makes the Filebeat CEL input available as an integration package.

packages/cel/data_stream/generic/agent/stream/cel.yml.hbs

andrewkroh · 2023-03-20T20:36:51Z

packages/cel/data_stream/generic/fields/base-fields.yml

+- name: event.dataset
+  type: constant_keyword
+  description: Event dataset
+  value: cel.generic


With custom inputs I think having a static event.dataset is problematic because users can customize the data stream.dataset value. When they choose a value different than cel.generic then data cannot be indexed due to the document not matching what is found in the mapping. So I think if you remove the value so that it's like the data_stream.dataset on line 4 it will be good.

All of our issues for event.dataset and data_stream.dataset (there are several other issues here as well like component templates) are all resolved if we move the package to be of type input, as it has custom UI elements specifically for data_stream. Maybe we should look at that?

packages/cel/data_stream/generic/manifest.yml

P1llus · 2023-03-23T12:19:27Z

Just in case, please do not merge it yet, I am trying to find some time this week at least to do a full review, since this is our first input type package, we should ensure it is in top shape, so we can use this as a template for future work 👍

efd6 · 2023-03-23T20:39:52Z

No worries, merging this is blocked on a resolution to #5539 (comment) anyway.

Option naming reflect that used in HTTPJSON (request instead of resource) for clarity and because we are tracing only requests, not all resource interactions.

andrewkroh

LGTM.

efd6 · 2023-04-14T03:48:35Z

This needs to be updated to make use of trace input ID substitution implemented as exists in HTTPJSON.

efd6 · 2023-04-14T03:53:05Z

Actually, the ID replacement functionality is not there yet. What would be your preference, merge this as is and change them both together, or change HTTPJSON to use the id-sub functionality and do the change for CEL here?

andrewkroh · 2023-04-14T04:38:39Z

IMO it can be merged without the ID replacement. We can push that update later and users shouldn't notice (except there will be a higher stack version requirement).

elasticmachine · 2023-04-17T13:21:21Z

Package cel - 0.1.0 containing this change is available at https://epr.elastic.co/search?package=cel

efd6 added enhancement New feature or request Team:Security-External Integrations labels Mar 15, 2023

efd6 self-assigned this Mar 15, 2023

efd6 force-pushed the cel branch from 06a6293 to 915e6b8 Compare March 15, 2023 02:36

efd6 commented Mar 15, 2023

View reviewed changes

packages/cel/data_stream/generic/manifest.yml Outdated Show resolved Hide resolved

packages/cel/data_stream/generic/manifest.yml Outdated Show resolved Hide resolved

packages/cel/data_stream/generic/manifest.yml Outdated Show resolved Hide resolved

efd6 force-pushed the cel branch from 915e6b8 to 3656bfa Compare March 15, 2023 03:07

efd6 changed the title ~~cel: new input~~ cel: new generic integration Mar 15, 2023

efd6 force-pushed the cel branch from 3656bfa to 4f52a12 Compare March 15, 2023 03:29

efd6 requested a review from a team March 15, 2023 03:49

efd6 marked this pull request as ready for review March 15, 2023 03:49

efd6 force-pushed the cel branch from 4f52a12 to 468b93f Compare March 15, 2023 03:50

cel: new generic integration

d168462

This makes the Filebeat CEL input available as an integration package.

efd6 force-pushed the cel branch from 468b93f to d168462 Compare March 16, 2023 03:24

andrewkroh reviewed Mar 20, 2023

View reviewed changes

efd6 mentioned this pull request Mar 20, 2023

x-pack/filebeat/input/cel: fix doc entry title indentation elastic/beats#34861

Merged

6 tasks

efd6 added 2 commits March 21, 2023 08:09

address pr comments

2cf76d2

convert to input package

3c789cf

efd6 added 2 commits March 24, 2023 08:49

use textarea input and escape_string for program input

b7e60da

add option for request tracing

58bc8c8

Option naming reflect that used in HTTPJSON (request instead of resource) for clarity and because we are tracing only requests, not all resource interactions.

andrewkroh approved these changes Apr 11, 2023

View reviewed changes

efd6 merged commit a7d9678 into elastic:main Apr 14, 2023

andrewkroh added Integration:cel Custom API using Common Expression Language New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cel: new generic integration #5539

cel: new generic integration #5539

efd6 commented Mar 15, 2023 •

edited by andrewkroh

Loading

elasticmachine commented Mar 15, 2023 •

edited

Loading

Build stats

Test stats 🧪

elasticmachine commented Mar 15, 2023 •

edited

Loading

elasticmachine commented Mar 15, 2023

andrewkroh Mar 20, 2023

P1llus Mar 22, 2023

This comment was marked as outdated.

P1llus commented Mar 23, 2023

efd6 commented Mar 23, 2023

andrewkroh left a comment

efd6 commented Apr 14, 2023

efd6 commented Apr 14, 2023

andrewkroh commented Apr 14, 2023

elasticmachine commented Apr 17, 2023

cel: new generic integration #5539

cel: new generic integration #5539

Conversation

efd6 commented Mar 15, 2023 • edited by andrewkroh Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Mar 15, 2023 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

elasticmachine commented Mar 15, 2023 • edited Loading

🌐 Coverage report

elasticmachine commented Mar 15, 2023

andrewkroh Mar 20, 2023

Choose a reason for hiding this comment

P1llus Mar 22, 2023

Choose a reason for hiding this comment

This comment was marked as outdated.

P1llus commented Mar 23, 2023

efd6 commented Mar 23, 2023

andrewkroh left a comment

Choose a reason for hiding this comment

efd6 commented Apr 14, 2023

efd6 commented Apr 14, 2023

andrewkroh commented Apr 14, 2023

elasticmachine commented Apr 17, 2023

efd6 commented Mar 15, 2023 •

edited by andrewkroh

Loading

elasticmachine commented Mar 15, 2023 •

edited

Loading

elasticmachine commented Mar 15, 2023 •

edited

Loading