Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

gizas · 2025-01-30T14:56:44Z

Enhancement

Dont merge until I run once the tests as described here
We will need to put only 9.0.0 in constrains, trigger the buidkite pipeline and then revert again
Tested with command https://github.com/elastic/integrations/pull/12535#issuecomment-2627285200 : /test stack 9.0.0-SNAPSHOT

Proposed commit message

WHAT: Enabling support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations
WHY: Is needed in order to enable above integrations in version 9.0.0

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone Pr
elastic-package build with v0.109.1
elastic-package stack up -d -v --version=9.0.0-SNAPSHOT to install a local ES
Install an agent + Fleet with prementioned integrations enabled

Related issues

Relates #[Meta] Obs-ds-hosted-services Integration enablement for 9.0.0 #12529 (comment)

Logs

Below validation errors fixed for istio and containerd

Containerd

Error: building package failed: invalid content found in built zip package: found 7 validation errors:
   1. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/blkio/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   2. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/cpu/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   3. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/data_stream/memory/fields/fields.yml" is invalid: field 0: Additional property release is not allowed
   4. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field (root): Additional property license is not allowed
   5. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field (root): Additional property release is not allowed
   6. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field owner: type is required
   7. file "/Users/andreasgkizas/elastic/integrations/build/packages/containerd-0.5.0.zip/manifest.yml" is invalid: field conditions: Additional property kibana.version is not allowed


Istio:

Error: building package failed: invalid content found in built zip package: found 6 validation errors:
   1. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/data_stream/access_logs/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.47.remove.field: rename "message" to "event.original" processor requires remove "message" processor (JSE00001)
   2. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/data_stream/access_logs/elasticsearch/ingest_pipeline/default.yml" is invalid: field processors.47.remove.if: rename "message" to "event.original" processor requires remove "message" processor with if: 'ctx.event?.original != null' (JSE00001)
   3. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field (root): Additional property release is not allowed
   4. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field (root): Additional property license is not allowed
   5. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field owner: type is required
   6. file "/Users/andreasgkizas/elastic/integrations/build/packages/istio-0.7.0.zip/manifest.yml" is invalid: field conditions: Additional property kibana.version is not allowed

Screenshots

K8s

![k8s1](https://github.com/user-attachments/assets/1366461a-1fd6-43

6d-af4e-cfebfe13e0ff)

K8s Otel

Containerd

Istio

Ingress-Controller

Overall

Change property connection_string to be a secret like in the other integrations.

* Fix broken links * Update changelog and manifest

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

…0.108.0 (#12131) Bumps [github.com/elastic/elastic-package](https://github.com/elastic/elastic-package) from 0.107.2 to 0.108.0. - [Release notes](https://github.com/elastic/elastic-package/releases) - [Changelog](https://github.com/elastic/elastic-package/blob/main/.goreleaser.yml) - [Commits](elastic/elastic-package@v0.107.2...v0.108.0) --- updated-dependencies: - dependency-name: github.com/elastic/elastic-package dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Mario Rodriguez Molins <mario.rodriguez@elastic.co>

Changes added: - Add a limit parameter, that can be used to control the size of responses from TAXII servers (see https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107517) - To avoid fetching duplicate indicators every interval, now the response header X-Taxii-Date-Added-Last is stored in the cursor and used to populate the added_after parameter every iteration (see https://docs.oasis-open.org/cti/taxii/v2.1/os/taxii-v2.1-os.html#_Toc31107519)

* Update link * Update changelog and manifest

#11920) This is enabled per data stream to allow tuning of behaviour.

…nt" tag to documents with event.kind set to "pipeline_error" (#12108) This manually replays the changes in #12046.

…nd manually set to "pipeline_error" (#12109)

…at (#12110) * Include pipeline test examples to accommodate the new Cloudtrail format

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

- Correct the `dg_time` format (may have AM/PM). - Correct the `dg_processed_time` format (to `epoch_millis`). - Process additional timestamp fields: `dg_local_timestamp`, `pi_fal`, `pi_fcl`, `pi_fml`. - Add fallback/alternate formats for the string-valued time fields. - For data size fields, add parsing logic.

…ual_policies mode (#12115) * [Search Connectors] Use policy_templates_behavior option with individual_policies mode * Update PR in changelog * Re-trigger the build pipeline

Change property connection_string to be a secret like in the other integrations.

Added to the manifests of both data streams. Already used by the input config for the `access` data stream. Usage added in the input config for the `waf` data stream.

* Fix typos in SQL Input package

…ogs (#11983)

* add and update dashboards * update changelog entry * update dashboards * update kibana version * update changelog entry * add dashboards screen shots * fix changelog * Update packages/vsphere/changelog.yml Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com> * update remaining dashboards with id changes * update kibana version to 8.16.2 * Hosts Overview * Cluster and Network Status Insights * Comprehensive Storage Overview * Resource Pools Overview * VMs overview * update dashboards and its screen shots --------- Co-authored-by: Harnish Chavda <118714680+harnish-elastic@users.noreply.github.com>

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

The API documentation for `vulnerability` mentions the `lastSeen` parameter format should be `#:#` where `#` is in number of days. Although the existing format `@-@` where `@` is epoch seconds is still working, this discrepancy is addressed to avoid any future issues. Other changes: - Updated request's `header.User-Agent` versions in input files.

* add observability category

- Add initial implementation of the syslog router integration which will identify and route syslog-based events to security integrations.

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

pierrehilbert · 2025-02-05T12:40:12Z

@gizas Looks like you have a lot of conflicts to resolve here, could you please have a look?

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

gizas · 2025-02-05T14:08:11Z

/test stack 9.0.0-SNAPSHOT

gizas · 2025-02-05T14:08:44Z

thanks @pierrehilbert , now looks better

elasticmachine · 2025-02-05T14:23:45Z

⏳ Build in-progress, with failures

Buildkite Build
Commit: 52bffe4

Failed CI Steps

Check integrations istio

History

💔 Build #21756 failed 4bbd49f
💔 Build #21753 failed e4d5d8f
💔 Build #21751 failed c9c3a4f
💔 Build #21742 failed f659d27
💔 Build #21741 failed 7e4f714

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

gizas · 2025-02-05T14:47:47Z

/test stack 9.0.0-SNAPSHOT

elastic-sonarqube · 2025-02-05T15:23:53Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
82.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-02-05T15:24:09Z

💚 Build Succeeded

Buildkite Build
Commit: 643c505

elasticmachine · 2025-02-05T15:31:39Z

💚 Build Succeeded

Buildkite Build
Commit: 643c505

elastic-vault-github-plugin-prod · 2025-02-05T15:46:37Z

Package containerd - 0.5.0 containing this change is available at https://epr.elastic.co/package/containerd/0.5.0/

elastic-vault-github-plugin-prod · 2025-02-05T15:46:39Z

Package istio - 0.7.0 containing this change is available at https://epr.elastic.co/package/istio/0.7.0/

elastic-vault-github-plugin-prod · 2025-02-05T15:46:40Z

Package kubernetes - 1.80.0 containing this change is available at https://epr.elastic.co/package/kubernetes/1.80.0/

elastic-vault-github-plugin-prod · 2025-02-05T15:46:41Z

Package kubernetes_otel - 1.1.0 containing this change is available at https://epr.elastic.co/package/kubernetes_otel/1.1.0/

elastic-vault-github-plugin-prod · 2025-02-05T15:46:42Z

Package nginx_ingress_controller - 1.11.0 containing this change is available at https://epr.elastic.co/package/nginx_ingress_controller/1.11.0/

SimonKoetting and others added 30 commits December 17, 2024 08:22

[M365 Defender] change connection_string to secret (#12112)

aa7fb74

Change property connection_string to be a secret like in the other integrations.

Fix broken links in Cloud Defend integration package (#12094)

d0f1d61

* Fix broken links * Update changelog and manifest

chore: update pkgs in .github/ISSUE_TEMPLATE/integration_*.yml (#12128)

ab16c3c

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

ci: enable updates to the feature request template (#12125)

7d4187d

Update link in the Rapid7 Threat Command Integration (#12135)

46f5127

* Update link * Update changelog and manifest

cloudflare_logpush: enable memory limit checks for http_endpoint input (

f978f44

#11920) This is enabled per data stream to allow tuning of behaviour.

google_workspace,jamf_protect,ti_mandiant: add "preserve_original_eve…

af67252

…nt" tag to documents with event.kind set to "pipeline_error" (#12108) This manually replays the changes in #12046.

okta: do not remove event.original in main ingest (#12127)

206aee3

ssi_all: add "preserve_original_event" tag to documents with event.ki…

f4d34e0

…nd manually set to "pipeline_error" (#12109)

Include pipeline test examples to accommodate the new Cloudtrail form…

8bd8b91

…at (#12110) * Include pipeline test examples to accommodate the new Cloudtrail format

chore: update pkgs in .github/ISSUE_TEMPLATE/integration_*.yml (#12141)

b42b13c

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Fix broken links (#12134)

0f9e03a

[New Integration] Envoyproxy (#11215)

506189c

[Search Connectors] Use policy_templates_behavior option with individ…

c0225e7

…ual_policies mode (#12115) * [Search Connectors] Use policy_templates_behavior option with individual_policies mode * Update PR in changelog * Re-trigger the build pipeline

[Azure Frontdoor] change connection_string to secret (#12148)

bb2a59b

Change property connection_string to be a secret like in the other integrations.

[azure_frontdoor] Add storage_account_container variables (#12151)

5badb91

Added to the manifests of both data streams. Already used by the input config for the `access` data stream. Usage added in the input config for the `waf` data stream.

Update SQL input integration to fix typo (#11757)

cbced71

* Fix typos in SQL Input package

Fix broken links (#12133)

8725e01

[MS365] [One Drive] Add One Drive datastream (#11814)

4b637ec

[GCP][CDR] Add actor.entity.id and target.entity.id fields to audit l…

704c017

…ogs (#11983)

chore: update pkgs in .github/ISSUE_TEMPLATE/integration_*.yml (#12162)

9db7f07

Made with ❤️️ by updatecli Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Fix aws logs duplicated keys in agent template (#12100)

5f4e369

Update dga package to spec 3.0.4 (#12153)

0503bba

Add Observability category (#12165)

ba0c790

* add observability category

[syslog_router] Add syslog router integration (#11727)

2d9c125

- Add initial implementation of the syslog router integration which will identify and route syslog-based events to security integrations.

qcorporation force-pushed the k8senablement9.0 branch from 6c95832 to de5073f Compare February 4, 2025 18:13

qcorporation force-pushed the main branch from 7801e00 to ed31586 Compare February 4, 2025 18:34

gizas added 6 commits February 5, 2025 12:31

merging with main

d44b874

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

38f7624

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

f3e6fd9

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

7e4f714

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

f659d27

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

c9c3a4f

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

gizas added 5 commits February 5, 2025 14:42

merging with main

e4d5d8f

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

4bbd49f

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

4df3705

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

64a061a

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

merging with main

52bffe4

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

fixing pipeline error

643c505

Signed-off-by: Andreas Gkizas <andreas.gkizas@elastic.co>

gizas merged commit 043460a into main Feb 5, 2025
7 checks passed

gizas deleted the k8senablement9.0 branch February 5, 2025 15:32

gizas mentioned this pull request Feb 6, 2025

AWS Integration enable v9 Kibana support #12637

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

gizas commented Jan 30, 2025 •

edited

Loading

pierrehilbert commented Feb 5, 2025

gizas commented Feb 5, 2025

gizas commented Feb 5, 2025

elasticmachine commented Feb 5, 2025

gizas commented Feb 5, 2025

elastic-sonarqube bot commented Feb 5, 2025

elasticmachine commented Feb 5, 2025

elasticmachine commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

Enabling 9.0.0 kibana support for kubernetes, kubernetes_otel, nginx_ingress, istio and containerd integrations #12535

Conversation

gizas commented Jan 30, 2025 • edited Loading

Proposed commit message

Checklist

How to test this PR locally

Related issues

Logs

Screenshots

K8s

K8s Otel

Containerd

Istio

Ingress-Controller

Overall

pierrehilbert commented Feb 5, 2025

gizas commented Feb 5, 2025

gizas commented Feb 5, 2025

elasticmachine commented Feb 5, 2025

⏳ Build in-progress, with failures

Failed CI Steps

History

gizas commented Feb 5, 2025

elastic-sonarqube bot commented Feb 5, 2025

Quality Gate passed

elasticmachine commented Feb 5, 2025

💚 Build Succeeded

elasticmachine commented Feb 5, 2025

💚 Build Succeeded

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

elastic-vault-github-plugin-prod bot commented Feb 5, 2025

gizas commented Jan 30, 2025 •

edited

Loading