Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Nginx] Add SLO's with Nginx Package #10269

Closed
wants to merge 12 commits into from
4 changes: 3 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ require (
github.com/elastic/go-ucfg v0.8.8 // indirect
github.com/elastic/go-windows v1.0.1 // indirect
github.com/elastic/gojsonschema v1.2.1 // indirect
github.com/elastic/kbncontent v0.1.3 // indirect
github.com/elastic/kbncontent v0.1.4 // indirect
github.com/elastic/package-spec/v3 v3.1.4 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/evanphx/json-patch v5.7.0+incompatible // indirect
Expand Down Expand Up @@ -208,3 +208,5 @@ require (
sigs.k8s.io/structured-merge-diff/v4 v4.4.1 // indirect
sigs.k8s.io/yaml v1.3.0 // indirect
)

replace github.com/elastic/package-spec/v3 => github.com/elastic/package-spec/v3 v3.2.1-0.20240626070810-2758f3ed9689
Copy link

@mgiota mgiota Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muthu-mps I just want to verify if this version is correct v3.2.1-0.20240626070810-2758f3ed9689

Copy link

@mgiota mgiota Jun 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano If I understand correctly, this addition is temporary? I mean once the next package-spec is released, do we still need this change? Maybe we should exclude this change from git and just rely on running go run github.com/elastic/elastic-package build locally to be able to test it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@muthu-mps I just want to verify if this version is correct v3.2.1-0.20240626070810-2758f3ed9689

Yes, With this version I am able to build the integrations successfully by adding SLO's. That means it is using the package-spec change to add slo's.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano If I understand correctly, this addition is temporary? I mean once the next package-spec is released, do we still need this change? Maybe we should exclude this change from git and just rely on running go run github.com/elastic/elastic-package build locally to be able to test it?

Yes, this is temporary to allow to run in CI till 3.3.0 is available. This PR cannot be merged till then.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This replace can be removed now, we have just released a version of elastic-package that includes the technical preview of 3.3.0.

8 changes: 4 additions & 4 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,12 @@ github.com/elastic/go-windows v1.0.1 h1:AlYZOldA+UJ0/2nBuqWdo90GFCgG9xuyw9SYzGUt
github.com/elastic/go-windows v1.0.1/go.mod h1:FoVvqWSun28vaDQPbj2Elfc0JahhPB7WQEGa3c814Ss=
github.com/elastic/gojsonschema v1.2.1 h1:cUMbgsz0wyEB4x7xf3zUEvUVDl6WCz2RKcQPul8OsQc=
github.com/elastic/gojsonschema v1.2.1/go.mod h1:biw5eBS2Z4T02wjATMRSfecfjCmwaDPvuaqf844gLrg=
github.com/elastic/kbncontent v0.1.3 h1:qT0RkshRhiAgH51dDD983tJloeLLMCYE/qlRRtn3R8k=
github.com/elastic/kbncontent v0.1.3/go.mod h1:kOPREITK9gSJsiw/WKe7QWSO+PRiZMyEFQCw+CMLAHI=
github.com/elastic/kbncontent v0.1.4 h1:GoUkJkqkn2H6iJTnOHcxEqYVVYyjvcebLQVaSR1aSvU=
github.com/elastic/kbncontent v0.1.4/go.mod h1:kOPREITK9gSJsiw/WKe7QWSO+PRiZMyEFQCw+CMLAHI=
github.com/elastic/package-registry v1.24.0 h1:LAhw2J3qg4av5yGBa/6ZhF6I87RlnAAKSQQnoxxzDf8=
github.com/elastic/package-registry v1.24.0/go.mod h1:c/xP7+n9csO7ts6UFe7x2CkN7M7KGxVg/uyGbbnzMxQ=
github.com/elastic/package-spec/v3 v3.1.4 h1:Z+hFWqHWwzPDBcCALo50haFNqIKVZd8V+zQAHZUeFDA=
github.com/elastic/package-spec/v3 v3.1.4/go.mod h1:o2bb1hKo+BYdXyurVfTidh/s11n0F1NdJ2BZsieUNhY=
github.com/elastic/package-spec/v3 v3.2.1-0.20240626070810-2758f3ed9689 h1:0Ghlg2p/jLPS1c07/S1ZX1x8ZZlRbFqi3Qp5wLYtNgc=
github.com/elastic/package-spec/v3 v3.2.1-0.20240626070810-2758f3ed9689/go.mod h1:dH3ca6UEfn4c4LX3SwznyOEV8ZJN/YJ2PE+x0WeNHvM=
github.com/emicklei/go-restful/v3 v3.11.0 h1:rAQeMHw1c7zTmncogyy8VvRZwtkmkZ4FxERmMY4rD+g=
github.com/emicklei/go-restful/v3 v3.11.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
Expand Down
10 changes: 10 additions & 0 deletions packages/nginx/_dev/build/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,13 @@ Find unusual activity in HTTP access logs.
| `source_ip_request_rate_nginx` | HTTP Access Logs: Detect unusual source IPs - high request rates |
| `low_request_rate_nginx` | HTTP Access Logs: Detect low request rates |

## SLOs

SLOs are usually defined in terms of metrics such as availability, response time, and throughput, and they are used to ensure that the service meets the needs of its users.
You can use the Nginx integration to measure the availability of the Nginx service. The SLOs gets created automatically during the installation of the integration. The user can view the created SLOs in the Observability -> SLOs page.
>Note: To create and manage SLOs you need an appropriate [license](https://www.elastic.co/subscriptions).

| Events | Query |
|-------------|-----------------------------------|
| Good Query | `http.response.status_code < 500` |
| Total Query | `http.response.status_code : *` |
5 changes: 5 additions & 0 deletions packages/nginx/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.22.0"
changes:
- description: Add SLO to the Nginx package.
type: enhancement
link: https://github.com/elastic/integrations/pull/10269
- version: "1.21.0"
changes:
- description: Add global filter on data_stream.dataset to improve performance.
Expand Down
10 changes: 10 additions & 0 deletions packages/nginx/docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -590,3 +590,13 @@ Find unusual activity in HTTP access logs.
| `source_ip_request_rate_nginx` | HTTP Access Logs: Detect unusual source IPs - high request rates |
| `low_request_rate_nginx` | HTTP Access Logs: Detect low request rates |

## SLOs

SLOs are usually defined in terms of metrics such as availability, response time, and throughput, and they are used to ensure that the service meets the needs of its users.
You can use the Nginx integration to measure the availability of the Nginx service. The SLOs gets created automatically during the installation of the integration. The user can view the created SLOs in the Observability -> SLOs page.
>Note: To create and manage SLOs you need an appropriate [license](https://www.elastic.co/subscriptions).

| Events | Query |
|-------------|-----------------------------------|
| Good Query | `http.response.status_code < 500` |
| Total Query | `http.response.status_code : *` |
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
{
"attributes": {
"name": "Nginx Availability",
"description": "",
"indicator": {
"type": "sli.kql.custom",
"params": {
"index": "logs-nginx.*",
"filter": "",
"good": "http.response.status_code < 500",
"total": "http.response.status_code : *",
"timestampField": "@timestamp"
}
},
"budgetingMethod": "occurrences",
"timeWindow": {
"duration": "7d",
"type": "rolling"
},
"objective": {
"target": 0.99
},
"tags": []
},
"id": "nginx-9eb25600-a1f0-11e7-928f-5dbe6f6f5510",
"references": [],
"type": "slo"
}
4 changes: 2 additions & 2 deletions packages/nginx/manifest.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: "3.0.2"
format_version: "3.3.0"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano - I have updated the version to 3.3.0 to test the behaviour and I am getting this error.

Error: building package failed: invalid content found in built zip package: could not load specification for version [3.3.0]: spec version "3.3.0" not found

Can you suggest on how to test this change with the pre-release version?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No released version of package spec contains this spec version, you will need to add a replace in the meantime, something like this:

go mod edit -replace github.com/elastic/package-spec/v3=github.com/elastic/package-spec/v3@main
go mod tidy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the format_version that we could use for testing the package?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is the format_version that we could use for testing the package?

format_version: 3.3.0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano - Tried the above step and with the format_version:3.3.0 we are still getting the same error.

Copy link
Member

@jsoriano jsoriano Jun 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, yes, this should be enough for CI.

If you want to use elastic-package locally with this dependency you can run it as go run github.com/elastic/elastic-package from the integrations repository.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jsoriano!

  • We are getting few validation errors in different package, I am skipping those for now as those are not related.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, after updating go.mod all tests are executed, but you can ignore the ones not related to your package.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No released version of package spec contains this spec version

@jsoriano When is the next release of the package spec?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jsoriano When is the next release of the package spec?

We release more or less on demand, and we have just released 3.2.1. Maybe we can release 3.2.2 soon with a couple of bugfixes. This release would also include 3.3.0 prerelease, what will allow to use SLOs in prerelease packages.

name: nginx
title: Nginx
version: "1.21.0"
version: "1.22.0"
description: Collect logs and metrics from Nginx HTTP servers with Elastic Agent.
type: integration
categories:
Expand Down
4 changes: 4 additions & 0 deletions packages/nginx/validation.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
errors:
exclude_checks:
- PSR00001 # References in dashboards.
- JSE00001 # Kibana version for saved tags.