[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

mdelapenya · 2020-10-01T15:51:57Z

What does this PR do?

It adds a contract tests with Pact.io, allowing two things:

From consumer perspective (the e2e tests), define the expected behaviours in the format of contracts
From provider perspective (Kibana Fleet), verify that any change in the code is not breaking the contracts provided by the consumers

It simply adds 3 tests in the contract:

when we query Fleet APIs, then there are integrations, including the proper status code (200 OK)
when we look for an existing integration, it's returned including the proper status code (200 OK)
when we look for a non-existing integration, the right status code is returned (500 server error)

With these tests we detected that in kibana-8.0.0-SNAPSHOT the error code was 404 not found

The generated contracts are stored locally for simplicity, instead of using a Pact Broker (see docs).

Why is it important?

This is a PoC, but it will allow decoupling Kibana from its consumers, verifying they are not broken in a proactive manner, instead of in a reactive one (when their tests fail, if executed)

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

Checked the life cicle of creating contracts and verifying them

How to test this PR locally

From root directory:

$ make pact-consumer
$ make verify-provider

Related issues

Backport of [PACT] feat: add Contract tests for Kibana Fleet #339

Follow-ups

Move the verification process to Kibana (or the sotware identified as provider)

We are creating it here, instead of in Kibana for simplicity

elasticmachine · 2020-10-01T16:13:57Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Branch indexing
Start Time: 2021-01-07T13:02:11.711+0000
Duration: 4 min 35 sec

Test stats 🧪

Test	Results
Failed	0
Passed	25
Skipped	0
Total	25

Steps errors

Expand to view the steps failures

`Install dependencies`

Took 0 min 16 sec . View more details on here
Description: .ci/scripts/install-dependencies.sh

Log output

Expand to view the last 100 lines of log output

[2021-01-07T13:05:31.299Z] github.com/docker/docker/api/types/swarm
[2021-01-07T13:05:31.299Z] github.com/docker/docker/api/types
[2021-01-07T13:05:31.299Z] github.com/docker/docker/api/types/volume
[2021-01-07T13:05:31.299Z] github.com/docker/docker/client
[2021-01-07T13:05:31.867Z] github.com/elastic/e2e-testing/cli/docker
[2021-01-07T13:05:31.867Z] github.com/testcontainers/testcontainers-go
[2021-01-07T13:05:31.867Z] github.com/elastic/e2e-testing/cli/services
[2021-01-07T13:05:31.867Z] github.com/elastic/e2e-testing/cli/cmd
[2021-01-07T13:05:32.126Z] github.com/elastic/e2e-testing/cli
[2021-01-07T13:05:35.413Z] go test -v -timeout=5m ./... --tags unit
[2021-01-07T13:05:41.983Z] ?   	github.com/elastic/e2e-testing/cli	[no test files]
[2021-01-07T13:05:41.983Z] ?   	github.com/elastic/e2e-testing/cli/cmd	[no test files]
[2021-01-07T13:05:41.983Z] === RUN   TestCheckConfigDirsCreatesWorkspaceAtHome
[2021-01-07T13:05:41.983Z] --- PASS: TestCheckConfigDirsCreatesWorkspaceAtHome (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithTimestamps
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithTimestamps (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithDebugLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithDebugLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithErrorLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithErrorLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithFatalLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithFatalLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithInfoLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithInfoLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithPanicLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithPanicLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithTraceLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithTraceLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithWarningLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithWarningLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestConfigureLoggerWithWrongLogLevel
[2021-01-07T13:05:41.983Z] --- PASS: TestConfigureLoggerWithWrongLogLevel (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestNewConfigPopulatesConfiguration
[2021-01-07T13:05:41.983Z] --- PASS: TestNewConfigPopulatesConfiguration (0.00s)
[2021-01-07T13:05:41.983Z] PASS
[2021-01-07T13:05:41.983Z] ok  	github.com/elastic/e2e-testing/cli/config	0.020s
[2021-01-07T13:05:41.983Z] ?   	github.com/elastic/e2e-testing/cli/docker	[no test files]
[2021-01-07T13:05:41.983Z] === RUN   TestBuild
[2021-01-07T13:05:41.983Z] --- PASS: TestBuild (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestBuildWithGitProtocol
[2021-01-07T13:05:41.983Z] --- PASS: TestBuildWithGitProtocol (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestBuildWithWrongRemote
[2021-01-07T13:05:41.983Z] --- PASS: TestBuildWithWrongRemote (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestBuildWithWellFormedRemote
[2021-01-07T13:05:41.983Z] --- PASS: TestBuildWithWellFormedRemote (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestClone
[2021-01-07T13:05:41.983Z] time="2021-01-07T13:05:41Z" level=info msg="Cloning project. This process could take long depending on its size" directory=/tmp/dir803784712/git/Hello-World url="https://github.com/octocat/Hello-World"
[2021-01-07T13:05:41.983Z] Enumerating objects: 7, done.
[2021-01-07T13:05:41.983Z] Total 7 (delta 0), reused 0 (delta 0), pack-reused 7
[2021-01-07T13:05:41.983Z] time="2021-01-07T13:05:41Z" level=info msg="Git clone succeed" url="https://github.com/octocat/Hello-World"
[2021-01-07T13:05:41.983Z] --- PASS: TestClone (0.37s)
[2021-01-07T13:05:41.983Z] === RUN   TestMkdirAll
[2021-01-07T13:05:41.983Z] --- PASS: TestMkdirAll (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestRecover
[2021-01-07T13:05:41.983Z] --- PASS: TestRecover (0.00s)
[2021-01-07T13:05:41.983Z] === RUN   TestUpdateCreatesStateFile
[2021-01-07T13:05:41.983Z] time="2021-01-07T13:05:41Z" level=error msg="Cannot write file" error="open /tmp/dir774353297/.op/myprofile-profile.run: is a directory" target=/tmp/dir774353297/.op/myprofile-profile.run
[2021-01-07T13:05:41.984Z] time="2021-01-07T13:05:41Z" level=error msg="Could not create state file" stateFile=/tmp/dir774353297/.op/myprofile-profile.run
[2021-01-07T13:05:41.984Z] --- PASS: TestUpdateCreatesStateFile (0.00s)
[2021-01-07T13:05:41.984Z] PASS
[2021-01-07T13:05:41.984Z] ok  	github.com/elastic/e2e-testing/cli/internal	0.385s
[2021-01-07T13:05:41.984Z] === RUN   TestGetIntegrations
[2021-01-07T13:05:41.984Z] --- PASS: TestGetIntegrations (0.00s)
[2021-01-07T13:05:41.984Z] === RUN   TestNewClient
[2021-01-07T13:05:41.984Z] --- PASS: TestNewClient (0.00s)
[2021-01-07T13:05:41.984Z] === RUN   TestNewKibanaClientWithPathStartingWithSlash
[2021-01-07T13:05:41.984Z] --- PASS: TestNewKibanaClientWithPathStartingWithSlash (0.00s)
[2021-01-07T13:05:41.984Z] === RUN   TestNewKibanaClientWithPathStartingWithoutSlash
[2021-01-07T13:05:41.984Z] --- PASS: TestNewKibanaClientWithPathStartingWithoutSlash (0.00s)
[2021-01-07T13:05:41.984Z] === RUN   TestNewKibanaClientWithMultiplePathsKeepsLastOne
[2021-01-07T13:05:41.984Z] --- PASS: TestNewKibanaClientWithMultiplePathsKeepsLastOne (0.00s)
[2021-01-07T13:05:41.984Z] PASS
[2021-01-07T13:05:41.984Z] ok  	github.com/elastic/e2e-testing/cli/services	0.036s
[2021-01-07T13:05:41.984Z] === RUN   TestGetEnvBool
[2021-01-07T13:05:41.984Z] --- PASS: TestGetEnvBool (0.00s)
[2021-01-07T13:05:41.984Z] PASS
[2021-01-07T13:05:41.984Z] ok  	github.com/elastic/e2e-testing/cli/shell	0.012s
[2021-01-07T13:05:41.984Z] make: Leaving directory '/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-340/src/github.com/elastic/e2e-testing/cli'
[2021-01-07T13:05:41.984Z] + go get -v -u github.com/jstemmer/go-junit-report
[2021-01-07T13:05:41.984Z] go: downloading github.com/jstemmer/go-junit-report v0.9.1
[2021-01-07T13:05:41.984Z] go: github.com/jstemmer/go-junit-report upgrade => v0.9.1
[2021-01-07T13:05:42.242Z] github.com/jstemmer/go-junit-report/parser
[2021-01-07T13:05:42.242Z] github.com/jstemmer/go-junit-report/formatter
[2021-01-07T13:05:42.242Z] github.com/jstemmer/go-junit-report
[2021-01-07T13:05:42.514Z] + go-junit-report
[2021-01-07T13:05:43.519Z] Post stage
[2021-01-07T13:05:43.541Z] Recording test results
[2021-01-07T13:05:43.892Z] [Checks API] No suitable checks publisher found.
[2021-01-07T13:05:43.912Z] Archiving artifacts
[2021-01-07T13:05:44.224Z] Stage "Build Docs" skipped due to earlier failure(s)
[2021-01-07T13:05:44.306Z] Stage "Contract Tests" skipped due to earlier failure(s)
[2021-01-07T13:05:44.384Z] Stage "End-To-End Tests" skipped due to earlier failure(s)
[2021-01-07T13:05:44.466Z] Stage "Release" skipped due to earlier failure(s)
[2021-01-07T13:05:45.916Z] Running on worker-395930 in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-340
[2021-01-07T13:05:46.029Z] [INFO] getVaultSecret: Getting secrets
[2021-01-07T13:05:46.113Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-01-07T13:05:48.315Z] + chmod 755 generate-build-data.sh
[2021-01-07T13:05:48.315Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-340/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-340/runs/1 FAILURE 215208
[2021-01-07T13:05:48.315Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-340/runs/1/steps/?limit=10000 -o steps-info.json
[2021-01-07T13:05:50.565Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-340/runs/1/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	25
Skipped	0
Total	25

cachedout

This is really cool. 💯

CONTRACT_TESTING.md

Co-authored-by: cachedout <mike.place@elastic.co>

elasticmachine · 2020-10-08T18:19:36Z

❄️ The following tests failed but also have a history of flakiness and may not be related to this change: [Initializing / End-To-End Tests / ingest-manager_agent_endpoint_integration / Adding the Endpoint Integration to an Agent makes the host to show in Security App – Agent Endpoint Integration, Initializing / End-To-End Tests / ingest-manager_agent_endpoint_integration / Deploying an Endpoint makes policies to appear in the Security App – Agent Endpoint Integration, Initializing / End-To-End Tests / ingest-manager_agent_endpoint_integration / Un-enrolling Elastic Agent stops Elastic Endpoint – Agent Endpoint Integration, Initializing / End-To-End Tests / ingest-manager_agent_endpoint_integration / Removing Endpoint from Agent configuration stops the connected Endpoint – Agent Endpoint Integration, Initializing / End-To-End Tests / ingest-manager_fleet_mode / Deploying the centos agent – Fleet Mode Agent, Initializing / End-To-End Tests / ingest-manager_fleet_mode / Deploying the debian agent – Fleet Mode Agent, Initializing / End-To-End Tests / ingest-manager_fleet_mode / Stopping the centos agent stops backend processes – Fleet Mode Agent, Initializing / End-To-End Tests / ingest-manager_fleet_mode / Stopping the debian agent stops backend processes – Fleet Mode Agent]

mdelapenya added 25 commits October 1, 2020 17:02

chore: add sequence diagrams for Fleet APIs

9c8fe41

chore: update diagrams for 7.9.x

8e88279

chore: add a unit test mocking Kibana response

352fb92

chore: bump minor version for Jeffail/gabs

b5d61b1

feat: first Pact tests for the E2E tests consumer

b29b4fc

chore: notice dependencies

c5a27c3

chore: apply basic auth in request headers

3b58a3e

chore: move pacts to the root dir

22850f1

chore: move pact to the root directory

97f4795

chore: use a struct for the expected response

f552046

chore: rename variable

6788901

chore: add first contract test for the provider

d8dc81b

We are creating it here, instead of in Kibana for simplicity

chore: set provider headers

9148e60

chore: add a test for getting an existing integration

041275f

chore: add a test for a non-existing integration

70b0b25

fix: kibana returns a 500 when the integration is not found

43410e0

chore: add request headers in consumer contract

df54b78

docs: document pact contracts

9f99efe

chore: add missing license header to tests

e8f915d

chore: startup and tear down runtime dependencies

bbcf017

chore: make sure the files are regen

b51ca21

docs: fix typo

9472a11

chore: decouple the destruction of runtime deps for pact tests

a3962f0

feat(ci): integrate contract tests in the CI

5752586

chore: archive pact logs

ff8b931

mdelapenya requested a review from a team October 1, 2020 15:52

mdelapenya added the backport label Oct 1, 2020

fix: forgot to add the when expression

db77744

chore: make the contract tests timeout to be configurable

049ce5a

mdelapenya added 6 commits October 1, 2020 18:50

fix: add docker login for private images

f5bf58b

chore: leverage Go build tags to skip pact tests from unit stage

d92c85d

chore: add missing license header to test

71379d3

fix: use declarative instead of classic IF

009ed46

chore: move contract tests stage to later in the pipeline

429fc0d

fixup

c644e98

mdelapenya force-pushed the pact-7.9.x branch from 6565f19 to c644e98 Compare October 2, 2020 14:25

fix(ci): use proper path (again)

0421b1a

cachedout approved these changes Oct 5, 2020

View reviewed changes

CONTRACT_TESTING.md Outdated Show resolved Hide resolved

CONTRACT_TESTING.md Outdated Show resolved Hide resolved

mdelapenya and others added 3 commits October 8, 2020 11:23

fix: typo

ff7dc23

Co-authored-by: cachedout <mike.place@elastic.co>

fix: typo

2686e50

Co-authored-by: cachedout <mike.place@elastic.co>

Merge branch '7.9.x' into pact-7.9.x

bccab3f

cachedout mentioned this pull request Oct 11, 2020

Fix formatting in flake report elastic/apm-pipeline-library#763

Merged

mdelapenya changed the title ~~[7.9.x] feat: add Contract tests for Kibana Fleet~~ [PACT] [7.9.x] feat: add Contract tests for Kibana Fleet Nov 9, 2020

mdelapenya closed this Aug 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

mdelapenya commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020 •

edited

Loading

Build stats

Test stats 🧪

`Install dependencies`

Test stats 🧪

cachedout left a comment

elasticmachine commented Oct 8, 2020 •

edited

Loading

[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

Conversation

mdelapenya commented Oct 1, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Follow-ups

elasticmachine commented Oct 1, 2020 • edited Loading

💔 Build Failed

Build stats

Test stats 🧪

Steps errors

Install dependencies

Log output

💚 Flaky test report

Test stats 🧪

cachedout left a comment

Choose a reason for hiding this comment

elasticmachine commented Oct 8, 2020 • edited Loading

mdelapenya commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020 •

edited

Loading

`Install dependencies`

elasticmachine commented Oct 8, 2020 •

edited

Loading