[PACT] feat: add Contract tests for Kibana Fleet #339

mdelapenya · 2020-10-01T15:47:43Z

What does this PR do?

It adds a contract tests with Pact.io, allowing two things:

From consumer perspective (the e2e tests), define the expected behaviours in the format of contracts
From provider perspective (Kibana Fleet), verify that any change in the code is not breaking the contracts provided by the consumers

It simply adds 3 tests in the contract:

when we query Fleet APIs, then there are integrations, including the proper status code (200 OK)
when we look for an existing integration, it's returned including the proper status code (200 OK)
when we look for a non-existing integration, the right status code is returned (404 not found)

With these tests we detected that in kibana-7.9.1 the error code was 500 server error

The generated contracts are stored locally for simplicity, instead of using a Pact Broker (see docs).

Why is it important?

This is a PoC, but it will allow decoupling Kibana from its consumers, verifying they are not broken in a proactive manner, instead of in a reactive one (when their tests fail, if executed)

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have run the Unit tests for the CLI, and they are passing locally
I have run the End-2-End tests for the suite I'm working on, and they are passing locally
I have noticed new Go dependencies (run make notice in the proper directory)

Author's Checklist

Checked the life cicle of creating contracts and verifying them

How to test this PR locally

From root directory:

$ make pact-consumer
$ make verify-provider

Follow-ups

Move the verification process to Kibana (or the sotware identified as provider)

We are creating it here, instead of in Kibana for simplicity

elasticmachine · 2020-10-01T16:10:31Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [Pull request #339 updated]
Start Time: 2020-10-13T17:36:22.349+0000
Duration: 56 min 30 sec

Test stats 🧪

Test	Results
Failed	17
Passed	54
Skipped	15
Total	86

Test errors

Expand to view the tests failures

Name: Initializing / Pre-Submit / Sanity checks / checkunicodenonbreakingspaces – pre_commit.lint
- Age: 1
- Duration: 0
- Error Details: error
Name: Initializing / Pre-Submit / Sanity checks / removeunicodenonbreakingspaces – pre_commit.lint
- Age: 1
- Duration: 0
- Error Details: error
Name: Initializing / End-To-End Tests / fleet_stand_alone_mode / Starting the ubi8 agent starts backend processes – Stand-alone Agent
- Age: 2
- Duration: 190.49274
- Error Details: Step the "metricbeat" process is in the "started" state on the host
Name: Initializing / End-To-End Tests / fleet_stand_alone_mode / Deploying a ubi8 stand-alone agent – Stand-alone Agent
- Age: 2
- Duration: 183.91039
- Error Details: Step there is new data in the index from agent: Not enough hits in the logs-elastic.agent-default index yet. Current: 0, Desired: 50
Name: Initializing / End-To-End Tests / fleet_agent_endpoint_integration / Adding the Endpoint Integration to an Agent makes the host to show in Security App – Agent Endpoint Integration
- Age: 2
- Duration: 193.93686
- Error Details: Step the "Elastic Endpoint Security" integration is "added" in the policy
Name: Initializing / End-To-End Tests / fleet_agent_endpoint_integration / Deploying an Endpoint makes policies to appear in the Security App – Agent Endpoint Integration
- Age: 2
- Duration: 190.63168
- Error Details: Step the policy response will be shown in the Security App
Name: Initializing / End-To-End Tests / fleet_agent_endpoint_integration / Changing an Agent policy is reflected in the Security App – Agent Endpoint Integration
- Age: 2
- Duration: 190.69608
- Error Details: Step the policy is updated to have "malware" in "detect" mode
Name: Initializing / End-To-End Tests / fleet_agent_endpoint_integration / Un-enrolling Elastic Agent stops Elastic Endpoint – Agent Endpoint Integration
- Age: 2
- Duration: 190.741
- Error Details: Step the agent is un-enrolled
Name: Initializing / End-To-End Tests / fleet_agent_endpoint_integration / Removing Endpoint from Agent policy stops the connected Endpoint – Agent Endpoint Integration
- Age: 2
- Duration: 184.46349
- Error Details: Step the "Elastic Endpoint Security" integration is "removed" in the policy
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Deploying the centos agent – Fleet Mode Agent
- Age: 2
- Duration: 196.9434
- Error Details: Step the "filebeat" process is in the "started" state on the host
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Deploying the debian agent – Fleet Mode Agent
- Age: 2
- Duration: 193.23634
- Error Details: Step the "filebeat" process is in the "started" state on the host
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Restarting the centos host with persistent agent restarts backend processes – Fleet Mode Agent
- Age: 2
- Duration: 204.06195
- Error Details: Step the "filebeat" process is in the "started" state on the host
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Restarting the debian host with persistent agent restarts backend processes – Fleet Mode Agent
- Age: 2
- Duration: 195.60785
- Error Details: Step the "metricbeat" process is in the "started" state on the host
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Un-enrolling the centos agent – Fleet Mode Agent
- Age: 2
- Duration: 190.26399
- Error Details: Step the agent is listed in Fleet as "inactive"
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Un-enrolling the debian agent – Fleet Mode Agent
- Age: 2
- Duration: 194.79884
- Error Details: Step the agent is listed in Fleet as "inactive"
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Re-enrolling the centos agent – Fleet Mode Agent
- Age: 2
- Duration: 19.364483
- Error Details: Step the agent is listed in Fleet as "online"
Name: Initializing / End-To-End Tests / fleet_fleet_mode / Re-enrolling the debian agent – Fleet Mode Agent
- Age: 2
- Duration: 20.145098
- Error Details: Step the agent is listed in Fleet as "online"

Steps errors

Expand to view the steps failures

Name: Run functional tests for fleet:agent_endpoint_integration
- Description:
- Duration: 21 min 53 sec
- Start Time: 2020-10-13T18:03:03.490+0000
- log
Name: Error signal
- Description:
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:23:56.808+0000
- log
Name: General Build Step
- Description: [2020-10-13T18:23:57.306Z] Archiving artifacts
  hudson.AbortException: script returned exit code 1
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:23:57.300+0000
- log
Name: Run functional tests for fleet:stand_alone_mode
- Description:
- Duration: 12 min 31 sec
- Start Time: 2020-10-13T18:03:03.112+0000
- log
Name: Error signal
- Description:
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:14:34.125+0000
- log
Name: General Build Step
- Description: [2020-10-13T18:14:34.601Z] Archiving artifacts
  hudson.AbortException: script returned exit code 1
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:14:34.596+0000
- log
Name: Run functional tests for fleet:fleet_mode
- Description:
- Duration: 29 min 47 sec
- Start Time: 2020-10-13T18:03:02.001+0000
- log
Name: Error signal
- Description:
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:31:48.772+0000
- log
Name: General Build Step
- Description: [2020-10-13T18:31:49.327Z] Archiving artifacts
  hudson.AbortException: script returned exit code 1
- Duration: 0 min 0 sec
- Start Time: 2020-10-13T18:31:49.320+0000
- log

Log output

Expand to view the last 100 lines of log output

[2020-10-13T18:31:34.430Z] Synchronizing state of elastic-agent.service with SysV service script with /lib/systemd/systemd-sysv-install.
[2020-10-13T18:31:34.430Z] Executing: /lib/systemd/systemd-sysv-install enable elastic-agent
[2020-10-13T18:31:34.696Z] time="2020-10-13T18:31:34Z" level=debug msg="Docker compose executed." cmd="[exec -T debian-systemd systemctl enable elastic-agent]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm612366407 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm centos_systemdContainerName:fleet_centos-systemd_centos-systemd_2 centos_systemdTag:latest debian_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-amd64.deb388959994 debian_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-amd64.deb debian_systemdContainerName:fleet_debian-systemd_debian-systemd_2 debian_systemdTag:stretch kibanaConfigPath:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/src/github.com/elastic/e2e-testing/e2e/_suites/fleet/configurations/kibana.config.yml stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2020-10-13T18:31:35.651Z] The Elastic Agent is currently in BETA and should not be used in production
[2020-10-13T18:31:35.652Z] 
[2020-10-13T18:31:35.923Z] 2020-10-13T18:31:35.905Z	DEBUG	kibana/client.go:170	Request method: POST, path: /api/fleet/agents/enroll
[2020-10-13T18:31:36.200Z] fail to enroll: fail to execute request to Kibana: Status code: 401, Kibana returned an error: Unauthorized, message: [security_exception] missing authentication credentials for REST request [/_security/_authenticate], with { header={ WWW-Authenticate={ 0="Basic realm=\"security\" charset=\"UTF-8\"" & 1="ApiKey" } } }
[2020-10-13T18:31:36.201Z] time="2020-10-13T18:31:36Z" level=error msg="Could not execute command in container" command="[elastic-agent enroll http://kibana:5601 SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw== -f --insecure]" error="Could not run compose file: [/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml] - Local Docker compose exited abnormally whilst running docker-compose: [exec -T debian-systemd elastic-agent enroll http://kibana:5601 SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw== -f --insecure]. exit status 1" service=debian-systemd
[2020-10-13T18:31:36.201Z] time="2020-10-13T18:31:36Z" level=error msg="Could not enroll the agent with the token" command="[elastic-agent enroll http://kibana:5601 SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw== -f --insecure]" error="Could not run compose file: [/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml] - Local Docker compose exited abnormally whilst running docker-compose: [exec -T debian-systemd elastic-agent enroll http://kibana:5601 SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw== -f --insecure]. exit status 1" image=debian-systemd service=debian-systemd tag=stretch token="SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw=="
[2020-10-13T18:31:36.201Z] time="2020-10-13T18:31:36Z" level=debug msg="As expected, it's not possible to enroll an agent with a revoked token" err="Could not run compose file: [/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml] - Local Docker compose exited abnormally whilst running docker-compose: [exec -T debian-systemd elastic-agent enroll http://kibana:5601 SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw== -f --insecure]. exit status 1" token="SUZvN0kzVUJWTFd1NkpfVEpxQTQ6cFMtb1owX3hUQjY0R2RBR3JJOXNYdw=="
[2020-10-13T18:31:36.201Z] time="2020-10-13T18:31:36Z" level=warning msg="Could not create command in container" command="[cat /etc/elastic-agent/.elastic-agent.active.commit]" container=27cd375980e9 detach=false error="Error response from daemon: No such container: 27cd375980e9" tty=false
[2020-10-13T18:31:36.201Z] time="2020-10-13T18:31:36Z" level=error msg="Could not get agent hash in the container" containerName=27cd375980e9 error="Error response from daemon: No such container: 27cd375980e9"
[2020-10-13T18:31:36.464Z] time="2020-10-13T18:31:36Z" level=debug msg="Un-enrolling agent in Fleet" agentID=eac60cc2-6c6d-4612-9d82-e411ed4cd05a hostname=27cd375980e9
[2020-10-13T18:31:37.039Z] time="2020-10-13T18:31:36Z" level=debug msg="Fleet agent was unenrolled" agentID=eac60cc2-6c6d-4612-9d82-e411ed4cd05a
[2020-10-13T18:31:37.993Z] Stopping fleet_debian-systemd_debian-systemd_2 ... 
[2020-10-13T18:31:39.387Z] 
Stopping fleet_debian-systemd_debian-systemd_2 ... done
Removing fleet_debian-systemd_debian-systemd_2 ... 
[2020-10-13T18:31:39.387Z] 
Removing fleet_debian-systemd_debian-systemd_2 ... done
Going to remove fleet_debian-systemd_debian-systemd_2
[2020-10-13T18:31:39.387Z] time="2020-10-13T18:31:39Z" level=debug msg="Docker compose executed." cmd="[rm -fvs debian-systemd]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm612366407 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm centos_systemdContainerName:fleet_centos-systemd_centos-systemd_2 centos_systemdTag:latest debian_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-amd64.deb388959994 debian_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-amd64.deb debian_systemdContainerName:fleet_debian-systemd_debian-systemd_2 debian_systemdTag:stretch kibanaConfigPath:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/src/github.com/elastic/e2e-testing/e2e/_suites/fleet/configurations/kibana.config.yml stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2020-10-13T18:31:39.387Z] time="2020-10-13T18:31:39Z" level=debug msg="Service removed from compose" profile=fleet service=debian-systemd
[2020-10-13T18:31:40.334Z] time="2020-10-13T18:31:40Z" level=debug msg="The token was deleted" tokenID=0009af1e-934b-413b-bc6e-d571705aa583
[2020-10-13T18:31:40.334Z] time="2020-10-13T18:31:40Z" level=info msg="Integration deleted from the configuration" integration= packageConfigId= policyID=4164b280-0d82-11eb-b9fd-d331745a681c version=
[2020-10-13T18:31:46.971Z] time="2020-10-13T18:31:46Z" level=debug msg="The policy was deleted" policyID=4164b280-0d82-11eb-b9fd-d331745a681c
[2020-10-13T18:31:46.971Z] time="2020-10-13T18:31:46Z" level=debug msg="Destroying Fleet runtime dependencies"
[2020-10-13T18:31:46.971Z] Stopping fleet_kibana_1           ... 
[2020-10-13T18:31:46.971Z] Stopping fleet_package-registry_1 ... 
[2020-10-13T18:31:46.971Z] Stopping fleet_elasticsearch_1    ... 
[2020-10-13T18:31:48.717Z] 
Stopping fleet_kibana_1           ... done

Stopping fleet_package-registry_1 ... done

Stopping fleet_elasticsearch_1    ... done
Removing fleet_kibana_1           ... 
[2020-10-13T18:31:48.717Z] Removing fleet_package-registry_1 ... 
[2020-10-13T18:31:48.717Z] Removing fleet_elasticsearch_1    ... 
[2020-10-13T18:31:48.717Z] 
Removing fleet_package-registry_1 ... done

Removing fleet_kibana_1           ... done

Removing fleet_elasticsearch_1    ... done
Removing network fleet_default
[2020-10-13T18:31:48.717Z] time="2020-10-13T18:31:48Z" level=debug msg="Docker compose executed." cmd="[down]" composeFilePaths="[/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml]" env="map[centos_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm612366407 centos_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm centos_systemdContainerName:fleet_centos-systemd_centos-systemd_2 centos_systemdTag:latest debian_systemdAgentBinarySrcPath:/tmp/elastic-agent-8.0.0-SNAPSHOT-amd64.deb388959994 debian_systemdAgentBinaryTargetPath:/elastic-agent-8.0.0-SNAPSHOT-amd64.deb debian_systemdContainerName:fleet_debian-systemd_debian-systemd_2 debian_systemdTag:stretch kibanaConfigPath:/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/src/github.com/elastic/e2e-testing/e2e/_suites/fleet/configurations/kibana.config.yml stackVersion:8.0.0-SNAPSHOT]" profile=fleet
[2020-10-13T18:31:48.717Z] time="2020-10-13T18:31:48Z" level=debug msg="Elastic Agent binary was removed." installer=centos-systemd path=/tmp/elastic-agent-8.0.0-SNAPSHOT-x86_64.rpm612366407
[2020-10-13T18:31:48.717Z] time="2020-10-13T18:31:48Z" level=debug msg="Elastic Agent binary was removed." installer=debian-systemd path=/tmp/elastic-agent-8.0.0-SNAPSHOT-amd64.deb388959994
[2020-10-13T18:31:48.717Z] <?xml version="1.0" encoding="UTF-8"?>
[2020-10-13T18:31:48.717Z] <testsuites name="main" tests="12" skipped="0" failures="8" errors="0" time="1687.905728449">
[2020-10-13T18:31:48.717Z]   <testsuite name="Fleet Mode Agent" tests="12" skipped="0" failures="8" errors="0" time="1545.025954569">
[2020-10-13T18:31:48.717Z]     <testcase name="Deploying the centos agent" status="failed" time="196.943409781">
[2020-10-13T18:31:48.717Z]       <failure message="Step the &#34;elastic-agent&#34; process is in the &#34;started&#34; state on the host: elastic-agent process is not running in the container yet"></failure>
[2020-10-13T18:31:48.717Z]       <error message="Step the &#34;filebeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step the &#34;metricbeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step the agent is listed in Fleet as &#34;online&#34;" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step system package dashboards are listed in Fleet" type="skipped"></error>
[2020-10-13T18:31:48.717Z]     </testcase>
[2020-10-13T18:31:48.717Z]     <testcase name="Deploying the debian agent" status="failed" time="193.236343185">
[2020-10-13T18:31:48.717Z]       <failure message="Step the &#34;elastic-agent&#34; process is in the &#34;started&#34; state on the host: elastic-agent process is not running in the container yet"></failure>
[2020-10-13T18:31:48.717Z]       <error message="Step the &#34;filebeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step the &#34;metricbeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step the agent is listed in Fleet as &#34;online&#34;" type="skipped"></error>
[2020-10-13T18:31:48.717Z]       <error message="Step system package dashboards are listed in Fleet" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Stopping the centos agent stops backend processes" status="passed" time="11.107209518"></testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Stopping the debian agent stops backend processes" status="passed" time="13.24816017"></testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Restarting the centos host with persistent agent restarts backend processes" status="failed" time="204.061954924">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;elastic-agent&#34; process is in the &#34;started&#34; state on the host: elastic-agent process is not running in the container yet"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the &#34;filebeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.718Z]       <error message="Step the &#34;metricbeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Restarting the debian host with persistent agent restarts backend processes" status="failed" time="195.607853101">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;filebeat&#34; process is in the &#34;started&#34; state on the host: filebeat process is not running in the container yet"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the &#34;metricbeat&#34; process is in the &#34;started&#34; state on the host" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Un-enrolling the centos agent" status="failed" time="190.263986191">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;elastic-agent&#34; process is in the &#34;started&#34; state on the host: elastic-agent process is not running in the container yet"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the agent is listed in Fleet as &#34;inactive&#34;" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Un-enrolling the debian agent" status="failed" time="194.798840638">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;elastic-agent&#34; process is in the &#34;started&#34; state on the host: elastic-agent process is not running in the container yet"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the agent is listed in Fleet as &#34;inactive&#34;" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Re-enrolling the centos agent" status="failed" time="19.364483319">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;elastic-agent&#34; process is &#34;started&#34; on the host: Could not run compose file: [/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/centos-systemd/docker-compose.yml] - Local Docker compose exited abnormally whilst running docker-compose: [exec -T centos-systemd systemctl start elastic-agent]. exit status 1"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the agent is listed in Fleet as &#34;online&#34;" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Re-enrolling the debian agent" status="failed" time="20.14509846">
[2020-10-13T18:31:48.718Z]       <failure message="Step the &#34;elastic-agent&#34; process is &#34;started&#34; on the host: Could not run compose file: [/var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/profiles/fleet/docker-compose.yml /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339/.op/compose/services/debian-systemd/docker-compose.yml] - Local Docker compose exited abnormally whilst running docker-compose: [exec -T debian-systemd systemctl start elastic-agent]. exit status 1"></failure>
[2020-10-13T18:31:48.718Z]       <error message="Step the agent is listed in Fleet as &#34;online&#34;" type="skipped"></error>
[2020-10-13T18:31:48.718Z]     </testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Revoking the enrollment token for the centos agent" status="passed" time="32.48303963"></testcase>
[2020-10-13T18:31:48.718Z]     <testcase name="Revoking the enrollment token for the debian agent" status="passed" time="24.161823342"></testcase>
[2020-10-13T18:31:48.718Z]   </testsuite>
[2020-10-13T18:31:48.718Z] </testsuites>make: *** [functional-test] Error 1
[2020-10-13T18:31:48.718Z] Makefile:46: recipe for target 'functional-test' failed
[2020-10-13T18:31:48.718Z] + echo 'ERROR: functional-test failed'
[2020-10-13T18:31:48.718Z] ERROR: functional-test failed
[2020-10-13T18:31:48.718Z] + exit_status=1
[2020-10-13T18:31:48.718Z] + sed -e 's/^[ \t]*//; s#>.*failed$#>#g' outputs/TEST-fleet-fleet_mode
[2020-10-13T18:31:48.718Z] + grep -E '^<.*>$'
[2020-10-13T18:31:48.726Z] + exit 1
[2020-10-13T18:31:48.785Z] Recording test results
[2020-10-13T18:31:49.327Z] Archiving artifacts
[2020-10-13T18:31:49.470Z] Failed in branch fleet_fleet_mode
[2020-10-13T18:31:50.581Z] Stage "Release" skipped due to earlier failure(s)
[2020-10-13T18:31:51.475Z] Running on worker-1244230 in /var/lib/jenkins/workspace/e2e-tests_e2e-testing-mbp_PR-339
[2020-10-13T18:31:51.564Z] [INFO] getVaultSecret: Getting secrets
[2020-10-13T18:31:51.695Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-10-13T18:31:54.020Z] + chmod 755 generate-build-data.sh
[2020-10-13T18:31:54.020Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-339/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-339/runs/17 FAILURE 3330277
[2020-10-13T18:31:54.020Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-339/runs/17/steps/?limit=10000 -o steps-info.json
[2020-10-13T18:32:05.001Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-339/runs/17/tests/?status=FAILED -o tests-errors.json
[2020-10-13T18:32:05.701Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/e2e-tests/e2e-testing-mbp/PR-339/runs/17/log/ -o pipeline-log.txt

mdelapenya · 2020-10-01T16:11:39Z

Pinging @MadameSheema and @jfsiii, as they were interested in contract testing

jfsiii · 2020-10-01T17:43:18Z

@mdelapenya Thanks for the ping. I'm excited to watch this and learn more.

Is there any tooling to go from OpenAPI to the Pact JSON format? I don't know much about Pact and would like to avoid describing the HTTP two different ways in two different locations. i.e. OpenAPI in the plugin and Pact in this repo

mdelapenya · 2020-10-01T21:35:24Z

@jfsiii https://docs.pact.io/faq/convinceme/#but-i-use-swaggeropenapi

BTW, did not mention this pr is the result of an space time project, probably not going into the project...

So this PoC performs the verification, but it should be done at kibana side.

v1v · 2020-10-05T13:19:04Z

CONTRACT_TESTING.md

+$ make pact-provider              # will run the verification of the contracts by itself
+$ make verify-provider            # will tear down the runtime dependencies


verify-provider seems to call some other make goals -> https://github.com/elastic/e2e-testing/pull/339/files#diff-b67911656ef5d18c4ae36cb6741b7965R47

Is the comment correct?

Yes, it's on purpose: verify is the last piece in the chain, defining a well-known life cycle: prepare > verify > destroy

I did this that way to support decoupling the destroy goal in the post stage of the CI pipeline (or to be run manually)

cachedout

💯

Co-authored-by: cachedout <mike.place@elastic.co>

This reverts commit c52de78.

mdelapenya added 24 commits September 30, 2020 17:13

chore: add sequence diagrams for Fleet APIs

82ec16c

chore: add a unit test mocking Kibana response

edb05f4

chore: bump minor version for Jeffail/gabs

d599c47

feat: first Pact tests for the E2E tests consumer

bc67521

chore: notice dependencies

e67ac8d

chore: apply basic auth in request headers

19dcd35

chore: move pacts to the root dir

c7a3199

chore: move pact to the root directory

d1deb1a

chore: use a struct for the expected response

dc95dcf

chore: rename variable

81ba767

chore: add first contract test for the provider

53e0679

We are creating it here, instead of in Kibana for simplicity

chore: set provider headers

67ca63c

chore: add a test for getting an existing integration

b87e441

chore: add a test for a non-existing integration

23468ba

fix: kibana returns a 500 when the integration is not found

3834cc7

chore: add request headers in consumer contract

ebd8a75

fix: in master kibana returns a 404 not found, as expected

9a717ff

docs: document pact contracts

cdcf262

chore: add missing license header to tests

b7ca07a

chore: startup and tear down runtime dependencies

4394dee

chore: make sure the files are regen

9bcc623

docs: fix typo

3a3a9fd

chore: decouple the destruction of runtime deps for pact tests

a304d8f

feat(ci): integrate contract tests in the CI

0396fce

mdelapenya self-assigned this Oct 1, 2020

mdelapenya requested a review from a team October 1, 2020 15:47

mdelapenya added 2 commits October 1, 2020 17:48

chore: archive pact logs

f171fcb

fix: forgot to add the when expression

d64ff6a

mdelapenya mentioned this pull request Oct 1, 2020

[PACT] [7.9.x] feat: add Contract tests for Kibana Fleet #340

Closed

9 tasks

mdelapenya added 6 commits October 1, 2020 18:17

chore: make the contract tests timeout to be configurable

b7d1e7a

fix: add docker login for private images

1d84d84

chore: leverage Go build tags to skip pact tests from unit stage

ce53845

chore: add missing license header to test

87a56c3

fix: use declarative instead of classic IF

e4ce4a4

chore: move contract tests stage to later in the pipeline

114ea7c

mdelapenya added 2 commits October 2, 2020 16:26

fixup

f0c8968

fix(ci): use proper path (again)

945876d

v1v reviewed Oct 5, 2020

View reviewed changes

cachedout approved these changes Oct 5, 2020

View reviewed changes

mdelapenya and others added 10 commits October 8, 2020 11:24

fix: typo

8b67b46

Co-authored-by: cachedout <mike.place@elastic.co>

fix: typo

1097e35

Co-authored-by: cachedout <mike.place@elastic.co>

Merge branch 'master' into pact

0190346

Merge branch 'master' into pact

0864146

chore: update fleet APIs

bbae5b0

chore: update fleet paths in contract

87832b9

chore: bump eastic-endpoint version

b5d1a4f

fix: update fleet path

f6d67f7

fix: EPR returns 502 with not found integrations

c52de78

Revert "fix: EPR returns 502 with not found integrations"

0aa330e

This reverts commit c52de78.

mdelapenya mentioned this pull request Oct 13, 2020

[PoC] feat: implement Contract tests for Fleet with Pact.io elastic/kibana#80384

Closed

chore: update diagrams with new endpoints

bb6c7b3

mdelapenya changed the title ~~feat: add Contract tests for Kibana Fleet~~ [PACT] feat: add Contract tests for Kibana Fleet Nov 9, 2020

mdelapenya closed this Aug 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[PACT] feat: add Contract tests for Kibana Fleet #339

[PACT] feat: add Contract tests for Kibana Fleet #339

mdelapenya commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020 •

edited

Loading

Build stats

Test stats 🧪

mdelapenya commented Oct 1, 2020

jfsiii commented Oct 1, 2020

mdelapenya commented Oct 1, 2020 •

edited

Loading

v1v Oct 5, 2020

mdelapenya Oct 5, 2020

cachedout left a comment

		$ make pact-provider # will run the verification of the contracts by itself
		$ make verify-provider # will tear down the runtime dependencies

[PACT] feat: add Contract tests for Kibana Fleet #339

[PACT] feat: add Contract tests for Kibana Fleet #339

Conversation

mdelapenya commented Oct 1, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Follow-ups

elasticmachine commented Oct 1, 2020 • edited Loading

💔 Tests Failed

Build stats

Test stats 🧪

Test errors

Steps errors

Log output

mdelapenya commented Oct 1, 2020

jfsiii commented Oct 1, 2020

mdelapenya commented Oct 1, 2020 • edited Loading

v1v Oct 5, 2020

Choose a reason for hiding this comment

mdelapenya Oct 5, 2020

Choose a reason for hiding this comment

cachedout left a comment

Choose a reason for hiding this comment

mdelapenya commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020 •

edited

Loading

mdelapenya commented Oct 1, 2020 •

edited

Loading