This document provides a summary of the tests included in the Kubernetes conformance test suite. Each test lists a set of formal requirements that a platform that meets conformance requirements must adhere to.
The tests are a subset of the "e2e" tests that make up the Kubernetes testing infrastructure.
Each test is identified by the presence of the [Conformance] keyword in the ginkgo descriptive function calls.
The contents of this document are extracted from comments preceding those [Conformance] keywords
and those comments are expected to include a descriptive overview of what the test is validating using
RFC2119 keywords. This will provide a clear distinction between which bits of code in the tests are
there for the purposes of validating the platform rather than simply infrastructure logic used to setup, or
clean up the tests.
Example:
/*
Testname: Kubelet-OutputToLogs
Description: By default the stdout and stderr from the process
being executed in a pod MUST be sent to the pod's logs.
*/
// Note this test needs to be fixed to also test for stderr
It("it should print the output to logs [Conformance]", func() {would generate the following documentation for the test. Note that the "TestName" from the Documentation above will be used to document the test which make it more human readable. The "Description" field will be used as the documentation for that test.
By default the stdout and stderr from the process being executed in a pod MUST be sent to the pod's logs. Note this test needs to be fixed to also test for stderr
Notational Conventions when documenting the tests with the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are to be interpreted as described in RFC 2119.
Note: Please see the Summary at the end of this document to find the number of tests documented for conformance.
Create a random Custom Resource Definition and make sure the API returns success.
Make sure config map value can be used as an environment variable in the container (on container.env field)
Make sure config map value can be used as an source for environment variables in the container (on container.envFrom field)
Make sure config map without mappings works by mounting it to a volume with a custom path (mapping) on the pod with no other settings.
Make sure config map without mappings works by mounting it to a volume with a custom path (mapping) on the pod with defaultMode set
Make sure config map without mappings works by mounting it to a volume with a custom path (mapping) on the pod as non-root.
Make sure config map works by mounting it to a volume with a custom path (mapping) on the pod with no other settings and make sure the pod actually consumes it.
Make sure config map works with an item mode (e.g. 0400) for the config map item.
Make sure config map works when it is mounted as non-root.
Make sure update operation is working on config map and the result is observed on volumes mounted in containers.
Make sure Create, Update, Delete operations are all working on config map and the result is observed on volumes mounted in containers.
Make sure config map works when it mounted as two different volumes on the same node.
Make sure that pod with readiness probe should not be ready before initial delay and never restart.
Make sure that pod with readiness probe that fails should never be ready and never restart.
Make sure the pod is restarted with a cat /tmp/health liveness probe.
Make sure the pod is not restarted with a cat /tmp/health liveness probe.
Make sure when http liveness probe fails, the pod should be restarted.
Slow by design (5 min)
Make sure when a pod gets restarted, its start count should increase.
Make sure when http liveness probe succeeds, the pod should not be restarted.
Make sure that the pod is restarted with a docker exec liveness probe with timeout.
When a Pod is created neither 'command' nor 'args' are provided for a Container, ensure that the docker image's default command and args are used.
When a Pod is created and 'args' are provided for a Container, ensure that they take precedent to the docker image's default arguments, but that the default command is used.
Note: when you override the entrypoint, the image's arguments (docker cmd) are ignored.
When a Pod is created and 'command' is provided for a Container, ensure that it takes precedent to the docker image's default command.
When a Pod is created and 'command' and 'args' are provided for a Container, ensure that they take precedent to the docker image's default command and arguments.
Ensure that downward API can provide pod's name, namespace and IP address as environment variables.
Ensure that downward API can provide an IP address for host node as an environment variable.
Ensure that downward API can provide CPU/memory limit and CPU/memory request as environment variables.
Ensure that downward API can provide default node allocatable values for CPU and memory as environment variables if CPU and memory limits are not specified for a container.
Ensure that downward API can provide pod UID as an environment variable.
Ensure that downward API can provide pod's name through DownwardAPIVolumeFiles.
Ensure that downward API can set default file permission mode for DownwardAPIVolumeFiles if no mode is specified.
Ensure that downward API can set file premission mode for DownwardAPIVolumeFiles.
Ensure that downward API updates labels in DownwardAPIVolumeFiles when pod's labels get modified.
Ensure that downward API updates annotations in DownwardAPIVolumeFiles when pod's annotations get modified.
Ensure that downward API can provide container's CPU limit through DownwardAPIVolumeFiles.
Ensure that downward API can provide container's memory limit through DownwardAPIVolumeFiles.
Ensure that downward API can provide container's CPU request through DownwardAPIVolumeFiles.
Ensure that downward API can provide container's memory request through DownwardAPIVolumeFiles.
Ensure that downward API can provide default node allocatable value for CPU through DownwardAPIVolumeFiles if CPU limit is not specified for a container.
Ensure that downward API can provide default node allocatable value for memory through DownwardAPIVolumeFiles if memory limit is not specified for a container.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure the volume has 0777 unix file permissions and tmpfs mount type.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a root owned file with 0644 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a root owned file with 0666 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a root owned file with 0777 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a user owned file with 0644 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a user owned file with 0666 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume with 'medium' of 'Memory', ensure a user owned file with 0777 unix file permissions is created correctly, has tmpfs mount type, and enforces the permissions.
For a Pod created with an 'emptyDir' Volume, ensure the volume has 0777 unix file permissions.
For a Pod created with an 'emptyDir' Volume, ensure a root owned file with 0644 unix file permissions is created and enforced correctly.
For a Pod created with an 'emptyDir' Volume, ensure a root owned file with 0666 unix file permissions is created and enforced correctly.
For a Pod created with an 'emptyDir' Volume, ensure a root owned file with 0777 unix file permissions is created and enforced correctly.
For a Pod created with an 'emptyDir' Volume, ensure a user owned file with 0644 unix file permissions is created and enforced correctly.
For a Pod created with an 'emptyDir' Volume, ensure a user owned file with 0666 unix file permissions is created and enforced correctly.
For a Pod created with an 'emptyDir' Volume, ensure a user owned file with 0777 unix file permissions is created and enforced correctly.
Make sure environment variables can be set using an expansion of previously defined environment variables
Make sure a container's commands can be set using an expansion of environment variables.
Make sure a container's args can be set using an expansion of environment variables.
For a Pod created with a 'HostPath' Volume, ensure the volume is a directory with 0777 unix file permissions and that is has the sticky bit (mode flag t) set.
Make sure Kubelet correctly manages /etc/hosts and mounts it into the container.
Try to hit all endpoints through a test container, retry 5 times, expect exactly one unique hostname. Each of these endpoints reports its own hostname.
Try to hit test endpoints from a test container and make sure each of them can report a unique hostname.
Try to hit test endpoints from a test container using udp and make sure each of them can report a unique hostname.
Try to hit test endpoints from the pod and make sure each of them can report a unique hostname.
Try to hit test endpoints from the pod using udp and make sure each of them can report a unique hostname.
Make sure when a pod is created that it is assigned a host IP Address.
Makes sure a pod is created, a watch can be setup for the pod, pod creation was observed, pod is deleted, and pod deletion is observed.
Make sure it is possible to successfully update a pod's labels.
Make sure it is possible to create a pod, update its activeDeadlineSecondsValue, and then waits for the deadline to pass and verifies the pod is terminated.
Make sure that when a pod is created it contains environment variables for each active service.
Simple projected Secret test with no defaultMode set.
Simple projected Secret test with defaultMode set.
Simple projected Secret test as non-root with defaultMode and fsGroup set.
Simple projected Secret test, by setting a secret and mounting it to a volume with a custom path (mapping) on the pod with no other settings and make sure the pod actually consumes it.
Repeat the projected-secret-simple-mapped but this time with an item mode (e.g. 0400) for the secret map item.
Make sure secrets works when mounted as two different volumes on the same node.
Make sure secrets works when optional updates included.
Part 2/3 - ConfigMaps
Make sure that a projected volume with a configMap with no mappings succeeds properly.
Make sure that a projected volume configMap is consumable with defaultMode set.
Make sure that a projected volume configMap is consumable by a non-root userID.
Simplest projected ConfigMap test, by setting a config map and mounting it to a volume with a custom path (mapping) on the pod with no other settings and make sure the pod actually consumes it.
Repeat the projected-configmap-simple-mapped but this time with an item mode (e.g. 0400) for the config map item
Repeat the projected-config-map-simple-mapped but this time with a user other than root.
Make sure that if a projected volume has configMaps, that the values in these configMaps can be updated, deleted, and created.
Make sure that if a projected volume has optional configMaps, that the values in these configMaps can be updated, deleted, and created.
Make sure config map works when it mounted as two different volumes on the same node.
Ensure that downward API can provide pod's name through DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API can set default file permission mode for DownwardAPIVolumeFiles if no mode is specified in a projected volume.
Ensure that downward API can set file permission mode for DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API updates labels in DownwardAPIVolumeFiles when pod's labels get modified in a projected volume.
Ensure that downward API updates annotations in DownwardAPIVolumeFiles when pod's annotations get modified in a projected volume.
Ensure that downward API can provide container's CPU limit through DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API can provide container's memory limit through DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API can provide container's CPU request through DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API can provide container's memory request through DownwardAPIVolumeFiles in a projected volume.
Ensure that downward API can provide default node allocatable value for CPU through DownwardAPIVolumeFiles if CPU limit is not specified for a container in a projected volume.
Ensure that downward API can provide default node allocatable value for memory through DownwardAPIVolumeFiles if memory limit is not specified for a container in a projected volume.
Test multiple projections
This test projects a secret and configmap into the same directory to ensure projection is working as intended.
Ensure that secret can be consumed via environment variables.
Ensure that secret can be consumed via source of a set of ConfigMaps.
Ensure that secret can be mounted without mapping to a pod volume.
Ensure that secret can be mounted without mapping to a pod volume in default mode.
Ensure that secret can be mounted without mapping to a pod volume as non-root in default mode with fsGroup set.
Ensure that secret can be mounted with mapping to a pod volume.
Ensure that secret can be mounted with mapping to a pod volume in item mode.
Ensure that secret can be mounted to multiple pod volumes.
Ensure that optional update change to secret can be reflected on a mounted volume.
TODO: test proxy options (static, prefix, etc)
Make sure that DNS can resolve the names of clusters.
Make sure that DNS can resolve the names of services.
Port here has to be kept in sync with default kubelet port.
Ensure that proxy on node logs works with generic top level prefix proxy and explicit kubelet port.
Ensure that proxy on node logs works with generic top level prefix proxy.
Ensure that proxy on node logs works with node proxy subresource and explicit kubelet port.
Ensure that proxy on node logs works with node proxy subresource.
Ensure that proxy through a service and a pod works with both generic top level prefix proxy and proxy subresource.
Make sure kubernetes service does exist.
Ensure a service with no pod, one pod or two pods has valid/accessible endpoints (same port number for service and pods).
Ensure a service with no pod, one pod or two pods has valid/accessible endpoints (different port number for pods).
Ensure service endpoint's latency is not high (e.g. p50 < 20 seconds and p99 < 50 seconds). If any call to the service endpoint fails, the test will also fail.
Flaky issue #36821.
Makes sure a pod's preStop handler is successfully invoked immediately before a container is terminated.
This test verifies we don't allow scheduling of pods in a way that sum of limits of pods is greater than machines capacity. It assumes that cluster add-on pods stay stable and cannot be run in parallel with any other test that touches Nodes or Pods. It is so because we need to have precise control on what's running in the cluster. Test scenario:
- Find the amount CPU resources on each node.
- Create one pod with affinity to each node that uses 70% of the node CPU.
- Wait for the pods to be scheduled.
- Create another pod with no affinity to any node that need 50% of the largest node CPU.
- Make sure this additional pod is not scheduled.
Ensure that scheduler accounts node resources correctly and respects pods' resource requirements during scheduling.
Test Nodes does not have any label, hence it should be impossible to schedule Pod with nonempty Selector set.
Ensure that scheduler respects the NodeSelector field of PodSpec during scheduling (when it does not match any node).
Ensure that scheduler respects the NodeSelector field of PodSpec during scheduling (when it matches).
Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart exec hook properly
Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop exec hook properly
Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly
Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop http hook properly
container runtime conformance blackbox test when starting a container that exits it should run with the expected status
Total Conformance Tests: 159, total legacy tests that need conversion: 0, while total tests that need comment sections: 37