Implement custom tester functionality in Skaffold #5451

PriyaModali · 2021-02-25T02:57:22Z

Fixes: ##5395

Description
Add new custom tester functionality to Skaffold. This feature enable users to run custom tests in the Skaffold pipeline.

The test command:

Implements Runner interface
Enables ability to run custom commands between build & deploy phases

Run skaffold dev to run the command in the Skaffold pipeline.

Added unit tests and integrations tests for the newly added test command.

Related:
#5333

Note:
Right now, changing a test dependency file does not trigger a build but triggers only deployment.

Follow up work:
Changing a test dependency file does not trigger a build but will trigger test.

Example

The functionality can be tested using the example in the integration tests:
skaffold/integration/examples/custom-tests

Testing notes

Here are some sample outputs:

skaffold dev - first loop that does the output in test phase

Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Command finished successfully
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully
Tags used in deployment:
 - skaffold-example -> skaffold-example:0a3c7c9f048a204d72b2e4dbc017351754ad8d283505d9cd43e54023eb073ee5
Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
Deployments stabilized in 42.507007ms
Press Ctrl+C to exit
Watching for changes...
[getting-started] ...
....

skaffold dev - changing a test file does not trigger a build

Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Command finished successfully
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully
Tags used in deployment:
 - skaffold-example -> skaffold-example:5b2a46985b6af5b3a545443d67ae6198c100080d897ffd900db702eca7eea9ba
Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
Deployments stabilized in 43.06159ms
Press Ctrl+C to exit
Watching for changes...
[getting-started] ...
....
....
Tags used in deployment:
 - skaffold-example -> skaffold-example:5b2a46985b6af5b3a545443d67ae6198c100080d897ffd900db702eca7eea9ba
Starting deploy...
Waiting for deployments to stabilize...
Deployments stabilized in 54.700905ms
Watching for changes...
[getting-started] ...
....
....

skaffold dev - changing build dependency triggers build and test

Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Command finished successfully
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully
Tags used in deployment:
 - skaffold-example -> skaffold-example:bfe229ee3daf1af14d4db031daf84ebdcb9ddd97a0530fa8fa8d52e66d4c0780
Starting deploy...
 - pod/getting-started created
Waiting for deployments to stabilize...
Deployments stabilized in 49.206668ms
Press Ctrl+C to exit
Watching for changes...
[getting-started] ...
....
....
Generating tags...
 - skaffold-example -> skaffold-example:v1.19.0-85-g55659bd41-dirty
Found [minikube] context, using local docker daemon.
Building [skaffold-example]...
Sending build context to Docker daemon  3.584kB
....
....
Successfully built d21f428215c1
Successfully tagged skaffold-example:v1.19.0-85-g55659bd41-dirty
Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Command finished successfully
Running custom test command: "echo Hello world!!"
Hello world!!
Command finished successfully
Tags used in deployment:
 - skaffold-example -> skaffold-example:d21f428215c1ac7896825508279f6dd06830c93d5a8f189baf8627c58e5d353f
Starting deploy...
 - pod/getting-started configured
Waiting for deployments to stabilize...
Deployments stabilized in 24.410772ms
Watching for changes...
[getting-started] ...
....
....

skaffold dev - Test failure with a timeout

Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Command timed out
Pruning images...
exiting dev mode because test failed after first build: running tests: running custom test command: context deadline exceeded

skaffold dev - Test failure with a non-0 exit code

Testing images...
Running custom test command: "./test.sh" with timeout 60 s
go custom test 
ok      github.com/GoogleContainerTools/skaffold/integration/examples/custom-tests      0.812s
Pruning images...
exiting dev mode because test failed after first build: running tests: running custom test command: Command finished with non-0 exit code: exit status 20

…fold into custom-cmd

…() to reduce code redundancy.

…ewTester() to reduce code redundancy." This reverts commit 5e1d859.

…() to reduce code redundancy.

…fold into custom-cmd

codecov · 2021-02-25T03:09:56Z

Codecov Report

Merging #5451 (27238e5) into master (14d4128) will decrease coverage by 0.08%.
The diff coverage is 61.25%.

@@            Coverage Diff             @@
##           master    #5451      +/-   ##
==========================================
- Coverage   71.49%   71.41%   -0.09%     
==========================================
  Files         397      398       +1     
  Lines       14577    14653      +76     
==========================================
+ Hits        10422    10464      +42     
- Misses       3387     3410      +23     
- Partials      768      779      +11

Impacted Files	Coverage Δ
pkg/skaffold/schema/latest/config.go	`58.82% <ø> (ø)`
pkg/skaffold/test/test_factory.go	`72.09% <33.33%> (-6.86%)`	⬇️
pkg/skaffold/test/custom/custom.go	`58.33% <58.33%> (ø)`
pkg/skaffold/schema/validation/validation.go	`90.41% <100.00%> (+0.46%)`	⬆️
pkg/skaffold/util/tar.go	`50.66% <0.00%> (-5.34%)`	⬇️
pkg/skaffold/docker/image.go	`78.13% <0.00%> (-1.40%)`	⬇️
pkg/skaffold/docker/parse.go	`83.24% <0.00%> (+1.08%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 14d4128...27238e5. Read the comment docs.

…om test fields are not available in the current version of the skaffold config.

Outdated.

PriyaModali · 2021-03-03T00:15:54Z

@briandealwis & @tejal29, thank you for the reviews. I have addressed all the feedback. Welcome more. Otherwise, I'd like to get this into a good state for approval so that I can move on to #5431.

tejal29 · 2021-03-03T04:13:22Z

@briandealwis & @tejal29, thank you for the reviews. I have addressed all the feedback. Welcome more. Otherwise, I'd like to get this into a good state for approval so that I can move on to #5431.

Thanks @PriyaModali. Can you mention little bit in the description what are the output changes. It will be great to include example running skaffold dev on the custom-tests example you have.

I would probably do the following testing and add Testing notes e.g. 0, 1

skaffold dev, - first loop what does the output in test phase look like
skaffold dev - changing a test file does not trigger a build but only test
skaffold dev - changing build dependency triggers build and test.

PriyaModali · 2021-03-03T05:29:49Z

@briandealwis & @tejal29, thank you for the reviews. I have addressed all the feedback. Welcome more. Otherwise, I'd like to get this into a good state for approval so that I can move on to #5431.

Thanks @PriyaModali. Can you mention little bit in the description what are the output changes. It will be great to include example running skaffold dev on the custom-tests example you have.

I would probably do the following testing and add Testing notes e.g. 0, 1

skaffold dev, - first loop what does the output in test phase look like

skaffold dev - changing a test file does not trigger a build but only test

skaffold dev - changing build dependency triggers build and test.

Added Testing notes and future work to the description.

pkg/skaffold/test/custom/custom_test.go

pkg/skaffold/test/custom/custom.go

pkg/skaffold/schema/validation/validation_test.go

pkg/skaffold/test/custom/custom.go

pkg/skaffold/test/custom/custom_test.go

tejal29

something went wrong with github and not all comment went through last time.

…fold into custom-cmd

briandealwis

This is almost done. The integration test needs another instance to verify the test-dependencies-from-a-command, particularly for Windows.

integration/examples/custom-tests/skaffold.yaml

pkg/skaffold/test/custom/custom.go

integration/examples/custom-tests/skaffold.yaml

…fold into custom-cmd

briandealwis

Two small changes and good to go.

Note that the ./test.sh will fail on Windows, but we're not running our integration tests on Windows. I suspect we'll have many other fixes requires for integration tests on Windows.

integration/examples/custom-tests/skaffold.yaml

…fold into custom-cmd

PriyaModali added 16 commits February 21, 2021 16:38

Adding new test runner for custom tester.

b99f88d

Adding new custom tester config to Skaffold comfig.

ddc260d

Adding usage example for newly added custon tester.

2a6e3d8

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

ca99633

…fold into custom-cmd

Extracting structure tests run logic into a seperate method.

7c2083f

Moving structure test's dependency file extraction logic to NewTester…

5e1d859

…() to reduce code redundancy.

Revert "Moving structure test's dependency file extraction logic to N…

83a1dd7

…ewTester() to reduce code redundancy." This reverts commit 5e1d859.

Moving structure test's dependency file extraction logic to NewTester…

3c1c9bc

…() to reduce code redundancy.

Making the custom command a required field.[D

5a71420

Adding new error codes for custom tester.

8163d30

Updating custom test example.

e478fd7

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

79e5a05

…fold into custom-cmd

Removing errors file.

01150a3

Updating custom test example.

d6a272b

Adding generated schema files.

5cbe3d0

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

82ada83

…fold into custom-cmd

PriyaModali added kind/feature-request area/testing Issues concerning the testing phase of Skaffold priority/p1 High impact feature/bug. labels Feb 25, 2021

pull-request-size bot added the size/XL label Feb 25, 2021

google-cla bot added the cla: yes label Feb 25, 2021

PriyaModali added 5 commits February 24, 2021 19:25

Added custom test example to integration/run_test.go.

b36520b

Added custom test example in integration/examples directory.

72ee55e

Updating the apiVersion in example skaffold.yaml.

9307a16

Updating the apiVersion in example skaffold.yaml.

fb26168

Removing the custom-test form examples folder as the newly added cust…

08b0489

…om test fields are not available in the current version of the skaffold config.

PriyaModali marked this pull request as ready for review February 25, 2021 17:05

PriyaModali requested a review from a team as a code owner February 25, 2021 17:05

PriyaModali requested a review from briandealwis February 25, 2021 17:05

tejal29 reviewed Mar 3, 2021

View reviewed changes

PriyaModali added 4 commits March 3, 2021 02:08

Updated unit tests per review feedback.

20d4f0c

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

8823cb5

…fold into custom-cmd

Updated unit tests to use mock by overriding util.RunCmd method.

5694eee

Updating windows specific unit test.

d962ed6

PriyaModali added kokoro:run runs the kokoro jobs on a PR kokoro:force-run forces a kokoro re-run on a PR labels Mar 3, 2021

briandealwis added kokoro:force-run forces a kokoro re-run on a PR and removed kokoro:run runs the kokoro jobs on a PR kokoro:force-run forces a kokoro re-run on a PR labels Mar 3, 2021

kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Mar 3, 2021

briandealwis reviewed Mar 3, 2021

View reviewed changes

integration/examples/custom-tests/skaffold.yaml Show resolved Hide resolved

pkg/skaffold/test/custom/custom.go Show resolved Hide resolved

integration/examples/custom-tests/skaffold.yaml Outdated Show resolved Hide resolved

PriyaModali added 3 commits March 3, 2021 17:31

Added another instance of custom command to the integration test.

c9fe4c5

Updated custom test dependencies.

3dbdea8

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

97f6339

…fold into custom-cmd

PriyaModali mentioned this pull request Mar 4, 2021

Supporting custom tests in Skaffold #5333

Closed

10 tasks

briandealwis approved these changes Mar 4, 2021

View reviewed changes

integration/examples/custom-tests/skaffold.yaml Show resolved Hide resolved

integration/examples/custom-tests/skaffold.yaml Show resolved Hide resolved

PriyaModali added 2 commits March 4, 2021 11:50

Updating custom test example with command dependencies.

d32ce22

Merge branch 'master' of https://github.com/GoogleContainerTools/skaf…

27238e5

…fold into custom-cmd

PriyaModali added the kokoro:force-run forces a kokoro re-run on a PR label Mar 4, 2021

kokoro-team removed the kokoro:force-run forces a kokoro re-run on a PR label Mar 4, 2021

PriyaModali merged commit 2e55bec into GoogleContainerTools:master Mar 4, 2021

PriyaModali mentioned this pull request Mar 8, 2021

[DRAFT - WIP] Add example for CustomTest command #5394

Closed

gofish-bot mentioned this pull request Mar 19, 2021

skaffold 1.21.0 fishworks/fish-food#1297

Merged

nkubala mentioned this pull request Sep 15, 2021

release: v1.32.0 #6603

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement custom tester functionality in Skaffold #5451

Implement custom tester functionality in Skaffold #5451

PriyaModali commented Feb 25, 2021 •

edited

Loading

codecov bot commented Feb 25, 2021 •

edited

Loading

PriyaModali commented Mar 3, 2021

tejal29 commented Mar 3, 2021 •

edited

Loading

PriyaModali commented Mar 3, 2021 •

edited

Loading

tejal29 left a comment

briandealwis left a comment

briandealwis left a comment

Implement custom tester functionality in Skaffold #5451

Implement custom tester functionality in Skaffold #5451

Conversation

PriyaModali commented Feb 25, 2021 • edited Loading

Example

Testing notes

codecov bot commented Feb 25, 2021 • edited Loading

Codecov Report

PriyaModali commented Mar 3, 2021

tejal29 commented Mar 3, 2021 • edited Loading

PriyaModali commented Mar 3, 2021 • edited Loading

tejal29 left a comment

Choose a reason for hiding this comment

briandealwis left a comment

Choose a reason for hiding this comment

briandealwis left a comment

Choose a reason for hiding this comment

PriyaModali commented Feb 25, 2021 •

edited

Loading

codecov bot commented Feb 25, 2021 •

edited

Loading

tejal29 commented Mar 3, 2021 •

edited

Loading

PriyaModali commented Mar 3, 2021 •

edited

Loading