Meaningful (e2e) tests for examples

[ncskier]

Expected Behavior

We should know when code changes break the examples we are hosting in the Triggers repository.

Actual Behavior

We do not currently run automated tests on all of the examples included in the Triggers repository. So, a code change can break the examples without us knowing about it. This was made apparent recently when a change broke the cron example (#190).

I think that we should try to have meaningful automated tests for all of the examples, and we should have examples for the most common use-cases of Triggers. This will help stop us from making changes that have unintended consequences, and will make it easy for people to start using Triggers.

I also hope that having meaningful tests for each example will force us to keep the number of examples to a small and manageable size. If we don't want to create tests for an example, then we probably shouldn't have the example in the Triggers repository.

Additional Info

I think that at the moment the examples that lack meaningful tests are:

• cron
• getting started guide
• documentation examples

The documentation examples seem to overlap with the cron example and getting started guide, so it might be beneficial to merge the documentation examples into one of the other examples?

[dibyom]

/area testing

[dibyom]

priority/important-soon
/maybe-next-milestone

[tekton-robot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.

/lifecycle stale

Send feedback to tektoncd/plumbing.

[tekton-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

[tekton-robot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.

/lifecycle rotten

Send feedback to tektoncd/plumbing.

[tekton-robot]

@tekton-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

/close

Send feedback to tektoncd/plumbing.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dibyom]

/lifecycle frozen

[dibyom]

Have a branch with a simple bash test runner: https://github.com/tektoncd/triggers/compare/master...dibyom:examples_test?expand=1

[dibyom]

There are 3 parts to this:

• Validate the Trigger YAMLs - This is simple. We can apply the YAMLs from each example and ensure they are created.
• Invoke the Trigger - This would involve connecting to the EventListener, and then sending a HTTP request with some predefined Headers and event body that is different for each example. For my PoC, I created a convention that each example folder would have a separate file for headers and body. The test script will look for these files and use them for crafting the curl request to send to the EventListener. (there are other ways to do this e.g. we could have an convention where each example folder contains a test.sh file with the curl command).
• Verify the Trigger processing succeeded - In my PoC, I was doing this by checking the status code of the HTTP response (201 would indicate the Trigger processing succeeded). However, Send EL response before Triggers finish processing  #1132 even successful trigger processing would return a 202 response. So, we'd have to come up with a way to track the TaskRuns/PipelineRuns that was created -- One way would be to parse the response that came back to find the eventID and then search for resources that were created with that eventID as a label.

I think this can happen incrementally. Even simply applying the YAMLs for all examples would be an improvement (currently, we only apply YAMLs for a subset of examples).

/cc @sm43