Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create end to end tests for our examples #108

Closed
bobcatfish opened this issue Oct 8, 2018 · 3 comments
Closed

Create end to end tests for our examples #108

bobcatfish opened this issue Oct 8, 2018 · 3 comments
Labels
area/test-infra Issues or PRs related to the testing infrastructure meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given
Milestone
(0.3 Alpha releas...

Comments

@bobcatfish
Copy link
Collaborator

Expected Behavior

We should have a set of example Pipelines and Tasks (currently in examples) which folks can look use as reference to create their own, or even use as is.

These should be regularly tested so we can be sure they are working. We expect the actual test coverage of the Pipeline itself to happen in our integration tests (in test), so this task is simply about making sure the examples work, not about adding coverage (tho we'll probably get that for free anyway).

Actual Behavior

Once #20 is resolved, we will be deploying the yamls in examples (via examples/smoke-test.sh) as part of integration testing, so we will know they comply with the schema, but not that they actually work (i.e. successfully do what they are trying to do).

Steps to Reproduce the Problem

If the examples were to try to run a Build step that contains an image that doesn't exist for example, the integration test script would still pass, but the example wouldn't actually work.

Additional Info

The Build CRD tests do something very similar to what we need here, though since we are using our integration tests for coverage, we probably don't need the invalid tests (just successful tests).
@bobcatfish bobcatfish added help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. labels Oct 8, 2018
bobcatfish referenced this issue in bobcatfish/pipeline Oct 8, 2018
@bobcatfish
Only have one set of examples and test that they are valid
2b31251 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
bobcatfish referenced this issue in bobcatfish/pipeline Oct 8, 2018
@bobcatfish
Only have one set of examples and test that they are valid
62cf6c3 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
  (removing the duplicated list of result endpoints, since we'll be
  getting this from the references instead)
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
@bobcatfish bobcatfish mentioned this issue Oct 8, 2018
Merged
@bobcatfish bobcatfish removed good first issue Denotes an issue ready for a new contributor, according to the "help wanted" guidelines. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. labels Oct 8, 2018
@bobcatfish
Copy link
Collaborator Author

image

I realized this is actually blocked by pretty much the whole october milestone at least, since nothing actually works and it would be hard for someone to jump in and add.

bobcatfish referenced this issue in bobcatfish/pipeline Oct 8, 2018
@bobcatfish
Only have one set of examples and test that they are valid
affcf9a 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
  (removing the duplicated list of result endpoints, since we'll be
  getting this from the references instead)
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
bobcatfish referenced this issue in bobcatfish/pipeline Oct 9, 2018
@bobcatfish
Only have one set of examples and test that they are valid
91793b5 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
  (removing the duplicated list of result endpoints, since we'll be
  getting this from the references instead)
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
knative-prow-robot pushed a commit that referenced this issue Oct 9, 2018
@bobcatfish @knative-prow-robot
Only have one set of examples and test that they are valid
9031ed4 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
  (removing the duplicated list of result endpoints, since we'll be
  getting this from the references instead)
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
@bobcatfish bobcatfish added meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given area/test-infra Issues or PRs related to the testing infrastructure labels Oct 12, 2018
@bobcatfish bobcatfish mentioned this issue Oct 17, 2018
Merged
bobcatfish referenced this issue in bobcatfish/pipeline Nov 2, 2018
@bobcatfish
Use image outputs in examples
cdf57d8 
While talking with @jonjohnsonjr about tektoncd#216 I wanted to show him an
example of a Pipeline wherein there are Tasks that output Images
Resources, which are then used as inputs for subsequent Tasks. Our
examples Pipelines were meant to do that, but unfortunately when I
looked at them, I realized these were not actually hooked up properly.
The Tasks were not actually using the Images that were built by previous
Tasks, so I have updated them to do that.

I also discovered that the templating was incorrect (#108 - adding tests
for these examples - and tektoncd#212 - making sure templated variables are
actually used - can't come quickly enough!) so I've updated that in the
examples as well.
@bobcatfish bobcatfish mentioned this issue Nov 2, 2018
Merged
@bobcatfish bobcatfish added this to the 0.0.1 Alpha release milestone Nov 3, 2018
bobcatfish referenced this issue in bobcatfish/pipeline Nov 7, 2018
@bobcatfish
Use image outputs in examples
30e13da 
While talking with @jonjohnsonjr about tektoncd#216 I wanted to show him an
example of a Pipeline wherein there are Tasks that output Images
Resources, which are then used as inputs for subsequent Tasks. Our
examples Pipelines were meant to do that, but unfortunately when I
looked at them, I realized these were not actually hooked up properly.
The Tasks were not actually using the Images that were built by previous
Tasks, so I have updated them to do that.

I also discovered that the templating was incorrect (#108 - adding tests
for these examples - and tektoncd#212 - making sure templated variables are
actually used - can't come quickly enough!) so I've updated that in the
examples as well.
bobcatfish referenced this issue in bobcatfish/pipeline Nov 7, 2018
@bobcatfish
Use image outputs in examples
09d4d46 
While talking with @jonjohnsonjr about tektoncd#216 I wanted to show him an
example of a Pipeline wherein there are Tasks that output Images
Resources, which are then used as inputs for subsequent Tasks. Our
examples Pipelines were meant to do that, but unfortunately when I
looked at them, I realized these were not actually hooked up properly.
The Tasks were not actually using the Images that were built by previous
Tasks, so I have updated them to do that.

I also discovered that the templating was incorrect (#108 - adding tests
for these examples - and tektoncd#212 - making sure templated variables are
actually used - can't come quickly enough!) so I've updated that in the
examples as well.
bobcatfish referenced this issue in bobcatfish/pipeline Nov 7, 2018
@bobcatfish
Use image outputs in examples
3e4657c 
While talking with @jonjohnsonjr about tektoncd#216 I wanted to show him an
example of a Pipeline wherein there are Tasks that output Images
Resources, which are then used as inputs for subsequent Tasks. Our
examples Pipelines were meant to do that, but unfortunately when I
looked at them, I realized these were not actually hooked up properly.
The Tasks were not actually using the Images that were built by previous
Tasks, so I have updated them to do that.

I also discovered that the templating was incorrect (#108 - adding tests
for these examples - and tektoncd#212 - making sure templated variables are
actually used - can't come quickly enough!) so I've updated that in the
examples as well.
knative-prow-robot pushed a commit that referenced this issue Nov 7, 2018
@bobcatfish @knative-prow-robot
Use image outputs in examples
7af42f1 
While talking with @jonjohnsonjr about #216 I wanted to show him an
example of a Pipeline wherein there are Tasks that output Images
Resources, which are then used as inputs for subsequent Tasks. Our
examples Pipelines were meant to do that, but unfortunately when I
looked at them, I realized these were not actually hooked up properly.
The Tasks were not actually using the Images that were built by previous
Tasks, so I have updated them to do that.

I also discovered that the templating was incorrect (#108 - adding tests
for these examples - and #212 - making sure templated variables are
actually used - can't come quickly enough!) so I've updated that in the
examples as well.
@bobcatfish
Copy link
Collaborator Author

In light of #147 and the fact that we'll be moving Build + BuildTemplate functionality over here, we may want to actually transplant the example test coverage as is from those repos.

bobcatfish referenced this issue in bobcatfish/pipeline Nov 28, 2018
@bobcatfish
Finally commit a demo example that works :D
eeef285 
This example will build two images and then deploy them. It uses the
microservice example in Skaffold (just an arbitrary choice of a simple
deployment situation with 2 apps).

This is the (updated version of) the demo that @dlorenc gave at Jenkins
World.

It's kind of weird that this adds yet another example; I suggest that as
part of #108 we remove the other examples since we've never actually run
them successfully: they were added at the beginning of the project as
examples of where we wanted to go.

Had to update step names b/c using camelcase in container names is not
allowed (opened tektoncd#273 to handle this).

Fixes #89
@bobcatfish bobcatfish mentioned this issue Nov 28, 2018
Merged
bobcatfish referenced this issue in bobcatfish/pipeline Nov 29, 2018
@bobcatfish
Remove guestbook and kritis examples
779b055 
In #108 (where we want to add test coverage to our examples)
@pivotal-nader-ziada struggled to get the examples to work and
had a lot of trouble. This was b/c we had created these examples well
before we had anything actually working, so they were more to show off
the API than anything else. Instead of wrestling with these to make them
work, we can remove them in favour of the simpler example which we know
(at least currently) works.

In the future we can add additional complexity to this Pipeline or just
add more examples.
bobcatfish referenced this issue in bobcatfish/pipeline Nov 29, 2018
@bobcatfish
Finally commit a demo example that works :D
f47adf8 
This example will build two images and then deploy them. It uses the
microservice example in Skaffold (just an arbitrary choice of a simple
deployment situation with 2 apps).

This is the (updated version of) the demo that @dlorenc gave at Jenkins
World.

It's kind of weird that this adds yet another example; I suggest that as
part of #108 we remove the other examples since we've never actually run
them successfully: they were added at the beginning of the project as
examples of where we wanted to go.

Had to update step names b/c using camelcase in container names is not
allowed (opened tektoncd#273 to handle this).

Fixes #89
bobcatfish referenced this issue in bobcatfish/pipeline Nov 29, 2018
@bobcatfish
Remove guestbook and kritis examples
6d08bf7 
In #108 (where we want to add test coverage to our examples)
@pivotal-nader-ziada struggled to get the examples to work and
had a lot of trouble. This was b/c we had created these examples well
before we had anything actually working, so they were more to show off
the API than anything else. Instead of wrestling with these to make them
work, we can remove them in favour of the simpler example which we know
(at least currently) works.

In the future we can add additional complexity to this Pipeline or just
add more examples.

Also keeping the example that demonstrates passing inputs and outputs
between Pipelines - I tried it locally and it works! Just gotta get that
automation from #108 in place so we can keep it that way :D
bobcatfish referenced this issue in bobcatfish/pipeline Nov 30, 2018
@bobcatfish
Remove guestbook and kritis examples
35e7269 
In #108 (where we want to add test coverage to our examples)
@pivotal-nader-ziada struggled to get the examples to work and
had a lot of trouble. This was b/c we had created these examples well
before we had anything actually working, so they were more to show off
the API than anything else. Instead of wrestling with these to make them
work, we can remove them in favour of the simpler example which we know
(at least currently) works.

In the future we can add additional complexity to this Pipeline or just
add more examples.

Also keeping the example that demonstrates passing inputs and outputs
between Pipelines - I tried it locally and it works! Just gotta get that
automation from #108 in place so we can keep it that way :D
knative-prow-robot pushed a commit that referenced this issue Nov 30, 2018
@bobcatfish @knative-prow-robot
Finally commit a demo example that works :D
54051bf 
This example will build two images and then deploy them. It uses the
microservice example in Skaffold (just an arbitrary choice of a simple
deployment situation with 2 apps).

This is the (updated version of) the demo that @dlorenc gave at Jenkins
World.

It's kind of weird that this adds yet another example; I suggest that as
part of #108 we remove the other examples since we've never actually run
them successfully: they were added at the beginning of the project as
examples of where we wanted to go.

Had to update step names b/c using camelcase in container names is not
allowed (opened #273 to handle this).

Fixes #89
knative-prow-robot pushed a commit that referenced this issue Nov 30, 2018
@bobcatfish @knative-prow-robot
Remove guestbook and kritis examples
c09cea6 
In #108 (where we want to add test coverage to our examples)
@pivotal-nader-ziada struggled to get the examples to work and
had a lot of trouble. This was b/c we had created these examples well
before we had anything actually working, so they were more to show off
the API than anything else. Instead of wrestling with these to make them
work, we can remove them in favour of the simpler example which we know
(at least currently) works.

In the future we can add additional complexity to this Pipeline or just
add more examples.

Also keeping the example that demonstrates passing inputs and outputs
between Pipelines - I tried it locally and it works! Just gotta get that
automation from #108 in place so we can keep it that way :D
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Dec 5, 2018
@nader-ziada
Run the examples yaml as an e2e test
01edc0c 
 - issue 108 tektoncd#108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked

Signed-off-by: Nader Ziada <nziada@pivotal.io>
@nader-ziada nader-ziada mentioned this issue Dec 5, 2018
Merged
@nader-ziada
Copy link
Member

/assign

nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Dec 6, 2018
@nader-ziada
Run the examples yaml as an e2e test
05d4ab9 
 - issue 108 tektoncd#108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked

Signed-off-by: Nader Ziada <nziada@pivotal.io>
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Dec 7, 2018
@nader-ziada
Run the examples yaml as an e2e test
6e1a7c1 
 - issue 108 tektoncd#108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked
 - inspired by knative-build
 (https://github.com/knative/build/blob/f7397a3ec59278416aac4890a5a08783c86e5353/test/e2e-common.sh)
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Dec 10, 2018
@nader-ziada
Run the examples yaml as an e2e test
6793c38 
 - issue 108 tektoncd#108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked
 - inspired by knative-build
 (https://github.com/knative/build/blob/f7397a3ec59278416aac4890a5a08783c86e5353/test/e2e-common.sh)

Signed-off-by: Nader Ziada <nziada@pivotal.io>
nader-ziada added a commit to nader-ziada/pipeline that referenced this issue Dec 10, 2018
@nader-ziada
Run the examples yaml as an e2e test
168230f 
 - issue 108 tektoncd#108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked
 - inspired by knative-build
 (https://github.com/knative/build/blob/f7397a3ec59278416aac4890a5a08783c86e5353/test/e2e-common.sh)

Signed-off-by: Nader Ziada <nziada@pivotal.io>
knative-prow-robot pushed a commit that referenced this issue Dec 10, 2018
@nader-ziada @knative-prow-robot
Run the examples yaml as an e2e test
2b3241f 
 - issue 108 #108
 - remove smoke test script and use e2e script running yaml test
 - check status of pipelineRun to make sure the yaml worked
 - inspired by knative-build
 (https://github.com/knative/build/blob/f7397a3ec59278416aac4890a5a08783c86e5353/test/e2e-common.sh)

Signed-off-by: Nader Ziada <nziada@pivotal.io>
vdemeester referenced this issue in openshift/tektoncd-pipeline Apr 5, 2019
@bobcatfish @knative-prow-robot
Only have one set of examples and test that they are valid
71b7abc 
We ended up in a state where we had both an `examples` dir and a
`samples` dir, and neither were tested, so both were slightly out of
sync with each other and the actual types.

This was b/c:
- When we used kubebuilder, we generated our types from an original set
  of examples - previously these were in `config/samples` and they ended
  up in `samples`. We used the validation tests generated by kubebuilder
  to test these, but when we removed kubebuilder we removed the tests
- Before presenting the API to the Build CRD working group, we wanted to
  have some more complex, real world examples, which @tejal29 created
  from the `kritis` project and the k8s guestbook example

This commit makes sure that all of the functionality demonstrated in
`samples` is in `examples`, and removes `samples`. This included:
- Fixing `passedConstraints` to be plural as the types expected
- Including references to the `PipelineParams` in the `TaskRun` example
  (removing the duplicated list of result endpoints, since we'll be
  getting this from the references instead)
- Fixing clusterBindings to refer to the actual names of the clusters
  involved

It also adds a step to the integration tests to deploy the examples,
which will fail if they do not match the expected schema. At this point
they are torn down immediately after creation, but in #108 we can expand
this to actually test that they are working.

Also had to make some tweaks to the types:
- actually including `ClusterBindings` in the `Pipeline`
- Using names which include `Resource` instead of `Source`, which we
  moved away from in #39

Fixes #20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/test-infra Issues or PRs related to the testing infrastructure meaty-juicy-coding-work This task is mostly about implementation!!! And docs and tests of course but that's a given
Projects
None yet
Milestone
(0.3 Alpha release) Docs + Output Res...
Development

No branches or pull requests
2 participants
@bobcatfish @nader-ziada