v0.4.0 build.yaml file is missing config-logging configmap, controller and webhook deployments

[khrisrichardson]

Expected Behavior

The v0.4.0 build.yaml should presumably include the config-logging configmap and the controller and webhook deployments. Not sure if it's by design, but relative to the v0.3.0 release, definitions of kind image are also missing.

Actual Behavior

Steps to Reproduce the Problem

Additional Info

[imjasonh]

This is a baffling bug.

Running the release script locally produces a release YAML which includes all the Deployments, etc., you'd expect.

The autorelease process that built the 0.4.0 release when it detected the release-0.4 branch did so after passing integration tests, which indicates it's capable of ko resolve-ing config/ into a full and valid config YAML, but it seems like it re-generates the release YAML fresh after passing those tests.

@nbarthwal are there any steps to debug the autorelease process, or even get its logs to see if it said anything useful while it built the release?

[nbarthwal]

Yes, the logs for auto-release can be seen. Since it is a prow job, like any other jobs - you can obtain the logs for the jobs. Locat the jobs you want to look at from: https://prow.knative.dev/?job=ci-knative-build-auto-release <https://prow.knative.dev/?job=ci-knative-build-auto-release> And then clicking on it will give the full logs. I think the logs for the job that build the 0.4.0 should be at : https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-build-auto-release/1098252910535380992/?log#log <https://gubernator.knative.dev/build/knative-prow/logs/ci-knative-build-auto-release/1098252910535380992/?log#log>

…

On Feb 22, 2019, at 1:56 PM, Jason Hall ***@***.***> wrote: This is a baffling bug. Running the release script locally produces a release YAML which includes all the Deployments, etc., you'd expect. The autorelease process that built the 0.4.0 release when it detected the release-0.4 branch did so after passing integration tests, which indicates it's capable of ko resolve-ing config/ into a full and valid config YAML, but it seems like it re-generates the release YAML fresh after passing those tests. @nbarthwal <https://github.com/nbarthwal> are there any steps to debug the autorelease process, or even get its logs to see if it said anything useful while it built the release? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#566 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AsL15r4ginZckQ1swrx-emxCaFeZTfcdks5vQGd3gaJpZM4bGr8p>.

[imjasonh]

@nbarthwal thanks I'll look through and see if anything stands out.

One thing I think we should fix is that integration tests are run against a config YAML that are then thrown away and regenerated from scratch. AFAICT no integration tests are run against the actual YAML we produce with the release.

[imjasonh]

One curious log line, not sure it's related:

I0220 16:35:17.750] 2019/02/20 16:35:17 error processing import paths in "config/999-cache.yaml": UNKNOWN: Project 'projects/knative-release' not found or deleted.

[jonjohnsonjr]

I'm pretty sure that's supposed to be knative-releases

[imjasonh]

The release didn't build an image for controller (possibly others) for v0.4.0: https://console.cloud.google.com/gcr/images/knative-releases/GLOBAL/github.com/knative/build/cmd/controller

I only see on line indicating an image was published at the end of the release script:

I0220 16:35:17.750] 2019/02/20 16:35:17 Published gcr.io/knative-releases/github.com/knative/build/cmd/nop@sha256:f767af095ef33cca414905fbf0088e1df2df1c92f85de29ba2c91d964a228a3a

[imjasonh]

I'm pretty sure that's supposed to be knative-releases

Yeah. It's a typo, I'm wondering how ko proceeded through that error at all, and I'm wondering if that's why the yaml is incomplete.

[imjasonh]

Re #566 (comment) the nop image seems to be the only one that was published during the release, but it wasn't even tagged v0.4.0 nor included in the final release yaml.

Current theory is that the knative-release typo caused ko to generate partial yaml and only push some images, but somehow succeed enough to convince the release to proceed.

[imjasonh]

So apparently ko resolve will output whatever yamls it can that don't contain references, then build and push images from the references it finds, then output yaml with those references replaced. The typo in .ko.yaml.release caused ko to barf on that second step, after it had output the non-reference-containing yaml.

Normally this would just cause the release to break because ko resolve failed. Somehow, it didn't, and this line failed but the release proceeded and "succeeded".

I'm not sure how that happened since according to this block it looks like we should be in set -o errexit mode.

[imjasonh]

Fixing the typo and rebasing release-0.4 and re-auto-releasing from that branch created a new v0.4. release which includes all the expected resources.