Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

porter archive --> Error preparing artifacts: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws" #1224

Open
digger18 opened this issue Aug 24, 2020 · 14 comments
Open

porter archive --> Error preparing artifacts: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws" #1224

digger18 opened this issue Aug 24, 2020 · 14 comments
Labels
stumped

Comments

@digger18
Copy link

Describe the bug

Unable to bundle several images from quay.io, probably related to google/go-containerregistry#377
Getting error Error preparing artifacts: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws" during porter archive.

Porter Command and Output

$ porter archive ziv-infra.tgz --debug
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Building bundle ===>
Running linters for each mixin used in the manifest...
not all mixins responded successfully: 1 error occurred:
	* error encountered from mixin "helm": exit status 1


Copying porter runtime ===> 
Copying mixins ===> 
Copying mixin exec ===> 
Copying mixin helm ===> 

Generating Dockerfile =======>
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:
actions:
  install: []
  uninstall: []
  upgrade: []

/root/.porter/mixins/helm/helm build --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:
actions:
  install:
  - exec:
      command: ./install.sh
      description: Install Infra Charts
  uninstall:
  - exec:
      command: ./uninstall.sh
      description: Uninstall Infra Charts
  upgrade: []

/root/.porter/mixins/exec/exec build --debug
DEBUG Parsed Input:
&helm.BuildInput{Config:helm.MixinConfig{ClientVersion:"", Repositories:map[string]helm.Repository(nil)}}
FROM debian:stretch

ARG BUNDLE_DIR

RUN apt-get update && apt-get install -y ca-certificates

# exec mixin has no buildtime dependencies

RUN apt-get update && \
 apt-get install -y curl && \
 curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz && \
 tar -xzf helm.tgz && \
 mv linux-amd64/helm /usr/local/bin && \
 rm helm.tgz
RUN helm init --client-only
RUN apt-get update && \
 apt-get install -y apt-transport-https curl && \
 curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl && \
 mv kubectl /usr/local/bin && \
 chmod a+x /usr/local/bin/kubectl

COPY . $BUNDLE_DIR
RUN rm -fr $BUNDLE_DIR/.cnab
COPY .cnab /cnab
COPY porter.yaml $BUNDLE_DIR/porter.yaml
WORKDIR $BUNDLE_DIR
CMD ["/cnab/app/run"]

Writing Dockerfile =======>
FROM debian:stretch

ARG BUNDLE_DIR

RUN apt-get update && apt-get install -y ca-certificates

# exec mixin has no buildtime dependencies

RUN apt-get update && \
 apt-get install -y curl && \
 curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz && \
 tar -xzf helm.tgz && \
 mv linux-amd64/helm /usr/local/bin && \
 rm helm.tgz
RUN helm init --client-only
RUN apt-get update && \
 apt-get install -y apt-transport-https curl && \
 curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl && \
 mv kubectl /usr/local/bin && \
 chmod a+x /usr/local/bin/kubectl

COPY . $BUNDLE_DIR
RUN rm -fr $BUNDLE_DIR/.cnab
COPY .cnab /cnab
COPY porter.yaml $BUNDLE_DIR/porter.yaml
WORKDIR $BUNDLE_DIR
CMD ["/cnab/app/run"]

Starting Invocation Image Build =======> 
Step 1/12 : FROM debian:stretch
 ---> 6d935b41319b
Step 2/12 : ARG BUNDLE_DIR
 ---> Using cache
 ---> 3ea32e8297a3
Step 3/12 : RUN apt-get update && apt-get install -y ca-certificates
 ---> Using cache
 ---> b0d1ddad9678
Step 4/12 : RUN apt-get update &&  apt-get install -y curl &&  curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz &&  tar -xzf helm.tgz &&  mv linux-amd64/helm /usr/local/bin &&  rm helm.tgz
 ---> Using cache
 ---> b8b063ed5d65
Step 5/12 : RUN helm init --client-only
 ---> Using cache
 ---> 640cbf401027
Step 6/12 : RUN apt-get update &&  apt-get install -y apt-transport-https curl &&  curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl &&  mv kubectl /usr/local/bin &&  chmod a+x /usr/local/bin/kubectl
 ---> Using cache
 ---> 903d94f9ea98
Step 7/12 : COPY . $BUNDLE_DIR
 ---> b1ee7f87d9ea
Step 8/12 : RUN rm -fr $BUNDLE_DIR/.cnab
 ---> Running in 5753ff3ad568
Removing intermediate container 5753ff3ad568
 ---> 3ab14cc4e13e
Step 9/12 : COPY .cnab /cnab
 ---> 686c86749dfa
Step 10/12 : COPY porter.yaml $BUNDLE_DIR/porter.yaml
 ---> 4e54bdb3f988
Step 11/12 : WORKDIR $BUNDLE_DIR
 ---> Running in 2473384942b6
Removing intermediate container 2473384942b6
 ---> d0e3989282a3
Step 12/12 : CMD ["/cnab/app/run"]
 ---> Running in 731f476909ae
Removing intermediate container 731f476909ae
 ---> d514c5a35113
Successfully built d514c5a35113
Successfully tagged gcr.io/rec-repo/infra-install-cnab-installer:1.1.4
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Error: Error preparing artifacts: unsupported MediaType: "application/vnd.docker.distribution.manifest.v1+prettyjws", see https://github.com/google/go-containerregistry/issues/377

My porter.yaml

# This is the configuration for Porter
# You must define steps for each action, but the rest is optional
# See https://porter.sh/author-bundles for documentation on how to configure your bundle
# Uncomment out the sections below to take full advantage of what Porter can do!

name: infra-install-cnab
version: 1.1.4
description: "Infra CNAB"
tag: gcr.io/rec-repo/infra-install-cnab:1.1.4

# Uncomment the line below to use a template Dockerfile for your invocation image
#dockerfile: Dockerfile.tmpl

mixins:
- exec
- helm

parameters:
- name: LOCAL_PATH_PROVISIONER
  description: "Install/Upgrade Local Path Provisioner"
  type: string
  default: "true"
  env: LOCAL_PATH_PROVISIONER

- name: CERT_MANAGER
  description: "Install/Upgrade Cert-Manager"
  type: string
  default: "true"
  env: CERT_MANAGER

- name: NGINX_INGRESS
  description: "Install/Upgrade Nginx Ingress"
  type: string
  default: "true"
  env: NGINX_INGRESS

- name: ARGO
  description: "Install/Upgrade Argo"
  type: string
  default: "false"
  env: ARGO

- name: CHARTMUSEUM
  description: "Install/Upgrade ChartMuseum"
  type: string
  default: "true"
  env: CHARTMUSEUM

- name: NGINX_LOCAL_REPO
  description: "Install/Upgrade Nginx Local Repository"
  type: string
  default: "true"
  env: NGINX_LOCAL_REPO

- name: RANCHER
  description: "Install/Upgrade Rancher"
  type: string
  default: "true"
  env: RANCHER

- name: PROMETHEUS_STACK
  description: "Install/Upgrade Prometheus Stack"
  type: string
  default: "true"
  env: PROMETHEUS_STACK

- name: METALLB
  description: "Install/Upgrade MetalLB"
  type: string
  default: "false"
  env: METALLB

- name: IP_ADDRESSES
  description: "IP address range allocated for MetalLB, e.g. {192.168.20.12-192.168.20.15} or {192.168.20.32/32,192.168.20.47/32}"
  type: string
  default: "" # example: {192.168.20.12-192.168.20.15} or {192.168.20.32/32,192.168.20.47/32}
  env: IP_ADDRESSES

  
images:
  metallb-speaker:
    repository: "docker.io/metallb/speaker"
    tag: "v0.9.3"

  metallb-controller:
    repository: "docker.io/metallb/controller"
    tag: "v0.9.3"

  nvidia-dcgm-exporter:
    repository: "nvidia/dcgm-exporter"
    tag: "1.4.6"

  nginx-ingress-controller:
    repository: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller"
    tag: "0.32.0"

  prometheus:
    repository: "quay.io/prometheus/prometheus"
    tag: "v2.17.2"

  prometheus-node-exporter:
    repository: "quay.io/prometheus/node-exporter"
    tag: "v0.18.0"

  prometheus-config-reloader:
    repository: "quay.io/coreos/prometheus-config-reloader"
    tag: "v0.31.1"

  prometheus-configmap-reload:
    repository: "quay.io/coreos/configmap-reload"
    tag: "v0.0.1"

  prometheus-pushgateway:
    repository: "prom/pushgateway"
    tag: "v0.9.1"

  prometheus-operator:
    repository: "quay.io/coreos/prometheus-operator"
    tag: "v0.31.1"

  kube-state-metrics:
    repository: "quay.io/coreos/kube-state-metrics"
    tag: "v1.7.2"

  rancher-agent:
    repository: "rancher/rancher-agent"
    tag: "v2.3.5"

  rancher:
    repository: "rancher/rancher"
    tag: "v2.3.5"

  busybox:
    repository: "busybox"
    tag: "latest"

  cert-manager-cainjector:
    repository: "quay.io/jetstack/cert-manager-cainjector"
    tag: "v0.12.0"

  cert-manager-controller:
    repository: "quay.io/jetstack/cert-manager-controller"
    tag: "v0.12.0"

  cert-manager-webhook:
    repository: "quay.io/jetstack/cert-manager-webhook"
    tag: "v0.12.0"

  kube-webhook-certgen:
    repository: "jettech/kube-webhook-certgen"
    tag: "v1.0.0"

  chartmuseum:
    repository: "chartmuseum/chartmuseum"
    tag: "v0.11.0"

  local-path-provisioner:
    repository: "rancher/local-path-provisioner"
    tag: "v0.0.12"

  defaultbackend-amd64:
    repository: "k8s.gcr.io/defaultbackend-amd64"
    tag: "1.5"

  nginx:
    repository: "nginx"
    tag: "1.17.3-alpine"

  kiwigrid-k8s-sidecar:
    repository: "kiwigrid/k8s-sidecar"
    tag: "0.1.20"

  grafana:
    repository: "grafana/grafana"
    tag: "6.3.5"

  squareup-ghostunnel:
    repository: "squareup/ghostunnel"
    tag: "v1.4.1"

  debian-tall:
    repository: "quay.io/gravitational/debian-tall"
    tag: "stretch"


install:
  - exec:
      description: "Install Infra Charts"
      command: ./install.sh

#upgrade:
#  - exec:
#      description: "World 2.0"
#      command: ./helpers.sh
#      arguments:
#        - upgrade

uninstall:
  - exec:
      description: "Uninstall Infra Charts"
      command: ./uninstall.sh

# See https://porter.sh/wiring/#credentials
credentials:
  - name: kubeconfig
    path: /root/.kube/config

Version

porter v0.27.2 (aee93e9)
@digger18 digger18 added the bug Oops, sorry! label Aug 24, 2020
@digger18
Copy link
Author

porter publish is not working as well:

$ porter publish
Pushing CNAB invocation image...
The push refers to repository [gcr.io/rec-repo/infra-install-cnab-installer]
ec03c7172184: Preparing
49cb89f64712: Preparing
65b8f8742880: Preparing
7ff5a03c4b77: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
7ff5a03c4b77: Layer already exists
65b8f8742880: Layer already exists
ec03c7172184: Layer already exists
49cb89f64712: Layer already exists
219410cd4347: Layer already exists
4e38024e7e09: Layer already exists
8e7363b9dcd4: Layer already exists
90ca8ca0957c: Layer already exists
9b7657a76455: Layer already exists
1.1.4: digest: sha256:f38bc637d97e47762c8625069a12ba2380e3164eed0b3a6bba6f6e5b129080c6 size: 2214
Rewriting CNAB bundle.json...
Starting to copy image gcr.io/rec-repo/infra-install-cnab-installer:1.1.4...
Completed image gcr.io/rec-repo/infra-install-cnab-installer:1.1.4 copy
Starting to copy image chartmuseum/chartmuseum:v0.11.0...
Completed image chartmuseum/chartmuseum:v0.11.0 copy
Starting to copy image quay.io/coreos/prometheus-config-reloader:v0.31.1...
Failed to copy image quay.io/coreos/prometheus-config-reloader:v0.31.1: failed commit on ref "unknown-sha256:5e508e2e0aba161359caad284b6974fa11119a55c2d5786d024ad32cffc61634": unexpected status: 400 Bad Request
Error: failed commit on ref "unknown-sha256:5e508e2e0aba161359caad284b6974fa11119a55c2d5786d024ad32cffc61634": unexpected status: 400 Bad Request

@vdice
Copy link
Member

vdice commented Aug 24, 2020

Thanks for the issue @digger18 . Alas, last we checked, Quay isn't a CNAB-compatible registry. This unfortunately means that these errors are currently expected.

An issue around improving Porter's UX in this area is currently being tracked in #822.

@digger18
Copy link
Author

Thanks for the issue @digger18 . Alas, last we checked, Quay isn't a CNAB-compatible registry. This unfortunately means that these errors are currently expected.

An issue around improving Porter's UX in this area is currently being tracked in #822.

I am not trying to push to quay.io, just pull images from it.
Is it still not compatible just for pulls?

@digger18
Copy link
Author

May I suggest taking a look at this PR for go-containerregistry: google/go-containerregistry#408 which should fix this issue, I assume that Porter uses go-containerregistry under the hood?

@vdice
Copy link
Member

vdice commented Aug 24, 2020

Ah, I see. Thank you for the PR link. Porter is currently on the following commit of go-containerregistry from Oct 15, '19: google/go-containerregistry@71da34e ... which appears to be more recent than the commit in google/go-containerregistry#408, so I'm assuming we have that particular change as well.

I'm also intrigued by the error from porter publish: failed commit on ref "unknown-sha256:5e508e2... This is also a new error to me. Will see what I can find.

@digger18
Copy link
Author

Another strange error while publishing to a local docker registry v2:

$ porter publish --tag localhost:5000/rec-repo/infra-install-cnab-installer:1.1.6 --debug
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Building bundle ===>
Running linters for each mixin used in the manifest...
not all mixins responded successfully: 1 error occurred:
	* error encountered from mixin "helm": exit status 1


Copying porter runtime ===> 
Copying mixins ===> 
Copying mixin exec ===> 
Copying mixin helm ===> 

Generating Dockerfile =======>
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:
actions:
  install: []
  uninstall: []
  upgrade: []

/root/.porter/mixins/helm/helm build --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:
actions:
  install:
  - exec:
      command: ./install.sh
      description: Install Infra Charts
  uninstall:
  - exec:
      command: ./uninstall.sh
      description: Uninstall Infra Charts
  upgrade: []

/root/.porter/mixins/exec/exec build --debug
DEBUG Parsed Input:
&helm.BuildInput{Config:helm.MixinConfig{ClientVersion:"", Repositories:map[string]helm.Repository(nil)}}
FROM debian:stretch

ARG BUNDLE_DIR

RUN apt-get update && apt-get install -y ca-certificates

RUN apt-get update && \
 apt-get install -y curl && \
 curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz && \
 tar -xzf helm.tgz && \
 mv linux-amd64/helm /usr/local/bin && \
 rm helm.tgz
RUN helm init --client-only
RUN apt-get update && \
 apt-get install -y apt-transport-https curl && \
 curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl && \
 mv kubectl /usr/local/bin && \
 chmod a+x /usr/local/bin/kubectl
# exec mixin has no buildtime dependencies


COPY . $BUNDLE_DIR
RUN rm -fr $BUNDLE_DIR/.cnab
COPY .cnab /cnab
COPY porter.yaml $BUNDLE_DIR/porter.yaml
WORKDIR $BUNDLE_DIR
CMD ["/cnab/app/run"]

Writing Dockerfile =======>
FROM debian:stretch

ARG BUNDLE_DIR

RUN apt-get update && apt-get install -y ca-certificates

RUN apt-get update && \
 apt-get install -y curl && \
 curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz && \
 tar -xzf helm.tgz && \
 mv linux-amd64/helm /usr/local/bin && \
 rm helm.tgz
RUN helm init --client-only
RUN apt-get update && \
 apt-get install -y apt-transport-https curl && \
 curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl && \
 mv kubectl /usr/local/bin && \
 chmod a+x /usr/local/bin/kubectl
# exec mixin has no buildtime dependencies


COPY . $BUNDLE_DIR
RUN rm -fr $BUNDLE_DIR/.cnab
COPY .cnab /cnab
COPY porter.yaml $BUNDLE_DIR/porter.yaml
WORKDIR $BUNDLE_DIR
CMD ["/cnab/app/run"]

Starting Invocation Image Build =======> 
Step 1/12 : FROM debian:stretch
 ---> 6d935b41319b
Step 2/12 : ARG BUNDLE_DIR
 ---> Using cache
 ---> 3ea32e8297a3
Step 3/12 : RUN apt-get update && apt-get install -y ca-certificates
 ---> Using cache
 ---> b0d1ddad9678
Step 4/12 : RUN apt-get update &&  apt-get install -y curl &&  curl -o helm.tgz https://get.helm.sh/helm-v2.15.2-linux-amd64.tar.gz &&  tar -xzf helm.tgz &&  mv linux-amd64/helm /usr/local/bin &&  rm helm.tgz
 ---> Using cache
 ---> b8b063ed5d65
Step 5/12 : RUN helm init --client-only
 ---> Using cache
 ---> 640cbf401027
Step 6/12 : RUN apt-get update &&  apt-get install -y apt-transport-https curl &&  curl -o kubectl https://storage.googleapis.com/kubernetes-release/release/v1.15.3/bin/linux/amd64/kubectl &&  mv kubectl /usr/local/bin &&  chmod a+x /usr/local/bin/kubectl
 ---> Using cache
 ---> 903d94f9ea98
Step 7/12 : COPY . $BUNDLE_DIR
 ---> 59ae86030246
Step 8/12 : RUN rm -fr $BUNDLE_DIR/.cnab
 ---> Running in be222fa9a0c3
Removing intermediate container be222fa9a0c3
 ---> a1d524fb2b65
Step 9/12 : COPY .cnab /cnab
 ---> 35a831b7220b
Step 10/12 : COPY porter.yaml $BUNDLE_DIR/porter.yaml
 ---> 75a96a9af4ed
Step 11/12 : WORKDIR $BUNDLE_DIR
 ---> Running in 1eafb61febb0
Removing intermediate container 1eafb61febb0
 ---> 5c2f9dccc2ff
Step 12/12 : CMD ["/cnab/app/run"]
 ---> Running in 3582ddf1738d
Removing intermediate container 3582ddf1738d
 ---> 220b2361b12e
Successfully built 220b2361b12e
Successfully tagged localhost:5000/rec-repo/infra-install-cnab-installer-installer:1.1.6
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Pushing CNAB invocation image...
The push refers to repository [localhost:5000/rec-repo/infra-install-cnab-installer-installer]
c73d9b1c28b6: Preparing
ee467518f309: Preparing
b343dced50d8: Preparing
c98aa429c2e9: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
90ca8ca0957c: Waiting
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
8e7363b9dcd4: Layer already exists
9b7657a76455: Layer already exists
4e38024e7e09: Layer already exists
b343dced50d8: Pushed
c73d9b1c28b6: Pushed
ee467518f309: Pushed
c98aa429c2e9: Pushed
1.1.6: digest: sha256:1cc5fba733388f43410ef670d4714431546b92f99c1ce538a29d3b18c3332c3f size: 2214

Rewriting CNAB bundle.json...
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Starting to copy image localhost:5000/rec-repo/infra-install-cnab-installer-installer:1.1.6...
Completed image localhost:5000/rec-repo/infra-install-cnab-installer-installer:1.1.6 copy
Starting to copy image chartmuseum/chartmuseum:v0.11.0...
Completed image chartmuseum/chartmuseum:v0.11.0 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image grafana/grafana:6.3.5...
Completed image grafana/grafana:6.3.5 copy
Starting to copy image jettech/kube-webhook-certgen:v1.0.0...
Completed image jettech/kube-webhook-certgen:v1.0.0 copy
Starting to copy image rancher/rancher:v2.3.5...
Completed image rancher/rancher:v2.3.5 copy
Starting to copy image rancher/rancher-agent:v2.3.5...
Failed to copy image rancher/rancher-agent:v2.3.5: failed commit on ref "manifest-sha256:94c4d5e47eb8119242ba0afc45b05a421669ed1a63a1b782ba1da3a34c4a2df9": unexpected status: 500 Internal Server Error
Error: failed commit on ref "manifest-sha256:94c4d5e47eb8119242ba0afc45b05a421669ed1a63a1b782ba1da3a34c4a2df9": unexpected status: 500 Internal Server Error

@vdice
Copy link
Member

vdice commented Aug 27, 2020 •
edited
Loading

I've created a repro bundle in https://github.com/vdice/porter-bundles/blob/master/wip/publish-repro/porter.yaml.

Indeed, I see the same (and other) errors from some of the images attempted to be copied. For testing, I've been publishing against a local Docker registry (via docker run -d -p 5000:5000 --name registry registry:2). In that linked Porter manifest, I've added the errors around each problematic image (if it failed, I commented it out).

The errors seen in the registry logs either seem to be:

  1. "canonical digest does match provided digest"
  2. "invalid URL on layer"

I'm surprised that these errors are encountered on well-known/supported images. I'm not sure if there's a setting we can flip in either go-containerregistry or cnab-to-oci to forgo these errors -- if even advisable. (Oh, by the way, I built/used a local Porter binary with the go-containerregistry dep bumped to their latest v0.1.2 release; Porter is already on the latest cnab-to-oci release).

Any experience in this area @radu-matei ?

@digger18 digger18 mentioned this issue Aug 28, 2020
Closed
@digger18
Copy link
Author

Any update on this?

@vdice
Copy link
Member

vdice commented Sep 28, 2020

Unfortunately, no updates here. These are errors coming straight from the referenced libraries (go-containerregistry and cnab-to-oci), both of which were tested with their latest variants. We would need to spend some time with both to debug the issues and/or inquire with the upstream dependencies.

@digger18
Copy link
Author

Unfortunately, no updates here. These are errors coming straight from the referenced libraries (go-containerregistry and cnab-to-oci), both of which were tested with their latest variants. We would need to spend some time with both to debug the issues and/or inquire with the upstream dependencies.

Should I create the issue there?

@vdice
Copy link
Member

vdice commented Sep 29, 2020

@digger18 I had some spare cycles this morning to revisit.

When reviewing my repro bundle (based on the bundle provided in this issue), the two main errors produced by the stock docker registry (v2) are/were:

  • "digest invalid"/"canonical digest does match provided digest"
  • "invalid URL on layer"/"manifest blob unknown"

My hunch was that the first class of errors was due to problematic image tags, so I looked up each affected image and bumped to a more recent tag. These images now succeeded in pushing/copying when publishing the bundle.

The second class only occurred for one image in the repro bundle: rancher/rancher-agent:v2.3.5. In looking at this v2.3.5 tag I see that it is a multi-os/multi-arch tag with support for both Windows and Linux arch combos. My theory is that tags like this may be an issue for certain registries (apparently, the stock docker v2 registry... and perhaps Harbor?) So, I changed the tag to v2.3.5-linux-amd64 and it then succeeded in publishing.

(One additional note on multi-arch tags: publishing didn't have an issue with tags with Linux-only multi-arch images, such as the configmap-reload v0.4.0 tag.)

For an updated porter.yaml of this repro bundle, see: https://github.com/vdice/porter-bundles/blob/ref/publish-repro/wip/publish-repro/porter.yaml

Can you try publishing this updated bundle (or your variant with similar bumped/changed tags) and see if you have better luck publishing?

Thank you!

@carolynvs
Copy link
Member

@vdice Assuming that your thoughts on multi-os/arch are correct, is this a bug with porter that needs fixing? Perhaps something that we need to document?

@carolynvs carolynvs added the hmm 🛑🤔 Needs more thinking time. Don't start on it yet, please. label May 10, 2021
@carolynvs carolynvs added this to the 1.0 milestone May 10, 2021
@vdice
Copy link
Member

vdice commented May 11, 2021

I revisited the same publish-repro bundle. While I still encounter similar errors when publishing to a local registry (using the same registry:2 image), I don't encounter the errors when publishing to my DockerHub account (and I'm able to install from this docker.io/vdice/publish-repro:v0.1.0 reference.) I'm confused as to the discrepancy.

Since it works for one but not the other, I'm not thinking it is a bug with Porter. As for documenting, it still seems like we'd need to pin down the real issue first, unfortunately.

@carolynvs
Copy link
Member

I am making a new issue label, stumped, just for this issue and removing from the 1.0 milestone. If anyone runs into something that looks like this and has more information, please comment! 😀

@carolynvs carolynvs added stumped and removed bug Oops, sorry! hmm 🛑🤔 Needs more thinking time. Don't start on it yet, please. labels May 12, 2021
@carolynvs carolynvs removed this from the 1.0 milestone Oct 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
stumped
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@carolynvs @vdice @digger18