Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error on publish to Harbor 2.0 (OCI-compliant) registry #1229

Closed
digger18 opened this issue Aug 25, 2020 · 20 comments
Closed

Error on publish to Harbor 2.0 (OCI-compliant) registry #1229

digger18 opened this issue Aug 25, 2020 · 20 comments
Labels
bug Oops, sorry!

Comments

@digger18
Copy link

Describe the bug

Harbor 2.0 should be fully OCI-compliant (https://goharbor.io/blog/harbor-2.0/) registry, but Porter fails to publish CNAB bundles to it.
In the Harbor UI, there are specific instructions on how to push CNAB packages using cnab-to-oci which makes even a stronger case why Porter should be able to publish to it.
The push command is: cnab-to-oci push CNAB_PATH --target 172.31.33.xxx/library/REPOSITORY[:TAG] --auto-update-bundle

Expected behavior

CNAB bundle is successfully pushed to Harbor and available for deployment.

Porter Command and Output

# porter publish --debug
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Pushing CNAB invocation image...
The push refers to repository [172.31.33.xxx/library/infra-cnab-installer]
08d3279ecc54: Preparing
b92dc245d8c2: Preparing
8363e3460c66: Preparing
4eb4a6386afd: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
90ca8ca0957c: Waiting
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
219410cd4347: Mounted from library/infra-install-cnab-installer
8363e3460c66: Pushed
08d3279ecc54: Pushed
90ca8ca0957c: Mounted from library/infra-install-cnab-installer
8e7363b9dcd4: Mounted from library/infra-install-cnab-installer
9b7657a76455: Mounted from library/infra-install-cnab-installer
4e38024e7e09: Mounted from library/infra-install-cnab-installer
4eb4a6386afd: Pushed
b92dc245d8c2: Pushed
0.0.1: digest: sha256:b1f065f32e9ca151fc47d6905c901ffb306ca7d01b92a096c130cb0d51989dca size: 2214

Rewriting CNAB bundle.json...
DEBUG name:    arm
DEBUG pkgDir: /root/.porter/mixins/arm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/arm/arm version --output json --debug
DEBUG name:    aws
DEBUG pkgDir: /root/.porter/mixins/aws
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/aws/aws version --output json --debug
DEBUG name:    az
DEBUG pkgDir: /root/.porter/mixins/az
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/az/az version --output json --debug
DEBUG name:    exec
DEBUG pkgDir: /root/.porter/mixins/exec
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/exec/exec version --output json --debug
DEBUG name:    gcloud
DEBUG pkgDir: /root/.porter/mixins/gcloud
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/gcloud/gcloud version --output json --debug
DEBUG name:    helm
DEBUG pkgDir: /root/.porter/mixins/helm
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/helm/helm version --output json --debug
DEBUG name:    kubernetes
DEBUG pkgDir: /root/.porter/mixins/kubernetes
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/kubernetes/kubernetes version --output json --debug
DEBUG name:    terraform
DEBUG pkgDir: /root/.porter/mixins/terraform
DEBUG file:     
DEBUG stdin:

/root/.porter/mixins/terraform/terraform version --output json --debug
Starting to copy image 172.31.33.xxx/library/infra-cnab-installer:0.0.1...
Failed to copy image 172.31.33.xxx/library/infra-cnab-installer:0.0.1: failed to resolve or push image for service "InvocationImage"
Error: failed to resolve or push image for service "InvocationImage"

Version

porter v0.27.2 (aee93e9)

@digger18 digger18 added the bug Oops, sorry! label Aug 25, 2020
@vdice
Copy link
Member

vdice commented Aug 27, 2020

To double-check, does cnab-to-oci push CNAB_PATH --target 172.31.33.xxx/library/REPOSITORY[:TAG] --auto-update-bundle work when the bundle.json generated by Porter is supplied? (It should exist at .cnab/bundle.json in the working directory of the Porter bundle, after porter build is run.)

@digger18
Copy link
Author

To double-check, does cnab-to-oci push CNAB_PATH --target 172.31.33.xxx/library/REPOSITORY[:TAG] --auto-update-bundle work when the bundle.json generated by Porter is supplied? (It should exist at .cnab/bundle.json in the working directory of the Porter bundle, after porter build is run.)

Just tested, it does not work either.
Exact same error.

@vdice
Copy link
Member

vdice commented Aug 27, 2020

@digger18 thank you for testing. As the errors are the same, I'm tempted to file this under the https://github.com/cnabio/cnab-to-oci repo as a result. Perhaps some maintainers of that repo might have some ideas.
Porter is currently on the most recent release of cnab-to-oci, but we can bump when the issue(s) and fix(es) make it into cnab-to-oci.

@vdice
Copy link
Member

vdice commented Aug 27, 2020

Oh wait, I did notice the invocation image name is: 172.31.33.xxx/library/infra-cnab-installer:0.0.1. It wouldn't be an official library image, right? I wonder if this is the cause of failure. Can the bundle tag value in the Porter manifest be provided? I wonder if somehow library is mistakenly replacing an org/domain value you've supplied.

@radu-matei
Copy link
Contributor

Thanks for reporting this, @digger18.
Could you please try using this minimal bundle from the cnab-to-oci repo?

(I don't have access to a Harbor instance right now, but I will try to reproduce it as well.)

@digger18
Copy link
Author

Thanks for reporting this, @digger18.
Could you please try using this minimal bundle from the cnab-to-oci repo?

(I don't have access to a Harbor instance right now, but I will try to reproduce it as well.)

It works with the minimal bundle:

# cnab-to-oci push bundle.json --insecure-registries 172.31.33.128:80 --target 172.31.33.128:80/library/helloworld:0.1.1 --auto-update-bundle
Starting to copy image cnab/helloworld:0.1.1...
Completed image cnab/helloworld:0.1.1 copy
Pushed successfully, with digest "sha256:55c6da48fe8553696bf3e2c940ea667e2d2fda3d55614a19dbd7d084780c0807"

It led me to the understanding that I require --insecure-registry in my porter publish command, but it still does not work with my bundle:

# porter publish --insecure-registry
Pushing CNAB invocation image...
The push refers to repository [172.31.33.128:80/library/infra-cnab-installer]
78260943389b: Preparing
06e97b022485: Preparing
8c42c6e74806: Preparing
1ee8b2c50107: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
8c42c6e74806: Pushed
8e7363b9dcd4: Layer already exists
9b7657a76455: Layer already exists
78260943389b: Pushed
4e38024e7e09: Layer already exists
06e97b022485: Pushed
1ee8b2c50107: Pushed
1.0.0: digest: sha256:f95c7056860e1f9ac90b7c9668cd7447a1d1d1260c6e81f6844ab50b63947874 size: 2214

Rewriting CNAB bundle.json...
Starting to copy image 172.31.33.128:80/library/infra-cnab-installer:1.0.0...
Failed to copy image 172.31.33.128:80/library/infra-cnab-installer:1.0.0: failed to resolve or push image for service "InvocationImage"
Error: failed to resolve or push image for service "InvocationImage"

My slightly revised porter.yaml:

# cat porter.yaml 
# This is the configuration for Porter
# You must define steps for each action, but the rest is optional
# See https://porter.sh/author-bundles for documentation on how to configure your bundle
# Uncomment out the sections below to take full advantage of what Porter can do!

name: infra-cnab
version: 1.0.0
description: "Infra CNAB"
tag: 172.31.33.128:80/library/infra-cnab:1.0.0

# Uncomment the line below to use a template Dockerfile for your invocation image
#dockerfile: Dockerfile.tmpl

mixins:
- exec
- helm

parameters:
- name: LOCAL_PATH_PROVISIONER
  description: "Install/Upgrade Local Path Provisioner"
  type: string
  default: "true"
  env: LOCAL_PATH_PROVISIONER

- name: CERT_MANAGER
  description: "Install/Upgrade Cert-Manager"
  type: string
  default: "true"
  env: CERT_MANAGER

- name: NGINX_INGRESS
  description: "Install/Upgrade Nginx Ingress"
  type: string
  default: "true"
  env: NGINX_INGRESS

- name: ARGO
  description: "Install/Upgrade Argo"
  type: string
  default: "false"
  env: ARGO

- name: CHARTMUSEUM
  description: "Install/Upgrade ChartMuseum"
  type: string
  default: "true"
  env: CHARTMUSEUM

- name: NGINX_LOCAL_REPO
  description: "Install/Upgrade Nginx Local Repository"
  type: string
  default: "true"
  env: NGINX_LOCAL_REPO

- name: RANCHER
  description: "Install/Upgrade Rancher"
  type: string
  default: "true"
  env: RANCHER

- name: PROMETHEUS_STACK
  description: "Install/Upgrade Prometheus Stack"
  type: string
  #default: "true"
  default: "false"
  env: PROMETHEUS_STACK

- name: METALLB
  description: "Install/Upgrade MetalLB"
  type: string
  default: "false"
  env: METALLB

#- name: IP_ADDRESSES
#  description: "IP address range allocated for MetalLB, e.g. {192.168.20.12-192.168.20.15} or {192.168.20.32/32,192.168.20.47/32}"
#  type: string
#  default: "" # example: {192.168.20.12-192.168.20.15} or {192.168.20.32/32,192.168.20.47/32}
#  env: IP_ADDRESSES

  
images:
  metallb-speaker:
    repository: "docker.io/metallb/speaker"
    tag: "v0.9.3"

  metallb-controller:
    repository: "docker.io/metallb/controller"
    tag: "v0.9.3"

  nvidia-dcgm-exporter:
    repository: "nvidia/dcgm-exporter"
    tag: "1.4.6"

  nginx-ingress-controller:
    repository: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller"
    tag: "0.32.0"

#  prometheus:
#    repository: "quay.io/prometheus/prometheus"
#    tag: "v2.17.2"
#
#  prometheus-node-exporter:
#    repository: "quay.io/prometheus/node-exporter"
#    tag: "v0.18.0"
#
#  prometheus-config-reloader:
#    repository: "quay.io/coreos/prometheus-config-reloader"
#    tag: "v0.31.1"
#
#  prometheus-configmap-reload:
#    repository: "quay.io/coreos/configmap-reload"
#    tag: "v0.0.1"
#
#  prometheus-pushgateway:
#    repository: "prom/pushgateway"
#    tag: "v0.9.1"
#
#  prometheus-operator:
#    repository: "quay.io/coreos/prometheus-operator"
#    tag: "v0.31.1"
#
#  kube-state-metrics:
#    repository: "quay.io/coreos/kube-state-metrics"
#    tag: "v1.7.2"
#
#  rancher-agent:
#    repository: "rancher/rancher-agent"
#    tag: "v2.3.5"
#
  rancher:
    repository: "rancher/rancher"
    tag: "v2.3.5"

  busybox:
    repository: "busybox"
    tag: "latest"

  cert-manager-cainjector:
    repository: "quay.io/jetstack/cert-manager-cainjector"
    tag: "v0.12.0"

  cert-manager-controller:
    repository: "quay.io/jetstack/cert-manager-controller"
    tag: "v0.12.0"

  cert-manager-webhook:
    repository: "quay.io/jetstack/cert-manager-webhook"
    tag: "v0.12.0"

  kube-webhook-certgen:
    repository: "jettech/kube-webhook-certgen"
    tag: "v1.0.0"

  chartmuseum:
    repository: "chartmuseum/chartmuseum"
    tag: "v0.11.0"

  local-path-provisioner:
    repository: "rancher/local-path-provisioner"
    tag: "v0.0.12"

  defaultbackend-amd64:
    repository: "k8s.gcr.io/defaultbackend-amd64"
    tag: "1.5"

  nginx:
    repository: "nginx"
    tag: "1.17.3-alpine"

  grafana-plugins:
    repository: "gcr.io/rec-repo/grafana-plugins"
    tag: "master"

  kiwigrid-k8s-sidecar:
    repository: "kiwigrid/k8s-sidecar"
    tag: "0.1.20"

  grafana:
    repository: "grafana/grafana"
    tag: "6.3.5"

  squareup-ghostunnel:
    repository: "squareup/ghostunnel"
    tag: "v1.4.1"

  debian-tall:
    repository: "quay.io/gravitational/debian-tall"
    tag: "stretch"

  container-tools:
    repository: "gcr.io/rec-repo/container-tools"
    tag: "1.0.7"


install:
  - exec:
      description: "Install Infra Charts"
      command: ./install.sh

#upgrade:
#  - exec:
#      description: "World 2.0"
#      command: ./helpers.sh
#      arguments:
#        - upgrade

uninstall:
  - exec:
      description: "Uninstall Infra Charts"
      command: ./uninstall.sh

# See https://porter.sh/wiring/#credentials
#credentials:
#  - name: kubeconfig
#    path: /root/.kube/config

@digger18
Copy link
Author

digger18 commented Aug 28, 2020

To double-check, does cnab-to-oci push CNAB_PATH --target 172.31.33.xxx/library/REPOSITORY[:TAG] --auto-update-bundle work when the bundle.json generated by Porter is supplied? (It should exist at .cnab/bundle.json in the working directory of the Porter bundle, after porter build is run.)

Just tested, it does not work either.
Exact same error.

When adding --insecure-registries 172.31.33.128:80 to cnab-to-oci push it actually works with the porter generated bundle.json:

# cnab-to-oci push .cnab/bundle.json --insecure-registries 172.31.33.128:80 --target 172.31.33.128:80/library/cnab-test:1.0.0 --auto-update-bundle
Starting to copy image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0...
Completed image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0 copy
Starting to copy image gcr.io/rec-repo/grafana-plugins:master...
Completed image gcr.io/rec-repo/grafana-plugins:master copy
Starting to copy image kiwigrid/k8s-sidecar:0.1.20...
Completed image kiwigrid/k8s-sidecar:0.1.20 copy
Starting to copy image quay.io/jetstack/cert-manager-controller:v0.12.0...
Completed image quay.io/jetstack/cert-manager-controller:v0.12.0 copy
Starting to copy image gcr.io/rec-repo/container-tools:1.0.7...
Completed image gcr.io/rec-repo/container-tools:1.0.7 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image grafana/grafana:6.3.5...
Completed image grafana/grafana:6.3.5 copy
Starting to copy image rancher/local-path-provisioner:v0.0.12...
Completed image rancher/local-path-provisioner:v0.0.12 copy
Starting to copy image rancher/rancher:v2.3.5...
Completed image rancher/rancher:v2.3.5 copy
Starting to copy image quay.io/jetstack/cert-manager-cainjector:v0.12.0...
Completed image quay.io/jetstack/cert-manager-cainjector:v0.12.0 copy
Starting to copy image quay.io/jetstack/cert-manager-webhook:v0.12.0...
Completed image quay.io/jetstack/cert-manager-webhook:v0.12.0 copy
Starting to copy image k8s.gcr.io/defaultbackend-amd64:1.5...
Completed image k8s.gcr.io/defaultbackend-amd64:1.5 copy
Starting to copy image nginx:1.17.3-alpine...
Completed image nginx:1.17.3-alpine copy
Starting to copy image nvidia/dcgm-exporter:1.4.6...
Completed image nvidia/dcgm-exporter:1.4.6 copy
Starting to copy image chartmuseum/chartmuseum:v0.11.0...
Completed image chartmuseum/chartmuseum:v0.11.0 copy
Starting to copy image jettech/kube-webhook-certgen:v1.0.0...
Completed image jettech/kube-webhook-certgen:v1.0.0 copy
Starting to copy image docker.io/metallb/speaker:v0.9.3...
Failed to copy image docker.io/metallb/speaker:v0.9.3: failed commit on ref "layer-sha256:aad63a9339440e7c3e1fff2b988991b9bfb81280042fa7f39a5e327023056819": unexpected size 3073062, expected 2803255
Error: failed commit on ref "layer-sha256:aad63a9339440e7c3e1fff2b988991b9bfb81280042fa7f39a5e327023056819": unexpected size 3073062, expected 2803255

... but fails on other error for a specific image(s) which I believe is related to another issue I opened few days ago (#1224)

My bundle.json file:

{"custom":{"sh.porter":{"manifest":"IyBUaGlzIGlzIHRoZSBjb25maWd1cmF0aW9uIGZvciBQb3J0ZXIKIyBZb3UgbXVzdCBkZWZpbmUgc3RlcHMgZm9yIGVhY2ggYWN0aW9uLCBidXQgdGhlIHJlc3QgaXMgb3B0aW9uYWwKIyBTZWUgaHR0cHM6Ly9wb3J0ZXIuc2gvYXV0aG9yLWJ1bmRsZXMgZm9yIGRvY3VtZW50YXRpb24gb24gaG93IHRvIGNvbmZpZ3VyZSB5b3VyIGJ1bmRsZQojIFVuY29tbWVudCBvdXQgdGhlIHNlY3Rpb25zIGJlbG93IHRvIHRha2UgZnVsbCBhZHZhbnRhZ2Ugb2Ygd2hhdCBQb3J0ZXIgY2FuIGRvIQoKbmFtZTogaW5mcmEtY25hYgp2ZXJzaW9uOiAxLjAuMApkZXNjcmlwdGlvbjogIkluZnJhIENOQUIiCnRhZzogMTcyLjMxLjMzLjEyODo4MC9jbmFiL2luZnJhLWNuYWI6MS4wLjAKCiMgVW5jb21tZW50IHRoZSBsaW5lIGJlbG93IHRvIHVzZSBhIHRlbXBsYXRlIERvY2tlcmZpbGUgZm9yIHlvdXIgaW52b2NhdGlvbiBpbWFnZQojZG9ja2VyZmlsZTogRG9ja2VyZmlsZS50bXBsCgptaXhpbnM6Ci0gZXhlYwotIGhlbG0KCnBhcmFtZXRlcnM6Ci0gbmFtZTogTE9DQUxfUEFUSF9QUk9WSVNJT05FUgogIGRlc2NyaXB0aW9uOiAiSW5zdGFsbC9VcGdyYWRlIExvY2FsIFBhdGggUHJvdmlzaW9uZXIiCiAgdHlwZTogc3RyaW5nCiAgZGVmYXVsdDogInRydWUiCiAgZW52OiBMT0NBTF9QQVRIX1BST1ZJU0lPTkVSCgotIG5hbWU6IENFUlRfTUFOQUdFUgogIGRlc2NyaXB0aW9uOiAiSW5zdGFsbC9VcGdyYWRlIENlcnQtTWFuYWdlciIKICB0eXBlOiBzdHJpbmcKICBkZWZhdWx0OiAidHJ1ZSIKICBlbnY6IENFUlRfTUFOQUdFUgoKLSBuYW1lOiBOR0lOWF9JTkdSRVNTCiAgZGVzY3JpcHRpb246ICJJbnN0YWxsL1VwZ3JhZGUgTmdpbnggSW5ncmVzcyIKICB0eXBlOiBzdHJpbmcKICBkZWZhdWx0OiAidHJ1ZSIKICBlbnY6IE5HSU5YX0lOR1JFU1MKCi0gbmFtZTogQVJHTwogIGRlc2NyaXB0aW9uOiAiSW5zdGFsbC9VcGdyYWRlIEFyZ28iCiAgdHlwZTogc3RyaW5nCiAgZGVmYXVsdDogImZhbHNlIgogIGVudjogQVJHTwoKLSBuYW1lOiBDSEFSVE1VU0VVTQogIGRlc2NyaXB0aW9uOiAiSW5zdGFsbC9VcGdyYWRlIENoYXJ0TXVzZXVtIgogIHR5cGU6IHN0cmluZwogIGRlZmF1bHQ6ICJ0cnVlIgogIGVudjogQ0hBUlRNVVNFVU0KCi0gbmFtZTogTkdJTlhfTE9DQUxfUkVQTwogIGRlc2NyaXB0aW9uOiAiSW5zdGFsbC9VcGdyYWRlIE5naW54IExvY2FsIFJlcG9zaXRvcnkiCiAgdHlwZTogc3RyaW5nCiAgZGVmYXVsdDogInRydWUiCiAgZW52OiBOR0lOWF9MT0NBTF9SRVBPCgotIG5hbWU6IFJBTkNIRVIKICBkZXNjcmlwdGlvbjogIkluc3RhbGwvVXBncmFkZSBSYW5jaGVyIgogIHR5cGU6IHN0cmluZwogIGRlZmF1bHQ6ICJ0cnVlIgogIGVudjogUkFOQ0hFUgoKLSBuYW1lOiBQUk9NRVRIRVVTX1NUQUNLCiAgZGVzY3JpcHRpb246ICJJbnN0YWxsL1VwZ3JhZGUgUHJvbWV0aGV1cyBTdGFjayIKICB0eXBlOiBzdHJpbmcKICAjZGVmYXVsdDogInRydWUiCiAgZGVmYXVsdDogImZhbHNlIgogIGVudjogUFJPTUVUSEVVU19TVEFDSwoKLSBuYW1lOiBNRVRBTExCCiAgZGVzY3JpcHRpb246ICJJbnN0YWxsL1VwZ3JhZGUgTWV0YWxMQiIKICB0eXBlOiBzdHJpbmcKICBkZWZhdWx0OiAiZmFsc2UiCiAgZW52OiBNRVRBTExCCgojLSBuYW1lOiBJUF9BRERSRVNTRVMKIyAgZGVzY3JpcHRpb246ICJJUCBhZGRyZXNzIHJhbmdlIGFsbG9jYXRlZCBmb3IgTWV0YWxMQiwgZS5nLiB7MTkyLjE2OC4yMC4xMi0xOTIuMTY4LjIwLjE1fSBvciB7MTkyLjE2OC4yMC4zMi8zMiwxOTIuMTY4LjIwLjQ3LzMyfSIKIyAgdHlwZTogc3RyaW5nCiMgIGRlZmF1bHQ6ICIiICMgZXhhbXBsZTogezE5Mi4xNjguMjAuMTItMTkyLjE2OC4yMC4xNX0gb3IgezE5Mi4xNjguMjAuMzIvMzIsMTkyLjE2OC4yMC40Ny8zMn0KIyAgZW52OiBJUF9BRERSRVNTRVMKCiAgCmltYWdlczoKICBtZXRhbGxiLXNwZWFrZXI6CiAgICByZXBvc2l0b3J5OiAiZG9ja2VyLmlvL21ldGFsbGIvc3BlYWtlciIKICAgIHRhZzogInYwLjkuMyIKCiAgbWV0YWxsYi1jb250cm9sbGVyOgogICAgcmVwb3NpdG9yeTogImRvY2tlci5pby9tZXRhbGxiL2NvbnRyb2xsZXIiCiAgICB0YWc6ICJ2MC45LjMiCgogIG52aWRpYS1kY2dtLWV4cG9ydGVyOgogICAgcmVwb3NpdG9yeTogIm52aWRpYS9kY2dtLWV4cG9ydGVyIgogICAgdGFnOiAiMS40LjYiCgogIG5naW54LWluZ3Jlc3MtY29udHJvbGxlcjoKICAgIHJlcG9zaXRvcnk6ICJxdWF5LmlvL2t1YmVybmV0ZXMtaW5ncmVzcy1jb250cm9sbGVyL25naW54LWluZ3Jlc3MtY29udHJvbGxlciIKICAgIHRhZzogIjAuMzIuMCIKCiMgIHByb21ldGhldXM6CiMgICAgcmVwb3NpdG9yeTogInF1YXkuaW8vcHJvbWV0aGV1cy9wcm9tZXRoZXVzIgojICAgIHRhZzogInYyLjE3LjIiCiMKIyAgcHJvbWV0aGV1cy1ub2RlLWV4cG9ydGVyOgojICAgIHJlcG9zaXRvcnk6ICJxdWF5LmlvL3Byb21ldGhldXMvbm9kZS1leHBvcnRlciIKIyAgICB0YWc6ICJ2MC4xOC4wIgojCiMgIHByb21ldGhldXMtY29uZmlnLXJlbG9hZGVyOgojICAgIHJlcG9zaXRvcnk6ICJxdWF5LmlvL2NvcmVvcy9wcm9tZXRoZXVzLWNvbmZpZy1yZWxvYWRlciIKIyAgICB0YWc6ICJ2MC4zMS4xIgojCiMgIHByb21ldGhldXMtY29uZmlnbWFwLXJlbG9hZDoKIyAgICByZXBvc2l0b3J5OiAicXVheS5pby9jb3Jlb3MvY29uZmlnbWFwLXJlbG9hZCIKIyAgICB0YWc6ICJ2MC4wLjEiCiMKIyAgcHJvbWV0aGV1cy1wdXNoZ2F0ZXdheToKIyAgICByZXBvc2l0b3J5OiAicHJvbS9wdXNoZ2F0ZXdheSIKIyAgICB0YWc6ICJ2MC45LjEiCiMKIyAgcHJvbWV0aGV1cy1vcGVyYXRvcjoKIyAgICByZXBvc2l0b3J5OiAicXVheS5pby9jb3Jlb3MvcHJvbWV0aGV1cy1vcGVyYXRvciIKIyAgICB0YWc6ICJ2MC4zMS4xIgojCiMgIGt1YmUtc3RhdGUtbWV0cmljczoKIyAgICByZXBvc2l0b3J5OiAicXVheS5pby9jb3Jlb3Mva3ViZS1zdGF0ZS1tZXRyaWNzIgojICAgIHRhZzogInYxLjcuMiIKIwojICByYW5jaGVyLWFnZW50OgojICAgIHJlcG9zaXRvcnk6ICJyYW5jaGVyL3JhbmNoZXItYWdlbnQiCiMgICAgdGFnOiAidjIuMy41IgojCiAgcmFuY2hlcjoKICAgIHJlcG9zaXRvcnk6ICJyYW5jaGVyL3JhbmNoZXIiCiAgICB0YWc6ICJ2Mi4zLjUiCgogIGJ1c3lib3g6CiAgICByZXBvc2l0b3J5OiAiYnVzeWJveCIKICAgIHRhZzogImxhdGVzdCIKCiAgY2VydC1tYW5hZ2VyLWNhaW5qZWN0b3I6CiAgICByZXBvc2l0b3J5OiAicXVheS5pby9qZXRzdGFjay9jZXJ0LW1hbmFnZXItY2FpbmplY3RvciIKICAgIHRhZzogInYwLjEyLjAiCgogIGNlcnQtbWFuYWdlci1jb250cm9sbGVyOgogICAgcmVwb3NpdG9yeTogInF1YXkuaW8vamV0c3RhY2svY2VydC1tYW5hZ2VyLWNvbnRyb2xsZXIiCiAgICB0YWc6ICJ2MC4xMi4wIgoKICBjZXJ0LW1hbmFnZXItd2ViaG9vazoKICAgIHJlcG9zaXRvcnk6ICJxdWF5LmlvL2pldHN0YWNrL2NlcnQtbWFuYWdlci13ZWJob29rIgogICAgdGFnOiAidjAuMTIuMCIKCiAga3ViZS13ZWJob29rLWNlcnRnZW46CiAgICByZXBvc2l0b3J5OiAiamV0dGVjaC9rdWJlLXdlYmhvb2stY2VydGdlbiIKICAgIHRhZzogInYxLjAuMCIKCiAgY2hhcnRtdXNldW06CiAgICByZXBvc2l0b3J5OiAiY2hhcnRtdXNldW0vY2hhcnRtdXNldW0iCiAgICB0YWc6ICJ2MC4xMS4wIgoKICBsb2NhbC1wYXRoLXByb3Zpc2lvbmVyOgogICAgcmVwb3NpdG9yeTogInJhbmNoZXIvbG9jYWwtcGF0aC1wcm92aXNpb25lciIKICAgIHRhZzogInYwLjAuMTIiCgogIGRlZmF1bHRiYWNrZW5kLWFtZDY0OgogICAgcmVwb3NpdG9yeTogIms4cy5nY3IuaW8vZGVmYXVsdGJhY2tlbmQtYW1kNjQiCiAgICB0YWc6ICIxLjUiCgogIG5naW54OgogICAgcmVwb3NpdG9yeTogIm5naW54IgogICAgdGFnOiAiMS4xNy4zLWFscGluZSIKCiAgZ3JhZmFuYS1wbHVnaW5zOgogICAgcmVwb3NpdG9yeTogImdjci5pby9yZWMtcmVwby9ncmFmYW5hLXBsdWdpbnMiCiAgICB0YWc6ICJtYXN0ZXIiCgogIGtpd2lncmlkLWs4cy1zaWRlY2FyOgogICAgcmVwb3NpdG9yeTogImtpd2lncmlkL2s4cy1zaWRlY2FyIgogICAgdGFnOiAiMC4xLjIwIgoKICBncmFmYW5hOgogICAgcmVwb3NpdG9yeTogImdyYWZhbmEvZ3JhZmFuYSIKICAgIHRhZzogIjYuMy41IgoKICBzcXVhcmV1cC1naG9zdHVubmVsOgogICAgcmVwb3NpdG9yeTogInNxdWFyZXVwL2dob3N0dW5uZWwiCiAgICB0YWc6ICJ2MS40LjEiCgogIGRlYmlhbi10YWxsOgogICAgcmVwb3NpdG9yeTogInF1YXkuaW8vZ3Jhdml0YXRpb25hbC9kZWJpYW4tdGFsbCIKICAgIHRhZzogInN0cmV0Y2giCgogIGNvbnRhaW5lci10b29sczoKICAgIHJlcG9zaXRvcnk6ICJnY3IuaW8vcmVjLXJlcG8vY29udGFpbmVyLXRvb2xzIgogICAgdGFnOiAiMS4wLjciCgoKaW5zdGFsbDoKICAtIGV4ZWM6CiAgICAgIGRlc2NyaXB0aW9uOiAiSW5zdGFsbCBJbmZyYSBDaGFydHMiCiAgICAgIGNvbW1hbmQ6IC4vaW5zdGFsbC5zaAoKI3VwZ3JhZGU6CiMgIC0gZXhlYzoKIyAgICAgIGRlc2NyaXB0aW9uOiAiV29ybGQgMi4wIgojICAgICAgY29tbWFuZDogLi9oZWxwZXJzLnNoCiMgICAgICBhcmd1bWVudHM6CiMgICAgICAgIC0gdXBncmFkZQoKdW5pbnN0YWxsOgogIC0gZXhlYzoKICAgICAgZGVzY3JpcHRpb246ICJVbmluc3RhbGwgSW5mcmEgQ2hhcnRzIgogICAgICBjb21tYW5kOiAuL3VuaW5zdGFsbC5zaAoKIyBTZWUgaHR0cHM6Ly9wb3J0ZXIuc2gvd2lyaW5nLyNjcmVkZW50aWFscwojY3JlZGVudGlhbHM6CiMgIC0gbmFtZToga3ViZWNvbmZpZwojICAgIHBhdGg6IC9yb290Ly5rdWJlL2NvbmZpZwo=","manifestDigest":"4c3eab818cd850fa082ba2ee9857b9d2b976e9125954adaad290b475202d6072","mixins":{"exec":{},"helm":{}}}},"definitions":{"ARGO-parameter":{"default":"false","description":"Install/Upgrade Argo","type":"string"},"CERT_MANAGER-parameter":{"default":"true","description":"Install/Upgrade Cert-Manager","type":"string"},"CHARTMUSEUM-parameter":{"default":"true","description":"Install/Upgrade ChartMuseum","type":"string"},"LOCAL_PATH_PROVISIONER-parameter":{"default":"true","description":"Install/Upgrade Local Path Provisioner","type":"string"},"METALLB-parameter":{"default":"false","description":"Install/Upgrade MetalLB","type":"string"},"NGINX_INGRESS-parameter":{"default":"true","description":"Install/Upgrade Nginx Ingress","type":"string"},"NGINX_LOCAL_REPO-parameter":{"default":"true","description":"Install/Upgrade Nginx Local Repository","type":"string"},"PROMETHEUS_STACK-parameter":{"default":"false","description":"Install/Upgrade Prometheus Stack","type":"string"},"RANCHER-parameter":{"default":"true","description":"Install/Upgrade Rancher","type":"string"},"porter-debug-parameter":{"$comment":"porter-internal","$id":"https://porter.sh/generated-bundle/#porter-debug","default":false,"description":"Print debug information from Porter when executing the bundle","type":"boolean"}},"description":"Infra CNAB","images":{"busybox":{"description":"","image":"busybox:latest","imageType":"docker"},"cert-manager-cainjector":{"description":"","image":"quay.io/jetstack/cert-manager-cainjector:v0.12.0","imageType":"docker"},"cert-manager-controller":{"description":"","image":"quay.io/jetstack/cert-manager-controller:v0.12.0","imageType":"docker"},"cert-manager-webhook":{"description":"","image":"quay.io/jetstack/cert-manager-webhook:v0.12.0","imageType":"docker"},"chartmuseum":{"description":"","image":"chartmuseum/chartmuseum:v0.11.0","imageType":"docker"},"container-tools":{"description":"","image":"gcr.io/rec-repo/container-tools:1.0.7","imageType":"docker"},"debian-tall":{"description":"","image":"quay.io/gravitational/debian-tall:stretch","imageType":"docker"},"defaultbackend-amd64":{"description":"","image":"k8s.gcr.io/defaultbackend-amd64:1.5","imageType":"docker"},"grafana":{"description":"","image":"grafana/grafana:6.3.5","imageType":"docker"},"grafana-plugins":{"description":"","image":"gcr.io/rec-repo/grafana-plugins:master","imageType":"docker"},"kiwigrid-k8s-sidecar":{"description":"","image":"kiwigrid/k8s-sidecar:0.1.20","imageType":"docker"},"kube-webhook-certgen":{"description":"","image":"jettech/kube-webhook-certgen:v1.0.0","imageType":"docker"},"local-path-provisioner":{"description":"","image":"rancher/local-path-provisioner:v0.0.12","imageType":"docker"},"metallb-controller":{"description":"","image":"docker.io/metallb/controller:v0.9.3","imageType":"docker"},"metallb-speaker":{"description":"","image":"docker.io/metallb/speaker:v0.9.3","imageType":"docker"},"nginx":{"description":"","image":"nginx:1.17.3-alpine","imageType":"docker"},"nginx-ingress-controller":{"description":"","image":"quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0","imageType":"docker"},"nvidia-dcgm-exporter":{"description":"","image":"nvidia/dcgm-exporter:1.4.6","imageType":"docker"},"rancher":{"description":"","image":"rancher/rancher:v2.3.5","imageType":"docker"},"squareup-ghostunnel":{"description":"","image":"squareup/ghostunnel:v1.4.1","imageType":"docker"}},"invocationImages":[{"image":"172.31.33.128:80/cnab/infra-cnab-installer:1.0.0","imageType":"docker"}],"name":"infra-cnab","parameters":{"ARGO":{"definition":"ARGO-parameter","description":"Install/Upgrade Argo","destination":{"env":"ARGO"}},"CERT_MANAGER":{"definition":"CERT_MANAGER-parameter","description":"Install/Upgrade Cert-Manager","destination":{"env":"CERT_MANAGER"}},"CHARTMUSEUM":{"definition":"CHARTMUSEUM-parameter","description":"Install/Upgrade ChartMuseum","destination":{"env":"CHARTMUSEUM"}},"LOCAL_PATH_PROVISIONER":{"definition":"LOCAL_PATH_PROVISIONER-parameter","description":"Install/Upgrade Local Path Provisioner","destination":{"env":"LOCAL_PATH_PROVISIONER"}},"METALLB":{"definition":"METALLB-parameter","description":"Install/Upgrade MetalLB","destination":{"env":"METALLB"}},"NGINX_INGRESS":{"definition":"NGINX_INGRESS-parameter","description":"Install/Upgrade Nginx Ingress","destination":{"env":"NGINX_INGRESS"}},"NGINX_LOCAL_REPO":{"definition":"NGINX_LOCAL_REPO-parameter","description":"Install/Upgrade Nginx Local Repository","destination":{"env":"NGINX_LOCAL_REPO"}},"PROMETHEUS_STACK":{"definition":"PROMETHEUS_STACK-parameter","description":"Install/Upgrade Prometheus Stack","destination":{"env":"PROMETHEUS_STACK"}},"RANCHER":{"definition":"RANCHER-parameter","description":"Install/Upgrade Rancher","destination":{"env":"RANCHER"}},"porter-debug":{"definition":"porter-debug-parameter","description":"Print debug information from Porter when executing the bundle","destination":{"env":"PORTER_DEBUG"}}},"schemaVersion":"v1.0.0","version":"1.0.0"}

@digger18
Copy link
Author

Oh wait, I did notice the invocation image name is: 172.31.33.xxx/library/infra-cnab-installer:0.0.1. It wouldn't be an official library image, right? I wonder if this is the cause of failure. Can the bundle tag value in the Porter manifest be provided? I wonder if somehow library is mistakenly replacing an org/domain value you've supplied.

I changed library to other project name (cnab), it had no effect.

@vdice
Copy link
Member

vdice commented Aug 28, 2020

@digger18 are you able to get the logs from the Harbor registry when the publish error(s) occur? Any leads there?

@jeffersonbenson
Copy link

Just wanted to add we're seeing this issue as well publishing inside an Openshift 3.11 environment to an insecure docker registry.

@coreyperkins
Copy link

@jeffersonbenson here's my output

I just ran into this one today using version 0.28.1. I reverted back to 0.27.0 and the problem went away.

Extracting bundle from archive /home/ocpadmin/my/my.tgz...
Beginning bundle publish to :5000/my/my-cnab:0.1.0. This may take some time.
Starting to copy image :5000/my/my-porter-installer@sha256:f12da10ce3f308808c939f87d754cd5bfa7223cc3f806471baa077b933a09e57...
Failed to copy image :5000/my/my-installer@sha256:f12da10ce3f308808c939f87d754cd5bfa7223cc3f806471baa077b933a09e57: failed to resolve or push image for service "InvocationImage"
Error: failed to resolve or push image for service "InvocationImage"

@vdice
Copy link
Member

vdice commented Aug 28, 2020

Thanks for the extra information, all. If anyone is/was able to capture logs from the registry being published to, it'd be much appreciated. Also, I'm curious if using 0.27.2 leads to the same error(s)? edit: it sounds like 0.27.2 works fine

In the meantime, I'll keep attempting to reproduce the issue... looking into spinning up a Harbor registry now.

@digger18
Copy link
Author

I can confirm that downgrading to Porter v0.27.2 solves this issue:

# porter publish --insecure-registry
Pushing CNAB invocation image...
The push refers to repository [172.31.33.128:80/cnab/infra-cnab-installer]
6e350c9f6a7f: Preparing
5ef651ff6703: Preparing
fd450e0398e7: Preparing
076e2fbff553: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
fd450e0398e7: Pushed
8e7363b9dcd4: Layer already exists
4e38024e7e09: Layer already exists
9b7657a76455: Layer already exists
6e350c9f6a7f: Pushed
5ef651ff6703: Pushed
076e2fbff553: Pushed
1.0.0: digest: sha256:d0d130b51fa35a6fbe745cd7e79be90c502777270d7736403607c97b8d31ef54 size: 2214

Rewriting CNAB bundle.json...
Starting to copy image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0...
Completed image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0 copy
Starting to copy image busybox:latest...
Completed image busybox:latest copy
Starting to copy image kiwigrid/k8s-sidecar:0.1.20...
Completed image kiwigrid/k8s-sidecar:0.1.20 copy
Starting to copy image nginx:1.17.3-alpine...
Completed image nginx:1.17.3-alpine copy
Starting to copy image squareup/ghostunnel:v1.4.1...
Completed image squareup/ghostunnel:v1.4.1 copy
Starting to copy image rancher/rancher:v2.3.5...
Completed image rancher/rancher:v2.3.5 copy
Starting to copy image quay.io/jetstack/cert-manager-cainjector:v0.12.0...
Completed image quay.io/jetstack/cert-manager-cainjector:v0.12.0 copy
Starting to copy image chartmuseum/chartmuseum:v0.11.0...
Completed image chartmuseum/chartmuseum:v0.11.0 copy
Starting to copy image k8s.gcr.io/defaultbackend-amd64:1.5...
Completed image k8s.gcr.io/defaultbackend-amd64:1.5 copy
Starting to copy image grafana/grafana:6.3.5...
Completed image grafana/grafana:6.3.5 copy
Starting to copy image gcr.io/rec-repo/grafana-plugins:master...
Completed image gcr.io/rec-repo/grafana-plugins:master copy
Starting to copy image rancher/local-path-provisioner:v0.0.12...
Completed image rancher/local-path-provisioner:v0.0.12 copy
Starting to copy image nvidia/dcgm-exporter:1.4.6...
Completed image nvidia/dcgm-exporter:1.4.6 copy
Starting to copy image quay.io/jetstack/cert-manager-controller:v0.12.0...
Completed image quay.io/jetstack/cert-manager-controller:v0.12.0 copy
Starting to copy image quay.io/jetstack/cert-manager-webhook:v0.12.0...
Completed image quay.io/jetstack/cert-manager-webhook:v0.12.0 copy
Starting to copy image gcr.io/rec-repo/container-tools:1.0.7...
Completed image gcr.io/rec-repo/container-tools:1.0.7 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0...
Completed image quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.32.0 copy
Starting to copy image jettech/kube-webhook-certgen:v1.0.0...
Completed image jettech/kube-webhook-certgen:v1.0.0 copy
Bundle tag 172.31.33.128:80/cnab/infra-cnab:1.0.0 pushed successfully, with digest "sha256:e650842d764c4cd53a81dbcd940f17c47dfa841c3b76410d72f89e3100b76c31"

However, I had to comment-out one image docker.io/metallb/speaker:v0.9.3, because otherwise I am facing a different issue which I think is related to #1224:

# porter publish --insecure-registry
Pushing CNAB invocation image...
The push refers to repository [172.31.33.128:80/cnab/infra-cnab-installer]
a24d18766e7d: Preparing
ab26aade72c5: Preparing
0f7465cfb4e2: Preparing
8a3fdc0430dc: Preparing
219410cd4347: Preparing
90ca8ca0957c: Preparing
8e7363b9dcd4: Preparing
9b7657a76455: Preparing
4e38024e7e09: Preparing
8e7363b9dcd4: Waiting
9b7657a76455: Waiting
4e38024e7e09: Waiting
90ca8ca0957c: Waiting
219410cd4347: Layer already exists
90ca8ca0957c: Layer already exists
8e7363b9dcd4: Layer already exists
a24d18766e7d: Pushed
9b7657a76455: Layer already exists
4e38024e7e09: Layer already exists
0f7465cfb4e2: Pushed
ab26aade72c5: Pushed
8a3fdc0430dc: Pushed
1.0.0: digest: sha256:b35dcd79f7d744fcb8247a3b46a65a435a3faade0e6235806a89b4d1a518e017 size: 2214

Rewriting CNAB bundle.json...
Starting to copy image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0...
Completed image 172.31.33.128:80/cnab/infra-cnab-installer:1.0.0 copy
Starting to copy image gcr.io/rec-repo/container-tools:1.0.7...
Completed image gcr.io/rec-repo/container-tools:1.0.7 copy
Starting to copy image quay.io/gravitational/debian-tall:stretch...
Completed image quay.io/gravitational/debian-tall:stretch copy
Starting to copy image docker.io/metallb/speaker:v0.9.3...
Failed to copy image docker.io/metallb/speaker:v0.9.3: failed to copy: httpReaderSeeker: failed open: failed to do request: context canceled
Error: failed to copy: httpReaderSeeker: failed open: failed to do request: context canceled

@vdice
Copy link
Member

vdice commented Sep 1, 2020

I still hadn't successfully installed Harbor via the Helm chart approach (Any tips? I tried to start w/o TLS, but maybe it'd be best to invest in the time to set it up). All the components appear to deploy successfully and are running, but when I attempt to login I see:

 $ docker login <harbor ip>:80
Username: username
Password:
Error response from daemon: Get http://<harbor ip>:80/v2/: Get https://core.harbor.domain/service/token?account=username&client_id=docker&offline_token=true&service=harbor-registry: Service Unavailable

However, I did experiment w/ the demo instance. I was able to successfully publish a few bundles and am as of yet unable to reproduce an error. However, latency is quite poor with the demo instance, as it appears to be shared with the rest of the public for testing purposes -- and, most importantly, I don't see a way to capture registry server logs if/when errors are encountered.

Anyone with a Harbor instance of their own able to capture registry server logs when error(s) are encountered?

@squillace
Copy link

@digger18 and @jeffersonbenson are you still having this problem? Harbor is darned important and OS well, you betcha

@jeffersonbenson
Copy link

@squillace We seem to have resolved the issue for now. I believe the issue was in how we had our registry configured. Thanks!

@squillace
Copy link

any tips for others that might help before you go poof? If we haven't gotten there yet, a short tl;dr: might be great if you have a moment...

...it's usually a typo...

@jeffersonbenson
Copy link

I guess tl;dr since this is an API call to Docker, ensure that your service fabric isn't having any known networking issues, and possible push to another known good registry (even temporarily) just to make sure it's not an issue with your bundle itself.
All else fails, try an older version of Porter, since it's really easy to shift versions if you're a lay user like myself 😁

@MChorfa
Copy link
Contributor

MChorfa commented Oct 1, 2020

The latest version of harbor works as a charm https://github.com/goharbor/harbor/releases/tag/v2.1.0 . Just follow the installation steps and you should be good to go.
image
image

@carolynvs
Copy link
Member

Sounds like this is resolved. Let me know if it's still a problem and we can investigate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Oops, sorry!
Projects
None yet
Development

No branches or pull requests

8 participants