Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Epicli hangs on importing GPG keys for kubernetes repository on RHEL #2115

Closed
9 tasks done
przemyslavic opened this issue Mar 10, 2021 · 5 comments
Closed
9 tasks done

[BUG] Epicli hangs on importing GPG keys for kubernetes repository on RHEL #2115

przemyslavic opened this issue Mar 10, 2021 · 5 comments
Assignees
@to-bar
Labels
area/repository priority/critical Show-stopper! You better start it now type/bug
Milestone
S20210506

Comments

@przemyslavic
Copy link
Collaborator

przemyslavic commented Mar 10, 2021 •
edited
Loading

Describe the bug
There is an issue with importing GPG keys for kubernetes repository on RedHat. Ansible hangs for ~10 hours on download-requirements task and then fails with error Failed to connect to the host via ssh: Shared connection to xx.xx.xx.xx closed."

How to reproduce
Steps to reproduce the behavior:

  1. execute epicli apply (repository component is enough to reproduce)

Expected behavior
The cluster has been deployed successfully.

Config files
If applicable, add config files to help explain your problem.

Environment

  • Cloud provider: [all]
  • OS: [RHEL]

Additional context
Log

Mar 08 15:38:49 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Adding repository: kubernetes
Mar 08 15:38:50 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Importing GPG key 0x836F4BEB:
Mar 08 15:38:50 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Userid     : "gLinux Rapture Automatic Signing Key (//depot/google3/production/borg/cloud-rapture/keys/cloud-rapture-pubkeysMar 08 15:38:50 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Fingerprint: xxxx
Mar 08 15:38:50 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Mar 08 15:38:51 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:51 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:52 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:52 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:53 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:53 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:54 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:54 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:54 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Adding repository: opendistroforelasticsearch
Mar 08 15:38:57 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:57 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Importing GPG key 0xE370325E:
Mar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Userid     : "OpenDistroForElasticsearch (Key For signing OpenDistroForElasticsearch artifacts.) <opendistroforelasticsearchMar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Fingerprint: xxxx
Mar 08 15:38:58 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: From       : https://d3g5vo6xdbdb9a.cloudfront.net/GPG-KEY-opendistroforelasticsearch
Mar 08 15:39:02 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature couldMar 08 15:39:02 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Trying other mirror.
Mar 08 15:39:03 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Importing GPG key 0xA7317B0F:
Mar 08 15:39:03 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Mar 08 15:39:03 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: Fingerprint: xxxx
Mar 08 15:39:03 ci-devazurrhelflannel-repository-vm-0 download-requirements.sh[14949]: From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg

DoD checklist

  • Changelog updated (if affected version was released)
  • COMPONENTS.md updated / doesn't need to be updated
  • Automated tests passed (QA pipelines)
    • apply
    • upgrade
  • Case covered by automated test (if possible) ℹ️ self-tested each time at runtime
  • Idempotency tested
  • Documentation doesn't need to be updated
  • All conversations in PR resolved
@przemyslavic przemyslavic added this to the S20210311 milestone Mar 10, 2021
@to-bar to-bar changed the title [BUG] Epicli hangs on importing GPG keys for kubernetes repository on RedHat [BUG] Epicli hangs on importing GPG keys for kubernetes repository on RHEL Mar 10, 2021
@przemyslavic przemyslavic self-assigned this Mar 12, 2021
@przemyslavic
Copy link
Collaborator Author

✅ The original issue seems to be gone, but this fix brings additional improvements anyway, so it's worth applying.

@mkyc mkyc modified the milestones: S20210311, S20210325 Mar 12, 2021
@mkyc mkyc closed this as completed Mar 16, 2021
@przemyslavic
Copy link
Collaborator Author

The issue is back. It blocks me from RedHat testing.

Mar 29 12:54:10 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: username =
Mar 29 12:54:14 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: docker-ce-stable-patched
Mar 29 12:54:14 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://download.docker.com/linux/centos/gpg
Mar 29 12:54:15 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Checking if 'docker-ce-stable-patched' repo is available
Mar 29 12:54:15 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: elastic-6
Mar 29 12:54:15 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Mar 29 12:54:16 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: elasticsearch-7
Mar 29 12:54:16 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Mar 29 12:54:18 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: elasticsearch-curator-5
Mar 29 12:54:18 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch
Mar 29 12:54:19 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: grafana
Mar 29 12:54:19 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://packages.grafana.com/gpg.key
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Importing GPG key 0x24098CB6:
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Userid     : "Grafana <info@grafana.com>"
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Fingerprint: xxxx
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: From       : https://packages.grafana.com/gpg.key
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Adding repository: kubernetes
Mar 29 12:54:20 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://packages.cloud.google.com/yum/doc/yum-key.gpg
Mar 29 12:54:21 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Executing: rpm --import https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Importing GPG key 0xA7317B0F:
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Fingerprint: xxxx
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature
Mar 29 12:54:22 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Trying other mirror.
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Importing GPG key 0xA7317B0F:
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Fingerprint: xxxx
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature
Mar 29 12:54:23 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Trying other mirror.
Mar 29 12:54:24 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Importing GPG key 0xA7317B0F:
Mar 29 12:54:24 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Userid     : "Google Cloud Packages Automatic Signing Key <gc-team@google.com>"
Mar 29 12:54:24 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: Fingerprint: xxxx
Mar 29 12:54:24 ip-10-1-11-136.eu-west-3.compute.internal download-requirements.sh[11124]: From       : https://packages.cloud.google.com/yum/doc/yum-key.gpg

@przemyslavic przemyslavic reopened this Mar 29, 2021
@przemyslavic przemyslavic modified the milestones: S20210325, S20210408 Mar 29, 2021
@rafzei
Copy link
Contributor

rafzei commented Mar 30, 2021

Can be useful:
https://gist.github.com/simbo1905/ba3e8af9a45435db6093aea35c6150e8
https://people.kernel.org/monsieuricon/run-gnupg-2-2-17-on-your-el7-system

@mkyc
Copy link
Contributor

mkyc commented Mar 30, 2021

@plirglo that might be useful for you.

@mkyc mkyc modified the milestones: S20210408, S20210422, S20210506 Apr 8, 2021
@to-bar
Copy link
Contributor

to-bar commented Apr 23, 2021 •
edited
Loading

We are not able to reproduce this issue now so made decision to close it and re-open if occurs again.

This is not fully confirmed but a workaround could be to disable verification of repository metadata for Kubernetes repo (repo_gpgcheck=0) as a fallback.

@to-bar to-bar closed this as completed Apr 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@to-bar to-bar

Labels
area/repository priority/critical Show-stopper! You better start it now type/bug
Projects
None yet
Milestone
S20210506
Development

No branches or pull requests
4 participants
@rafzei @mkyc @przemyslavic @to-bar