ci: install image from github releases #839

jcvenegas · 2018-01-11T15:54:37Z

This PR install a kernel an image from github.

Kernel:
https://github.com/clearcontainers/linux/releases

Image:
https://github.com/clearcontainers/osbuilder/releases

Both are installed using a new script install_asset.sh

install_asset.sh kernel latest 
install_asset.sh image latest

Fixes: #838

Signed-off-by: Jose Carlos Venegas Munoz jose.carlos.venegas.munoz@intel.com

clearcontainersbot · 2018-01-11T16:21:10Z

kubernetes qa-failed 👎

chavafg · 2018-01-11T16:25:24Z

On the k8s execution, got this error:

Install clear-containers image
./install_clear_image.sh: line 32: python: command not found

jcvenegas · 2018-01-11T18:26:18Z

@chavafg added python as installation dependency.

chavafg · 2018-01-11T19:06:06Z

.ci/install_clear_image.sh

@@ -16,34 +16,31 @@



I think we should rename the script to install_clearcontainers_image.sh

chavafg · 2018-01-11T19:11:38Z

.ci/install_clear_image.sh

+repo_name="osbuilder"
+cc_image_releases_url="https://github.com/${repo_owner}/${repo_name}/releases"
+
+function download_image() {


I see that this function has a lot in common with this one:
https://github.com/clearcontainers/tests/blob/master/.ci/install_clearcontainers_kernel.sh#L34

Do you think it would be possible to have a common function called download_component() (or something similar) and add it to the .ci/lib.sh and then just call it here?

Yep, agree with that .

clearcontainersbot · 2018-01-11T20:35:09Z

kubernetes qa-passed 👍

clearcontainersbot · 2018-01-11T23:33:38Z

kubernetes qa-passed 👍

clearcontainersbot · 2018-01-12T00:34:28Z

kubernetes qa-passed 👍

jcvenegas · 2018-01-18T19:44:59Z

.ci/install_asset.sh

 	if [ "${version}" == "latest" ]; then
+		echo "latest_release_info_url $latest_release_info_url" >&2
 		release_json="$(curl -s ${latest_release_info_url})"
 		parse_py='


Reminder to myself, dont use github api to avoid require a token.

:-) Just to clarify - you can use the ABI without a token, but it is then rate limited, and if a user has burnt up their allowed allocation (as one of my CI systems had), then the call will fail :-)

This PR looks like it has indeed fixed the issue on my CI - so, thanks!

clearcontainersbot · 2018-01-24T19:53:56Z

kubernetes qa-failed 👎

clearcontainersbot · 2018-01-25T04:13:08Z

kubernetes qa-passed 👍

jcvenegas · 2018-01-31T15:48:06Z

@chavafg ci is passing with latest image published in osbuilder, please take a look.

@erick0z @jodh-intel could you take a look to this PR?

jodh-intel

Just a few comments.

@klynnrif - could you take a look at the doc changes please?

jodh-intel · 2018-01-31T15:42:02Z

.ci/README.md

+
+## Clear Containers Image for CI ##
+
+The CI environment is configured download the guest OS image from the


"is configured to".

jodh-intel · 2018-01-31T15:43:41Z

.ci/install_asset.sh

+
+repo_owner="clearcontainers"
+#fake repository dir to query kernel and image  version from remote
+fake_repo_dir=$(mktemp -t -d cc-kernel.XXXX)


Nit: cc-asset.XXXX (or maybe just assets.XXXX to make it more portable)?

jodh-intel · 2018-01-31T15:44:06Z

.ci/install_asset.sh

+	exit 1
+}
+
+asset=$1


jodh-intel · 2018-01-31T15:45:23Z

.ci/install_asset.sh

+version="$2"
+[ -z "${asset}" ] && usage
+[ -z "${version}" ] && usage
+readonly runtime_versions_url="https://raw.githubusercontent.com/clearcontainers/runtime/master/versions.txt"


How about specifying the URL as:

repo=runtime : readonly runtime_versions_url="https://raw.githubusercontent.com/${repo_owner}/${repo}/master/versions.txt"

jodh-intel · 2018-01-31T15:46:18Z

.ci/install_asset.sh

+
+resolve_version() {
+	local repo=$1
+	local version=$2


Nit: Safest to double-quote all params (here and in functions below).

jodh-intel · 2018-01-31T15:46:56Z

.ci/install_asset.sh

+	[ -n "${repo}" ] || die "repo not provided"
+
+	if [ "${version}" == "latest" ]; then
+		version=$(get_latest_version "${repo}")


You could simplify this to:

[ "${version}" == "latest" ] && version=$(get_latest_version "${repo}")

jodh-intel · 2018-01-31T15:48:08Z

.ci/install_asset.sh

+			binaries_dir_fmt='image-%s-binaries'
+			;;
+		*)
+			die "unknown asset"


Could you add $asset to the error message.

jodh-intel · 2018-01-31T15:49:01Z

.ci/setup.sh

+"${cidir}/install_asset.sh" "image" "latest"
+
+echo "Install Clear Containers Kernel"
+"${cidir}/install_asset.sh" "kernel" "latest"


Sweet! 😄

clearcontainersbot · 2018-01-31T16:12:24Z

kubernetes qa-passed 👍

erick0z

lgtm

jcvenegas · 2018-01-31T19:08:24Z

@iphutch @klynnrif @rcaballeromx, please take a look to doc changes.

clearcontainersbot · 2018-01-31T19:15:32Z

kubernetes qa-passed 👍

devimc · 2018-01-31T20:01:45Z

lgtm

chavafg · 2018-01-31T21:02:32Z

lgtm

clearcontainersbot · 2018-01-31T21:40:31Z

kubernetes qa-failed 👎

clearcontainersbot · 2018-01-31T22:21:30Z

kubernetes qa-passed 👍

Python is tests depedency make sure is installed. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

clearcontainersbot · 2018-02-01T02:01:02Z

kubernetes qa-failed 👎

clearcontainersbot · 2018-02-01T02:26:26Z

kubernetes qa-failed 👎

clearcontainersbot · 2018-02-01T06:10:04Z

kubernetes qa-failed 👎

clearcontainersbot · 2018-02-01T06:39:45Z

kubernetes qa-failed 👎

Both assets kernel and image installation is similar merge it in only one script. Add documentation about ./install_asset.sh. - Install github image. - Install github kernel. Fixes: clearcontainers#838 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

remove not needed scirpts. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

clearcontainersbot · 2018-02-01T15:40:20Z

kubernetes qa-failed 👎

jcvenegas · 2018-02-01T16:43:42Z

@klynnrif @rcaballeromx ping , could you check this PR, it is ready to merge only need your +1.

klynnrif

lgtm

clearcontainersbot · 2018-02-01T19:09:39Z

kubernetes qa-failed 👎

Also remove installation from fedora and ubuntu scripts and refactor in setup.sh Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

grahamwhaley · 2018-02-02T15:19:27Z

.ci/setup.sh

@@ -38,6 +38,12 @@ if [ -n "$PULL_REQUEST_NUMBER" ] || [ -n "$LOCALCI_PR_NUMBER" ]; then
 	bash -f "${cidir}/run_fetch_branches_tool.sh"
 fi

+echo "Install Clear containers image"
+"${cidir}/install_asset.sh" "image" "latest"


Hi @jcvenegas
Sorry for posting a post-merge follow up. I think this has broken the metrics Jenkins CI instance.
The root problem I think is that on the metrics CI, which is a bare metal system (as opposed to the VM Azure systems we use QA CI) - so we have clear containers installed into docker as the default runtime from the previous PR test (we don't clean/reset the whole system like we effectively do with Azure - where we get a new clean VM machine for each PR).

This results in the known build failure for osbuilder (like clearcontainers/osbuilder#8), like:

17:26:17 Step 2/9 : RUN dnf install -y qemu-img parted gdisk make gcc bc git e2fsprogs libudev-devel pkgconfig 17:26:17 ---> Running in 6220fc9fd574 17:26:25 �[91mBDB0126 mmap: Invalid argument 17:26:25 �[0m�[91mhandle_proxy_response:616:Error response received from proxy at /run/virtcontainers/pods/6220fc9fd574d56d747fe5bcd635ae18ab67dc190fca8e4b97b6a9dda0e08d2f/proxy.sock: {"msg":"vm: unknown token 7yumQLpbAES-kAnPNgsZZswCu_WH7hkFBnhpKa-bMqg="} 17:26:25 17:26:25 /usr/libexec/clear-containers/cc-shim: Shim received an error in responseto ConnectShim command, exiting�[0mThe command '/bin/sh -c dnf install -y qemu-img parted gdisk make gcc bc git e2fsprogs libudev-devel pkgconfig' returned a non-zero code: 1 17:26:25 Makefile:114: recipe for target 'docker-build' failed

from https://clearlinux.org/cc-ci/job/clear-containers-runtime-16.04-PR/66/console

As the metrics CI system runs Ubuntu right now, I suspect we suffer from a combination of clearcontainers/osbuilder#8 and clearcontainers/osbuilder#22.

I can't think of a workaround or solution right now. The only thing that comes to mind is maybe for a system that has CC as the default runtime we will need to use the latest pre-built image or kernel from the repos rather than trying to build the latest from source.

Thoughts?

CI: Fix xurls URL

jcvenegas added the WIP label Jan 11, 2018

chavafg reviewed Jan 11, 2018

View reviewed changes

jcvenegas force-pushed the pull-image-gh branch from 01ee007 to 50486d0 Compare January 11, 2018 23:05

jcvenegas commented Jan 18, 2018

View reviewed changes

jcvenegas force-pushed the pull-image-gh branch from ff5c393 to 1b6b39e Compare January 24, 2018 19:27

jcvenegas force-pushed the pull-image-gh branch from 1b6b39e to b95dfcb Compare January 25, 2018 03:51

jcvenegas mentioned this pull request Jan 25, 2018

release: Add script to publish images on github clearcontainers/packaging#242

Merged

jcvenegas requested review from jodh-intel and erick0z January 31, 2018 15:35

jodh-intel reviewed Jan 31, 2018

View reviewed changes

jcvenegas force-pushed the pull-image-gh branch from b95dfcb to d2cf7ab Compare January 31, 2018 15:50

jcvenegas force-pushed the pull-image-gh branch from d2cf7ab to 70ffaa8 Compare January 31, 2018 18:06

erick0z approved these changes Jan 31, 2018

View reviewed changes

jcvenegas mentioned this pull request Jan 31, 2018

packages: Use osbuilder image instead of Clear Linux clearcontainers/packaging#261

Closed

jcvenegas removed the WIP label Jan 31, 2018

jcvenegas force-pushed the pull-image-gh branch from 70ffaa8 to bc00bd4 Compare January 31, 2018 21:10

jcvenegas force-pushed the pull-image-gh branch from bc00bd4 to c11957b Compare January 31, 2018 21:46

ci: Make sure python is installed

8f92c8e

Python is tests depedency make sure is installed. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas force-pushed the pull-image-gh branch 2 times, most recently from 427cfc1 to c11957b Compare February 1, 2018 01:37

jcvenegas force-pushed the pull-image-gh branch from c11957b to bad5d79 Compare February 1, 2018 05:59

jcvenegas force-pushed the pull-image-gh branch from bad5d79 to b6378f8 Compare February 1, 2018 06:14

jcvenegas added 2 commits February 1, 2018 08:29

ci: remove old install scripts

7ca7cb6

remove not needed scirpts. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas force-pushed the pull-image-gh branch from b6378f8 to 5426b0e Compare February 1, 2018 15:14

klynnrif approved these changes Feb 1, 2018

View reviewed changes

jcvenegas force-pushed the pull-image-gh branch from 5426b0e to f92e2e1 Compare February 1, 2018 18:44

jcvenegas merged commit d9ba61e into clearcontainers:master Feb 1, 2018

ci: Install kernel and image with install_asset.sh

f92e2e1

Also remove installation from fedora and ubuntu scripts and refactor in setup.sh Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

grahamwhaley reviewed Feb 2, 2018

View reviewed changes

mcastelino pushed a commit to mcastelino/tests that referenced this pull request Jan 23, 2019

Merge pull request clearcontainers#839 from jodh-intel/ci-fix-xurls-url

31fe4da

CI: Fix xurls URL


		## Clear Containers Image for CI ##

		The CI environment is configured download the guest OS image from the

ci: install image from github releases #839

ci: install image from github releases #839

Conversation

jcvenegas commented Jan 11, 2018 • edited Loading

clearcontainersbot commented Jan 11, 2018

chavafg commented Jan 11, 2018

jcvenegas commented Jan 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clearcontainersbot commented Jan 11, 2018

clearcontainersbot commented Jan 11, 2018

clearcontainersbot commented Jan 12, 2018

jcvenegas Jan 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clearcontainersbot commented Jan 24, 2018

clearcontainersbot commented Jan 25, 2018

jcvenegas commented Jan 31, 2018

jodh-intel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clearcontainersbot commented Jan 31, 2018

erick0z left a comment

Choose a reason for hiding this comment

jcvenegas commented Jan 31, 2018

clearcontainersbot commented Jan 31, 2018

devimc commented Jan 31, 2018 • edited by chavafg Loading

chavafg commented Jan 31, 2018 • edited Loading

clearcontainersbot commented Jan 31, 2018

clearcontainersbot commented Jan 31, 2018

clearcontainersbot commented Feb 1, 2018

clearcontainersbot commented Feb 1, 2018

clearcontainersbot commented Feb 1, 2018

clearcontainersbot commented Feb 1, 2018

clearcontainersbot commented Feb 1, 2018

jcvenegas commented Feb 1, 2018

klynnrif left a comment

Choose a reason for hiding this comment

clearcontainersbot commented Feb 1, 2018

Choose a reason for hiding this comment

jcvenegas commented Jan 11, 2018 •

edited

Loading

jcvenegas Jan 18, 2018 •

edited

Loading

devimc commented Jan 31, 2018 •

edited by chavafg

Loading

chavafg commented Jan 31, 2018 •

edited

Loading