openEuler builds not stable

[pkking]

Just to track the still failing builds.

+ SERIAL=20221201_19:04
+ distrobuilder --cache-dir /root/build/cache/ --timeout 7200 build-dir image.yaml rootfs -o image.serial=20221201_19:04 -o image.architecture=aarch64 -o image.release=22.03 -o image.variant=default
time="2022-12-01T19:04:09Z" level=info msg="Downloading source"
time="2022-12-01T19:04:20Z" level=info msg="Removing cache directory"
Error: Error while downloading source: Failed to get latest release by 22.03: Failed to read url: Get "https://ru-repo.openeuler.org/": net/http: TLS handshake timeout

[tomponline]

@monstermunchkin does this use http client from shared?

[pkking]

I have talk to openEuler community, ask them to extend more North America mirrors, as this lxc build case, could you please provides some public mirror sites for the openEuler team then they will consider to contact firstly?(maybe mirrors.kernel.org?)

[monstermunchkin]

@monstermunchkin does this use http client from shared?

@tomponline we use an empty httpClient and pass that to shared.DownloadFileHash: https://github.com/lxc/distrobuilder/blob/master/sources/common.go#L137

[monstermunchkin]

@pkking before adding North America mirrors, let's just see if lxc/distrobuilder#681 fixes the issue.

[pkking]

I found the builds on Node: canonical-bos02-amd64-04 works well, is there any aarch64 node in same region, maybe bind openEuler builds on these nodes?

[pkking]

Seems the timeout change has no luck? @monstermunchkin

[pkking]

Maybe switch to docker downloader before mirror issue resolved?

[pkking]

Hello @monstermunchkin, why the aarch64 image still goes no luck :(

[monstermunchkin]

Yeah, it looks like distrobuilder is pulling the amd64 image instead of aarch64. But the docker-companion package doesn't seem to have any architecture configuration so we need to look into that or disable aarch64 builds when using docker as the source.

[pkking]

Seems the dns lookup failed, can you please help to diagnose the error :) @monstermunchkin @tomponline

Error: Error while downloading source: Failed to get latest release by 22.09: Failed to read url: Get "https://mirrors.aliyun.com/openeuler/": dial tcp: lookup mirrors.aliyun.com on 127.0.0.53:53: read udp 127.0.0.1:37481->127.0.0.53:53: i/o timeout

[TommyLike]

@monstermunchkin @tomponline any progress on this issue? Btw Where does these CI machines located? Canada or the US.

[monstermunchkin]

any progress on this issue?

No, not yet. The issues are always the same. Either the mirror is not reachable, or the download is really slow (< 1MB/s). The latter then runs into our set timeout of 2 hours.

Btw Where does these CI machines located?

They're located in Canada.

[TommyLike]

any progress on this issue?

No, not yet. The issues are always the same. Either the mirror is not reachable, or the download is really slow (< 1MB/s). The latter then runs into our set timeout of 2 hours.

Btw Where does these CI machines located?

They're located in Canada.

thanks, @monstermunchkin do you happened to know any repo organize/company who is open to new linux distro 😄 we can sync our repo to Canada locally.

[pkking]

@monstermunchkin @tomponline @TommyLike i make a PR to try to get docker-companion to work with multiarch, could you please have a look? Hope that package wasn't dead

[pkking]

seems the new mirrors works perfect, but i noticed some variants still run in exceptions and can't figure out why from logs, @monstermunchkin @tomponline , any suggestions?

[pkking]

@stgraber Hi, i notice openEuler arm64 build fails and has been disabled, but i can't find the reason, it that a timeout or any suggestion?

[stgraber]

Both of the builds had been stuck for about 3h on some yum call.
Normally I'd say it'd be network related but the arm64 builders are on the exact same network as the amd64 ones, so not sure.

[pkking]

@stgraber Thanks the feedback, do you remember what's the day the problem first comes? Bcs we have a update at Jun 30 but i can see earlier logs.

[pkking]

The change we made i can think of related to the yum/dnf is we switch from a static repo into a metalink service,

The metalink will guide the yum/dnf to the fastest(mostly the nearest) mirror to downloads packages, maybe in Canada the result isn't as expected.

I'd like submit a PR to change the mirror to a static repo and maybe you can help to test it :)

[pkking]

@stgraber Submitted a new PR #772 , could you start a new build and we'll see :)

[stgraber]

Looks like we did get a clean run on intel and arm.