Stop crossbuilding using Debian 8 #34921

rdner · 2023-03-24T10:57:38Z

What does this PR do?

The GPG keys are expired and the main repository has been moved to the archive. We should start building for the more recent version of Debian.

According to our support matrix https://www.elastic.co/support/matrix we don't support Debian < 10 since 8.4.

Why is it important?

We cannot build snapshots right now:

06:29:52 Status: Downloaded newer image for docker.elastic.co/beats-dev/golang-crossbuild:1.18.10-main-debian8
06:29:58 >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=386, GOARM=, GOOS=linux, PLATFORM_ID=linux-386]
06:29:58 >> Building using: cmd='build/mage-linux-amd64 golangCrossBuild', env=[CC=gcc, CXX=g++, GOARCH=amd64, GOARM=, GOOS=linux, PLATFORM_ID=linux-amd64]
06:30:05 W: GPG error: http://archive.debian.org/ jessie Release: The following signatures were invalid: KEYEXPIRED 1587841717
06:30:05 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-amd64/Packages  404  Not Found [IP: 151.101.194.132 80]
06:30:05 
06:30:05 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-i386/Packages  404  Not Found [IP: 151.101.194.132 80]
06:30:05 
06:30:05 E: Some index files failed to download. They have been ignored, or old ones used instead.
06:30:05 Error: running "apt-get update" failed with exit code 100
06:30:05 Error: failed building for linux/386: exit status 100
06:30:05 failed building for linux/386: exit status 100
06:30:05 W: GPG error: http://archive.debian.org/ jessie Release: The following signatures were invalid: KEYEXPIRED 1587841717
06:30:05 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-amd64/Packages  404  Not Found [IP: 151.101.66.132 80]
06:30:05 
06:30:05 W: Failed to fetch http://security.debian.org/debian-security/dists/jessie/updates/main/binary-i386/Packages  404  Not Found [IP: 151.101.66.132 80]
06:30:05 
06:30:05 E: Some index files failed to download. They have been ignored, or old ones used instead.
06:30:05 Error: running "apt-get update" failed with exit code 100
06:30:05 Error: failed building for linux/amd64: exit status 100
06:30:05 failed building for linux/amd64: exit status 100

https://internal-ci.elastic.co/job/elastic+unified-release+master+snapshot-multijob-7.17/361/display/redirect

How to test locally

In Filebeat run:

mage package

The GPG keys are expired and we should start building for the more recent version of Debian.

elasticmachine · 2023-03-24T10:57:43Z

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

rdner · 2023-03-24T11:11:19Z

elasticsearch-ci/docs failure is a known issue related to https://github.blog/2023-03-23-we-updated-our-rsa-ssh-host-key/

ycombinator · 2023-03-24T11:41:13Z

dev-tools/mage/crossbuild.go

-		// Use an older version of libc to gain greater OS compatibility.
-		// Debian 8 uses glibc 2.19.
-		tagSuffix = "main-debian8"
+		tagSuffix = "main-debian10"


Looks like the main reason for using the older Debian was to build with an older version of libc. Perhaps we could use a newer version of Debian but also downgrade glibc to 2.19 (not sure if this is possible)? If possible, I think the downgrade would happen somewhere in https://github.com/elastic/golang-crossbuild/blob/main/go/base/Dockerfile.tmpl.

If this works #34922 I'm going to close this PR. The risk is high, as you mentioned.

according to our support matrix https://www.elastic.co/support/matrix we don't support Debian < 10 since 8.4, however 7.17 still on Debian 8,9,10.

So, I think we can actually merge this to main and 8.7 but we cannot backport it to 7.17.

cc @andrewkroh , I recall you mentioned something in the past about backward compatibility, do you happen to know whether this PR could cause any issues? Or the underline problem was already solved?

This PR did cause similar problems. I detailed it in #34938 (comment) and I fixed the Packetbeat build. I added tests such that we now have tests in place that check that {File,Audit,Packet,Metric}beat are all compatible with glibc version shipped with RHEL 7.

elasticmachine · 2023-03-24T12:32:06Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-03-24T16:53:08.776+0000
Duration: 11 min 46 sec

❕ Flaky test report

No test was executed to be analysed.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

rdner · 2023-03-24T13:02:48Z

I'll try a safer option first #34922

rdner · 2023-03-24T16:53:07Z

Sorry for extra pings everyone, I changed base on a wrong PR.

rdner · 2023-03-27T08:01:17Z

I tried to use the archived repositories for Debian 8 in #34922 but didn't succeed – the dependencies cannot be resolved for the packages we need.

The only option left for us is to manually find these packages for all required architectures and make a custom image for our CI. I'm not sure it's worth the trouble.

I think we should just switch to Debian 10 for both 8.x and 7.17.

For 8.7 there is no impact, the Beats support matrix already says that we support Debian >= 10 starting with 8.4.

But our 7.17 version still claims support for Debian >= 8. – which we cannot guarantee after switching 7.17 to Debian 10 because of libc compatibility issues in the past.

Debian 8 was released in April 2015 and its end of life was scheduled to be in June 2022.

It's actually surprising that the repositories were available for so long after reaching the end of life.

Debian 10 was released in July 2019 and its end of life is scheduled on June 30th, 2024.

Source https://wiki.debian.org/LTS

cc @pierrehilbert @nimarezainia @cmacknz

The GPG keys are expired and we should start building for the more recent version of Debian. (cherry picked from commit 5e71fe7)

andrewkroh · 2023-03-27T22:09:20Z

According to our support matrix https://www.elastic.co/support/matrix we don't support Debian < 10 since 8.4.

@rdner I see some Debian 8 check marks on the matrix. I'm was looking at the impact of this in #34938 (comment) and we have a problem because some Beats won't run.

nimarezainia · 2023-03-28T03:42:04Z

@andrewkroh thanks for pointing this out. copying @jlind23 also. Unfortunately I think the Metricbeat you highlight above may be erroneous. At 8.0 we removed Debian 8 from the matrix. This should have applied to Metricbeat also. Even though I am struggling to find the email trail for this, I'm certain Metricbeat would have been a part of that decision.

nimarezainia · 2023-03-28T03:48:15Z

@rdner & @pierrehilbert if we can't make this work for 7.17 we can drop support for it in the next 7.17 release and I will make that obvious in a foot note.

rdner · 2023-03-28T11:17:37Z

@nimarezainia @pierrehilbert backporting #34939 into 7.17 should solve it for now.

However, I saw some issues with Packetbeat running on a new version of libc here #34943 (8.x versions)

Not sure how we're going to resolve it and whether it's an issue for 7.17 as well.

Thanks @andrewkroh for bringing more context and for the fix!

nimarezainia · 2023-03-29T03:26:39Z

@rdner so are you saying that 7.17 is good on Debian 8?

rdner · 2023-03-29T10:31:06Z

@nimarezainia I believe so. It's unclear with Packetbeat at the moment but it should be okay.

It's good for now but we should schedule a switch to Debian 10 in near future. Perhaps 7.18?

Check the minimum required glibc on cross-build artifacts. This is a precautionary check to ensure binaries are aligned the current OS support matrix (https://www.elastic.co/support/matrix). Relates #34938 * {packetbeat,x-pack/packetbeat} Use debian9 crossbuild image for linux/386 because the change in elastic/beats##34921 broke support for older operating systems. * Fix golangci-lint warnings pkg.go:230:6: func `updateWithDarwinUniversal` is unused (unused) various: non-wrapping format verb for fmt.Errorf. Use `%w` to format errors (errorlint) import of package `github.com/pkg/errors` is blocked (gomodguard)

nimarezainia · 2023-03-30T01:07:50Z

@rdner 7.18?? :-)

rdner · 2023-03-30T07:54:33Z

@nimarezainia I mean I have no idea if we're going to release another minor 7.x, we just need to plan this drop at some point in the future, I suppose you'd know better when is the best time :)

The GPG keys are expired and we should start building for the more recent version of Debian.

Check the minimum required glibc on cross-build artifacts. This is a precautionary check to ensure binaries are aligned the current OS support matrix (https://www.elastic.co/support/matrix). Relates #34938 * {packetbeat,x-pack/packetbeat} Use debian9 crossbuild image for linux/386 because the change in elastic/beats##34921 broke support for older operating systems. * Fix golangci-lint warnings pkg.go:230:6: func `updateWithDarwinUniversal` is unused (unused) various: non-wrapping format verb for fmt.Errorf. Use `%w` to format errors (errorlint) import of package `github.com/pkg/errors` is blocked (gomodguard)

Stop crossbuilding using Debian 8

0d913fa

The GPG keys are expired and we should start building for the more recent version of Debian.

rdner added Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team backport-7.17 Automated backport to the 7.17 branch with mergify backport-v8.7.0 Automated backport with mergify labels Mar 24, 2023

rdner self-assigned this Mar 24, 2023

rdner requested a review from a team as a code owner March 24, 2023 10:57

rdner requested review from ycombinator and faec and removed request for a team March 24, 2023 10:57

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 24, 2023

Fix typo

aec5f70

belimawr self-requested a review March 24, 2023 11:20

ycombinator reviewed Mar 24, 2023

View reviewed changes

Remove deprecated flag

6d7d238

rdner removed the backport-7.17 Automated backport to the 7.17 branch with mergify label Mar 24, 2023

rdner changed the base branch from main to 7.17 March 24, 2023 16:51

rdner requested review from a team as code owners March 24, 2023 16:51

rdner requested review from blakerouse and removed request for a team March 24, 2023 16:51

rdner removed the request for review from constanca-m March 24, 2023 16:52

rdner requested review from cmacknz and joshdover and removed request for belimawr March 27, 2023 14:48

rdner added the backport-v8.6.0 Automated backport with mergify label Mar 27, 2023

faec approved these changes Mar 27, 2023

View reviewed changes

rdner merged commit 5e71fe7 into elastic:main Mar 27, 2023

rdner deleted the update-crossbuild-debian branch March 27, 2023 16:09

mergify bot pushed a commit that referenced this pull request Mar 27, 2023

Stop crossbuilding using Debian 8 (#34921)

96a1370

The GPG keys are expired and we should start building for the more recent version of Debian. (cherry picked from commit 5e71fe7)

mergify bot pushed a commit that referenced this pull request Mar 27, 2023

Stop crossbuilding using Debian 8 (#34921)

ee3add6

The GPG keys are expired and we should start building for the more recent version of Debian. (cherry picked from commit 5e71fe7)

This was referenced Mar 27, 2023

[8.6](backport #34921) Stop crossbuilding using Debian 8 #34940

Closed

[8.7](backport #34921) Stop crossbuilding using Debian 8 #34941

Closed

andrewkroh mentioned this pull request Mar 27, 2023

Packaging build is failing for 8.7 and main #34938

Closed

rdner mentioned this pull request Apr 18, 2023

[Automation] Bump Golang version to 1.19.8 elastic/golang-crossbuild#282

Merged

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

Stop crossbuilding using Debian 8 (#34921)

52e8ab4

The GPG keys are expired and we should start building for the more recent version of Debian.

v1v mentioned this pull request Sep 4, 2024

Use Debian 12 as the default docker image? elastic/golang-crossbuild#446

Open

v1v mentioned this pull request Sep 17, 2024

Fix test setup to run on ARM instances elastic/fleet-server#3904

Merged

rdner mentioned this pull request Oct 22, 2024

Agentbeat packaging failures: aarch64-linux-gnu/bin/ld.gold: internal error in maybe_apply_stub, at ../../gold/aarch64.cc:5407 #41270

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stop crossbuilding using Debian 8 #34921

Stop crossbuilding using Debian 8 #34921

rdner commented Mar 24, 2023 •

edited

Loading

elasticmachine commented Mar 24, 2023

rdner commented Mar 24, 2023

ycombinator Mar 24, 2023

rdner Mar 24, 2023

rdner Mar 24, 2023

v1v Apr 18, 2023

andrewkroh Apr 18, 2023

elasticmachine commented Mar 24, 2023 •

edited by jenkins-beats-ci bot

Loading

Build stats

rdner commented Mar 24, 2023

rdner commented Mar 24, 2023

rdner commented Mar 27, 2023

andrewkroh commented Mar 27, 2023

nimarezainia commented Mar 28, 2023

nimarezainia commented Mar 28, 2023

rdner commented Mar 28, 2023

nimarezainia commented Mar 29, 2023

rdner commented Mar 29, 2023

nimarezainia commented Mar 30, 2023

rdner commented Mar 30, 2023

Stop crossbuilding using Debian 8 #34921

Stop crossbuilding using Debian 8 #34921

Conversation

rdner commented Mar 24, 2023 • edited Loading

What does this PR do?

Why is it important?

How to test locally

elasticmachine commented Mar 24, 2023

rdner commented Mar 24, 2023

ycombinator Mar 24, 2023

Choose a reason for hiding this comment

rdner Mar 24, 2023

Choose a reason for hiding this comment

rdner Mar 24, 2023

Choose a reason for hiding this comment

v1v Apr 18, 2023

Choose a reason for hiding this comment

andrewkroh Apr 18, 2023

Choose a reason for hiding this comment

elasticmachine commented Mar 24, 2023 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

❕ Flaky test report

🤖 GitHub comments

rdner commented Mar 24, 2023

rdner commented Mar 24, 2023

rdner commented Mar 27, 2023

andrewkroh commented Mar 27, 2023

nimarezainia commented Mar 28, 2023

nimarezainia commented Mar 28, 2023

rdner commented Mar 28, 2023

nimarezainia commented Mar 29, 2023

rdner commented Mar 29, 2023

nimarezainia commented Mar 30, 2023

rdner commented Mar 30, 2023

rdner commented Mar 24, 2023 •

edited

Loading

elasticmachine commented Mar 24, 2023 •

edited by jenkins-beats-ci bot

Loading