ci: enable upload artifacts #1147

v1v · 2022-02-15T15:14:13Z

What is the problem this PR solves?

Uses elastic/apm-pipeline-library#1549 in order to upload the packages to a Google Storage bucket.

The folder structure is:

<bucket-name>/fleet-server/snapshots/
<bucket-name>/fleet-server/pull-requests/pr-<id>/
<bucket-name>/fleet-server/commit/<sha>/

<sha> is the git commit related to the build and package happened in the CI
pr-<id> is the pull request ID

Test

produced:

mergify · 2022-02-15T15:14:24Z

This pull request does not have a backport label. Could you fix it @v1v? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v/d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2022-02-15T15:29:52Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-02-21T17:19:08.449+0000
Duration: 16 min 12 sec

Test stats 🧪

Test	Results
Failed	0
Passed	260
Skipped	0
Total	260

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

.ci/Jenkinsfile

v1v · 2022-02-15T17:20:53Z

For some reason it copies files to a binary folder:

But it's not doing the same while using the commits folder:

.ci/Jenkinsfile

mergify · 2022-02-16T08:41:00Z

This pull request is now in conflicts. Could you fix it @v1v? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature/packaging-ci upstream/feature/packaging-ci
git merge upstream/main
git push upstream feature/packaging-ci

mdelapenya

I've seen certain differences in how Beats and Fleet Server repo store the artifacts:

While Beats uses the following structure:

📁 beats
- 📁 commits
  - 📁 SHA1
    - 📁 the_beat
      - 📦 [the_beat]-[version]-SNAPSHOT-[os]-[arch][extension], being [os] optional.

fleet-server uses the following structure:

📁 fleet-server
- 📁 commits
  - 📁 SHA1
    - 📁: fleet-server-[version]-[os]-[arch], being [os] always present in the folder name.
      - 📦 fleet-server[extension]

Is that on purpose? I'd have expected having both follow the same structure, with Beats adding an intermediate dir for the_beat

v1v · 2022-02-16T17:32:58Z

Is that on purpose? I'd have expected having both follow the same structure, with Beats adding an intermediate dir for the_beat

I'm a bit confused about this question, IIUC, the concern is regarding the intermediate folder, but fleet-server is not a monorepo, so there is no need to include an extra folder.

The fleet-server build system produces the output that's copied from, the only transformation done in this particular upload process within the CI is related to the repo-name/(commits/sha|snapshots|pull-requests/pr-id)

Is the issue related to the packaging system or the upload? if the later, what's the outcome?

mdelapenya · 2022-02-17T09:46:36Z

Is the issue related to the packaging system or the upload? if the later, what's the outcome?

I believe it's a packaging issue. I'd have expected fleet-server to generate a child file including all variables:

the other way around could also be possible: expecting beats to generate child files with just the name (elastic-agent|elastic-agent.exe)

📁 fleet-server
- 📁 commits
  - 📁 SHA1
    - 📦 fleet-server-[version]-SNAPSHOT-[os]-[arch][extension], being [os] optional.

Furthermore, I realised that the ARM support is different: meanwhile Beats produces files including aarch, fleet-server does not:

v1v · 2022-02-17T10:11:23Z

I believe it's a packaging issue. I'd have expected fleet-server to generate a child file including all variables:

What do we do?

Furthermore, I realised that the ARM support is different: meanwhile Beats produces files including aarch, fleet-server does not:

Infra Release team pushed to use the same aarch extension from 8.0 onwards, but they asked to revert this change:

Packaging: rename arm64 suffix to aarch64 in the tar.gz artifacts ONLY beats#28813 (comment)

We might need to support this :/

mdelapenya · 2022-02-17T15:00:57Z

Do you think it's something on the mage package command?

v1v · 2022-02-17T15:20:07Z

Do you think it's something on the mage package command?

Definitely, this upload is agnostic to the package command but copy files.

See

fleet-server/Makefile

Lines 136 to 154 in c2f9e50

    
           .PHONY: $(PLATFORM_TARGETS) 
        
           $(PLATFORM_TARGETS): release-%: 
        
           	$(eval $@_OS := $(firstword $(subst /, ,$(lastword $(subst release-, ,$@))))) 
        
           	$(eval $@_GO_ARCH := $(lastword $(subst /, ,$(lastword $(subst release-, ,$@))))) 
        
           	$(eval $@_ARCH := $(TARGET_ARCH_$($@_GO_ARCH))) 
        
           	$(eval $@_BUILDMODE:= $(BUILDMODE_$($@_OS)_$($@_GO_ARCH))) 
        
           	GOOS=$($@_OS) GOARCH=$($@_GO_ARCH) go build -ldflags="${LDFLAGS}" $($@_BUILDMODE) -o build/binaries/fleet-server-$(VERSION)-$($@_OS)-$($@_ARCH)/fleet-server . 
        
           	@$(MAKE) OS=$($@_OS) ARCH=$($@_ARCH) package-target 
        
           .PHONY: package-target 
        
           package-target: build/distributions 
        
           ifeq ($(OS),windows) 
        
           	@mv build/binaries/fleet-server-$(VERSION)-$(OS)-$(ARCH)/fleet-server build/binaries/fleet-server-$(VERSION)-$(OS)-$(ARCH)/fleet-server.exe 
        
           	@cd build/binaries && zip -q -r ../distributions/fleet-server-$(VERSION)-$(OS)-$(ARCH).zip fleet-server-$(VERSION)-$(OS)-$(ARCH) 
        
           	@cd build/distributions && shasum -a 512 fleet-server-$(VERSION)-$(OS)-$(ARCH).zip > fleet-server-$(VERSION)-$(OS)-$(ARCH).zip.sha512 
        
           else 
        
           	@tar -C build/binaries -zcf build/distributions/fleet-server-$(VERSION)-$(OS)-$(ARCH).tar.gz fleet-server-$(VERSION)-$(OS)-$(ARCH) 
        
           	@cd build/distributions && shasum -a 512 fleet-server-$(VERSION)-$(OS)-$(ARCH).tar.gz > fleet-server-$(VERSION)-$(OS)-$(ARCH).tar.gz.sha512 
        
           endif

that contains the specific implementation they built.

I don't know if that's something to be changed, but the release might be also affected if so. Maybe it's worth to contact the fleet-server team about this package misleading compared to the elastic-agent/beats?

v1v · 2022-02-21T17:22:52Z

Acceptance Test

gsutil ls -r gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147 | grep '/fleet-server$'
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server-8.2.0-darwin-arm64/fleet-server
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server-8.2.0-darwin-x86_64/fleet-server
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server-8.2.0-linux-arm64/fleet-server
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server-8.2.0-linux-x86/fleet-server
gs://beats-ci-artifacts/fleet-server/pull-requests/pr-1147/fleet-server-8.2.0-linux-x86_64/fleet-server

gsutil ls -r gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de | grep '/fleet-server$'
gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de/fleet-server-8.2.0-darwin-arm64/fleet-server
gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de/fleet-server-8.2.0-darwin-x86_64/fleet-server
gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de/fleet-server-8.2.0-linux-arm64/fleet-server
gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de/fleet-server-8.2.0-linux-x86/fleet-server
gs://beats-ci-artifacts/fleet-server/commits/b19f7c0713f6991c7f6e6ad5cdefad78fe5c37de/fleet-server-8.2.0-linux-x86_64/fleet-server

mdelapenya

With latest changes in elastic/e2e-testing#2172, I think it would be easy to create an edge case for Fleet Server

v1v · 2022-02-28T10:05:05Z

I'll merge this now

(cherry picked from commit 65681a3)

(cherry picked from commit 65681a3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

(cherry picked from commit 65681a3)

(cherry picked from commit 65681a3) # Conflicts: # .ci/Jenkinsfile

(cherry picked from commit 65681a3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

v1v added 2 commits February 15, 2022 15:07

test: if publish works

530551f

ci: enable upload artifacts

b987458

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Feb 15, 2022

v1v changed the title ~~Feature/packaging ci~~ ci: enable upload artifacts Feb 15, 2022

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

use top level folder

499da33

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

v1v and others added 3 commits February 15, 2022 16:00

Update .ci/Jenkinsfile

b56372e

add debug traces

e5fabe0

use path folders

a175257

v1v marked this pull request as ready for review February 15, 2022 16:55

v1v requested a review from a team as a code owner February 15, 2022 16:55

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

9f08506

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

8164dce

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

d10bc78

v1v commented Feb 15, 2022

View reviewed changes

.ci/Jenkinsfile Outdated Show resolved Hide resolved

Update .ci/Jenkinsfile

cc8d683

Merge branch 'main' into feature/packaging-ci

b19f7c0

kuisathaverat approved these changes Feb 16, 2022

View reviewed changes

mdelapenya reviewed Feb 16, 2022

View reviewed changes

Update .ci/Jenkinsfile

04604d5

v1v requested a review from mdelapenya February 22, 2022 17:18

v1v self-assigned this Feb 22, 2022

v1v added backport-v8.0.0 Automated backport with mergify backport-v8.1.0 Automated backport with mergify labels Feb 22, 2022

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Feb 22, 2022

mdelapenya approved these changes Feb 23, 2022

View reviewed changes

v1v merged commit 65681a3 into elastic:main Feb 28, 2022

mergify bot pushed a commit that referenced this pull request Feb 28, 2022

ci: enable upload artifacts (#1147)

572c93e

(cherry picked from commit 65681a3)

mergify bot mentioned this pull request Feb 28, 2022

[8.0](backport #1147) ci: enable upload artifacts #1177

Merged

mergify bot added a commit that referenced this pull request Feb 28, 2022

ci: enable upload artifacts (#1147) (#1177)

0982799

(cherry picked from commit 65681a3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

v1v mentioned this pull request Mar 17, 2022

ci: store build/distributions #1238

Merged

v1v added the backport-7.17 Automated backport to the 7.17 branch with mergify label May 10, 2022

mergify bot pushed a commit that referenced this pull request May 10, 2022

ci: enable upload artifacts (#1147)

21bf1d5

(cherry picked from commit 65681a3)

mergify bot mentioned this pull request May 10, 2022

[8.1](backport #1147) ci: enable upload artifacts #1450

Merged

mergify bot pushed a commit that referenced this pull request May 10, 2022

ci: enable upload artifacts (#1147)

708a17c

(cherry picked from commit 65681a3) # Conflicts: # .ci/Jenkinsfile

mergify bot mentioned this pull request May 10, 2022

[7.17](backport #1147) ci: enable upload artifacts #1451

Merged

mergify bot added a commit that referenced this pull request May 10, 2022

ci: enable upload artifacts (#1147) (#1450)

8bbed21

(cherry picked from commit 65681a3) Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

v1v pushed a commit that referenced this pull request May 10, 2022

[7.17](backport #1147) ci: enable upload artifacts (#1451)

6ed855b

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: enable upload artifacts #1147

ci: enable upload artifacts #1147

v1v commented Feb 15, 2022 •

edited

Loading

mergify bot commented Feb 15, 2022

elasticmachine commented Feb 15, 2022 •

edited

Loading

Build stats

Test stats 🧪

v1v commented Feb 15, 2022

mergify bot commented Feb 16, 2022

mdelapenya left a comment

v1v commented Feb 16, 2022

mdelapenya commented Feb 17, 2022 •

edited

Loading

v1v commented Feb 17, 2022

mdelapenya commented Feb 17, 2022

v1v commented Feb 17, 2022

v1v commented Feb 21, 2022

mdelapenya left a comment

v1v commented Feb 28, 2022

ci: enable upload artifacts #1147

ci: enable upload artifacts #1147

Conversation

v1v commented Feb 15, 2022 • edited Loading

What is the problem this PR solves?

Test

mergify bot commented Feb 15, 2022

elasticmachine commented Feb 15, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

v1v commented Feb 15, 2022

mergify bot commented Feb 16, 2022

mdelapenya left a comment

Choose a reason for hiding this comment

v1v commented Feb 16, 2022

mdelapenya commented Feb 17, 2022 • edited Loading

v1v commented Feb 17, 2022

mdelapenya commented Feb 17, 2022

v1v commented Feb 17, 2022

v1v commented Feb 21, 2022

Acceptance Test

mdelapenya left a comment

Choose a reason for hiding this comment

v1v commented Feb 28, 2022

v1v commented Feb 15, 2022 •

edited

Loading

elasticmachine commented Feb 15, 2022 •

edited

Loading

mdelapenya commented Feb 17, 2022 •

edited

Loading