Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Imager] Segmentation Violation when creating new images #8987

Closed
preeefix opened this issue Jul 7, 2024 · 6 comments · Fixed by #10124
Closed

[Imager] Segmentation Violation when creating new images #8987

preeefix opened this issue Jul 7, 2024 · 6 comments · Fixed by #10124
Assignees

Comments

@preeefix
Copy link

preeefix commented Jul 7, 2024

Bug Report

In the proccess of bringing up a new board on Talos (LibreTech ROC-RK3328-CC), manually building the overlay causes a segmentation violation during imager builds.

Description

A friend of mine has some LibreTech ROC-RK3328-CC boards that he'd like to include into his existing Talos cluster and asked if I could build an image for them.

I started out by using the OrangePi as a base in the siderolabs/sbc-rockchip respository. I managed to get the image to build (available as ghcr.io/preeefix/sbc-rockchip) but when using the included profile.yaml, all I would receive were segmentation violations.

I followed this up by creating a clean overlay via siderolabs/sbc-template, with only the roc-cc-rk3398 build setup (just u-boot), and it again compiles and saves the container, but imager spits out a segmentation violation.

It's likely that I'm missing something, but after searching around I'm unable to determine what's going wrong.

profile.yaml

arch: arm64
platform: metal
secureboot: false
version: v1.7.5
input:
  kernel:
    path: /usr/install/arm64/vmlinuz
  initramfs:
    path: /usr/install/arm64/vmlinuz
  baseInstaller:
    imageRef: ghcr.io/siderolabs/installer:v1.7.5
overlay:
  name: roc-cc-rk3328
  image:
    imageRef: ghcr.io/preeefix/sbc-roc-cc-rk3328:8574bb9-dirty@sha256:2d9b606d4289b95225013d05a512409fe5542d9ce139b07099ccd3eb0a24c6de
output:
  kind: image
  outFormat: .xz

Logs

cat in/profile.yaml | docker run --rm -i -v ./out:/out -v ./in:/in ghcr.io/siderolabs/imager:v1.7.5 -
assembling the finalized profile...
pulling overlay...
    pulling ghcr.io/preeefix/sbc-roc-cc-rk3328:8574bb9-dirty@sha256:2d9b606d4289b95225013d05a512409fe5542d9ce139b07099ccd3eb0a24c6de...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x174ff9b]

goroutine 1 [running]:
github.com/siderolabs/talos/pkg/imager/profile.(*Profile).Validate(0x620fb8?)
        /src/pkg/imager/profile/profile.go:101 +0x17b
github.com/siderolabs/talos/pkg/imager.(*Imager).handleProf(0xc00015a6c8)
        /src/pkg/imager/imager.go:248 +0x37b
github.com/siderolabs/talos/pkg/imager.(*Imager).Execute(0xc00015a6c8, {0x1f1c430, 0xc00022c780}, {0x1c1e8af, 0x4}, 0xc000366180)
        /src/pkg/imager/imager.go:77 +0x1e8
github.com/siderolabs/talos/cmd/installer/cmd/imager.init.func1.1({0x1f1c430, 0xc00022c780})
        /src/cmd/installer/cmd/imager/root.go:187 +0xbbe
github.com/siderolabs/talos/pkg/cli.WithContext({0x1f1c370?, 0x2f6c7c0?}, 0xc00051bc88)
        /src/pkg/cli/context.go:40 +0x1a5
github.com/siderolabs/talos/cmd/installer/cmd/imager.init.func1(0xc000262400?, {0xc000365e50?, 0x4?, 0x1c1e9a3?})
        /src/cmd/installer/cmd/imager/root.go:58 +0x52
github.com/spf13/cobra.(*Command).execute(0x2edd920, {0xc0001b0010, 0x1, 0x1})
        /.cache/mod/github.com/spf13/cobra@v1.8.0/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0x2edd920)
        /.cache/mod/github.com/spf13/cobra@v1.8.0/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        /.cache/mod/github.com/spf13/cobra@v1.8.0/command.go:1039
github.com/siderolabs/talos/cmd/installer/cmd/imager.Execute()
        /src/cmd/installer/cmd/imager/root.go:208 +0x1a
main.main()
        /src/cmd/installer/main.go:19 +0x47

Environment

  • Build Environment: Debian via WSL2, Docker Desktop, x86_64
@smira
Copy link
Member

smira commented Jul 8, 2024

In this particular case, you're generating a disk image, but disk image options are not specified:

output:
  kind: image
  imageOptions:
    diskSize: 8589934592
    diskFormat: raw
  outFormat: .xz

@preeefix
Copy link
Author

Sometimes, rereading the docs would lead to a better. And not filing GH Issues when I'm exhausted.

Ended up getting it to build correctly with the following profile.yaml.

arch: arm64
platform: metal
secureboot: false
version: v1.7.5
overlay:
  name: libretech-roc-rk3328-cc
  image:
    imageRef: ghcr.io/preeefix/sbc-rockchip:0c603d0-dirty
output:
  kind: image
  imageOptions:
    diskSize: 1306525696
    diskFormat: raw
  outFormat: .xz

Though that leads to a discussion that imager doesn't fail in

if p.Output.ImageOptions.DiskSize == 0 {
as part of the validation, though likely because the entire imageOptions block doesn't exist.

@smira
Copy link
Member

smira commented Jul 10, 2024

yep, it's actually a bug that it crashes, as imageOptions is nil in this case

@camrossi
Copy link

camrossi commented Jan 6, 2025

Is there also some limitation on the tag/version format ?

This make it crash:

/ # imager metal --arch arm64 --base-installer-image registry.camsab.me:443/talos/installer:v1.9-6.13-nosbclayer 
skipped pulling overlay (no overlay)
profile ready:
arch: arm64
platform: metal
secureboot: false
version: v1.9-6.13-rc6a
input:
  kernel:
    path: /usr/install/arm64/vmlinuz
  initramfs:
    path: /usr/install/arm64/initramfs.xz
  baseInstaller:
    imageRef: registry.camsab.me:443/talos/installer:v1.9-6.13-nosbclayer
output:
  kind: image
  imageOptions:
    diskSize: 1306525696
    diskFormat: raw
  outFormat: .zst
skipped initramfs rebuild (no system extensions)
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1174968]

then I changed the version: v1.9-6.13-rc6a to version: v1.9.1-6.13a and the crash went away...

@smira
Copy link
Member

smira commented Jan 8, 2025

@camrossi if you could attach a full stack trace, we are happy to fix this

@camrossi
Copy link

camrossi commented Jan 8, 2025

Hi @smira ! Sure thing! This is the output from my GitHub Action pipeline

2025-01-06T07:07:10.8366297Z ##[group]Run /bin/imager metal --arch arm64 --base-installer-image registry.camsab.me:443/talos/installer:v1.9-6.13-rc6a-nosbclayer --system-extension-image registry.camsab.me:443/talos/rk3588:v1.9.1-6.13 --system-extension-image registry.camsab.me:443/talos/mali-gpu-firmware:v1.9.1-6.13 --system-extension-image ghcr.io/siderolabs/util-linux-tools:2.40.2@sha256:5f042318bc3d727bbebd235290971678511738d861b8de9fd4e016cab3be88a4 --system-extension-image ghcr.io/siderolabs/iscsi-tools:v0.1.6@sha256:5adbd9bd67ffa39f65d0e5e534edadc904dd58e7e405ff7509889fd3d006ad18 --overlay-image registry.camsab.me:443/talos/sbc-rockchip:v1.9-6.13-rc6 --overlay-name=turingrk1
2025-01-06T07:07:10.8375229Z �[36;1m/bin/imager metal --arch arm64 --base-installer-image registry.camsab.me:443/talos/installer:v1.9-6.13-rc6a-nosbclayer --system-extension-image registry.camsab.me:443/talos/rk3588:v1.9.1-6.13 --system-extension-image registry.camsab.me:443/talos/mali-gpu-firmware:v1.9.1-6.13 --system-extension-image ghcr.io/siderolabs/util-linux-tools:2.40.2@sha256:5f042318bc3d727bbebd235290971678511738d861b8de9fd4e016cab3be88a4 --system-extension-image ghcr.io/siderolabs/iscsi-tools:v0.1.6@sha256:5adbd9bd67ffa39f65d0e5e534edadc904dd58e7e405ff7509889fd3d006ad18 --overlay-image registry.camsab.me:443/talos/sbc-rockchip:v1.9-6.13-rc6 --overlay-name=turingrk1�[0m
2025-01-06T07:07:10.8380112Z shell: sh -e {0}
2025-01-06T07:07:10.8395131Z env:
2025-01-06T07:07:10.8395933Z   ORIGIN_REPO: siderolabs/talos
2025-01-06T07:07:10.8418504Z   ORIGIN_TAG: v1.9.1
2025-01-06T07:07:10.8419317Z   KERNEL_TAG: v1.9-6.13-rc6
2025-01-06T07:07:10.8419898Z   RK3588_TAG: v1.9.1-6.13
2025-01-06T07:07:10.8420437Z   SBC_TAG: v1.9-6.13-rc6
2025-01-06T07:07:10.8422065Z   PUBLISH_TAG: v1.9-6.13-rc6a
2025-01-06T07:07:10.8422616Z   REGISTRY: registry.camsab.me:443
2025-01-06T07:07:10.8423162Z   USERNAME: talos
2025-01-06T07:07:10.8423938Z   LINUX_TOOLS: 2.40.2@sha256:5f042318bc3d727bbebd235290971678511738d861b8de9fd4e016cab3be88a4
2025-01-06T07:07:10.8425154Z   ISCSI_TOOLS: v0.1.6@sha256:5adbd9bd67ffa39f65d0e5e534edadc904dd58e7e405ff7509889fd3d006ad18
2025-01-06T07:07:10.8426067Z ##[endgroup]
2025-01-06T07:07:10.8482952Z ##[group]Run '/home/runner/k8s/index.js'
2025-01-06T07:07:10.8485043Z shell: /home/runner/externals/node20/bin/node {0}
2025-01-06T07:07:10.8485729Z ##[endgroup]
2025-01-06T07:07:11.5959830Z assembling the finalized profile...
2025-01-06T07:07:11.8320681Z pulling overlay...
2025-01-06T07:07:11.8321501Z     pulling registry.camsab.me:443/talos/sbc-rockchip:v1.9-6.13-rc6...
2025-01-06T07:07:13.9908158Z profile ready:
2025-01-06T07:07:13.9909760Z arch: arm64
2025-01-06T07:07:13.9911349Z platform: metal
2025-01-06T07:07:13.9912802Z secureboot: false
2025-01-06T07:07:13.9914427Z version: v1.9-6.13-rc6a
2025-01-06T07:07:13.9915955Z input:
2025-01-06T07:07:13.9917209Z   kernel:
2025-01-06T07:07:13.9918624Z     path: /usr/install/arm64/vmlinuz
2025-01-06T07:07:13.9920397Z   initramfs:
2025-01-06T07:07:13.9921952Z     path: /usr/install/arm64/initramfs.xz
2025-01-06T07:07:13.9923831Z   baseInstaller:
2025-01-06T07:07:13.9926071Z     imageRef: registry.camsab.me:443/talos/installer:v1.9-6.13-rc6a-nosbclayer
2025-01-06T07:07:13.9928713Z   overlayInstaller:
2025-01-06T07:07:13.9930986Z     imageRef: registry.camsab.me:443/talos/sbc-rockchip:v1.9-6.13-rc6
2025-01-06T07:07:13.9933462Z   systemExtensions:
2025-01-06T07:07:13.9935440Z     - imageRef: registry.camsab.me:443/talos/rk3588:v1.9.1-6.13
2025-01-06T07:07:13.9938517Z     - imageRef: registry.camsab.me:443/talos/mali-gpu-firmware:v1.9.1-6.13
2025-01-06T07:07:13.9943126Z     - imageRef: ghcr.io/siderolabs/util-linux-tools:2.40.2@sha256:5f042318bc3d727bbebd235290971678511738d861b8de9fd4e016cab3be88a4
2025-01-06T07:07:13.9949210Z     - imageRef: ghcr.io/siderolabs/iscsi-tools:v0.1.6@sha256:5adbd9bd67ffa39f65d0e5e534edadc904dd58e7e405ff7509889fd3d006ad18
2025-01-06T07:07:13.9953323Z overlay:
2025-01-06T07:07:13.9954659Z   name: turingrk1
2025-01-06T07:07:13.9956030Z   image:
2025-01-06T07:07:13.9959991Z     imageRef: registry.camsab.me:443/talos/sbc-rockchip:v1.9-6.13-rc6
2025-01-06T07:07:13.9962555Z output:
2025-01-06T07:07:13.9963926Z   kind: image
2025-01-06T07:07:13.9965294Z   imageOptions:
2025-01-06T07:07:13.9966779Z     diskSize: 1306525696
2025-01-06T07:07:13.9968456Z     diskFormat: raw
2025-01-06T07:07:13.9969905Z   outFormat: .zst
2025-01-06T07:07:13.9971571Z rebuilding initramfs with system extensions...
2025-01-06T07:07:13.9974497Z     copying /usr/install/arm64/initramfs.xz to /tmp/imager3472683238/initramfs.xz
2025-01-06T07:07:14.5001518Z rebuilding initramfs with system extensions...
2025-01-06T07:07:14.5004204Z     pulling registry.camsab.me:443/talos/rk3588:v1.9.1-6.13...
2025-01-06T07:07:14.6328149Z rebuilding initramfs with system extensions...
2025-01-06T07:07:14.6331226Z     pulling registry.camsab.me:443/talos/mali-gpu-firmware:v1.9.1-6.13...
2025-01-06T07:07:14.7073018Z rebuilding initramfs with system extensions...
2025-01-06T07:07:14.7075004Z     pulling ghcr.io/siderolabs/util-linux-tools:2.40.2@sha256:5f042318bc3d727bbebd235290971678511738d861b8de9fd4e016cab3be88a4...
2025-01-06T07:07:16.7884675Z rebuilding initramfs with system extensions...
2025-01-06T07:07:16.7888922Z     pulling ghcr.io/siderolabs/iscsi-tools:v0.1.6@sha256:5adbd9bd67ffa39f65d0e5e534edadc904dd58e7e405ff7509889fd3d006ad18...
2025-01-06T07:07:19.2442096Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2442983Z     discovered system extensions:
2025-01-06T07:07:19.2443718Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2444357Z     NAME                VERSION       AUTHOR
2025-01-06T07:07:19.2444978Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2446381Z     rk3588              v1.9.1-6.13   Nico Berlee
2025-01-06T07:07:19.2447067Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2447729Z     mali-gpu-firmware   v1.9.1-6.13   Camillo Rossi
2025-01-06T07:07:19.2448405Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2449082Z     util-linux-tools    2.40.2        Sidero Labs
2025-01-06T07:07:19.2449736Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2450386Z     iscsi-tools         v0.1.6        Sidero Labs
2025-01-06T07:07:19.2451036Z rebuilding initramfs with system extensions...
2025-01-06T07:07:19.2451652Z     validating system extensions
2025-01-06T07:07:19.3845163Z panic: runtime error: invalid memory address or nil pointer dereference
2025-01-06T07:07:19.3848722Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xdabc28]
2025-01-06T07:07:19.3850778Z 
2025-01-06T07:07:19.3851409Z goroutine 1 [running]:
2025-01-06T07:07:19.3853838Z github.com/siderolabs/talos/pkg/machinery/imager/quirks.Quirks.Version(...)
2025-01-06T07:07:19.3856945Z 	/src/pkg/machinery/imager/quirks/quirks.go:34
2025-01-06T07:07:19.3861116Z github.com/siderolabs/talos/pkg/imager/extensions.(*Builder).validateExtensions(0x40003c15b8, {0x40003122c0, 0x4, 0x4000851080?})
2025-01-06T07:07:19.3865730Z 	/src/pkg/imager/extensions/extensions.go:102 +0x68
2025-01-06T07:07:19.3869128Z github.com/siderolabs/talos/pkg/imager/extensions.(*Builder).Build(0x40003c15b8)
2025-01-06T07:07:19.3873020Z 	/src/pkg/imager/extensions/extensions.go:52 +0xc8
2025-01-06T07:07:19.3876934Z github.com/siderolabs/talos/pkg/imager.(*Imager).buildInitramfs(0x40001eec88, {0x1948228, 0x40001fbcc0}, 0x4000538030)
2025-01-06T07:07:19.3881199Z 	/src/pkg/imager/imager.go:313 +0x438
2025-01-06T07:07:19.3890442Z github.com/siderolabs/talos/pkg/imager.(*Imager).Execute(0x40001eec88, {0x1948228, 0x40001fbcc0}, {0x1677129, 0x4}, 0x4000538030)
2025-01-06T07:07:19.3894873Z 	/src/pkg/imager/imager.go:92 +0x1a8
2025-01-06T07:07:19.3898239Z github.com/siderolabs/talos/cmd/installer/cmd/imager.init.func1.1({0x1948228, 0x40001fbcc0})
2025-01-06T07:07:19.3904784Z 	/src/cmd/installer/cmd/imager/root.go:217 +0xbcc
2025-01-06T07:07:19.3907926Z github.com/siderolabs/talos/pkg/cli.WithContext({0x1948140?, 0x295d460?}, 0x40000bdc68)
2025-01-06T07:07:19.3913293Z 	/src/pkg/cli/context.go:40 +0x174
2025-01-06T07:07:19.3920813Z github.com/siderolabs/talos/cmd/installer/cmd/imager.init.func1(0x400021fa00?, {0x400021fe00?, 0x4?, 0x1677221?})
2025-01-06T07:07:19.3924914Z 	/src/cmd/installer/cmd/imager/root.go:62 +0x50
2025-01-06T07:07:19.3931905Z github.com/spf13/cobra.(*Command).execute(0x2909e20, {0x4000148010, 0x10, 0x11})
2025-01-06T07:07:19.3935341Z 	/.cache/mod/github.com/spf13/cobra@v1.8.1/command.go:985 +0x834
2025-01-06T07:07:19.3944147Z github.com/spf13/cobra.(*Command).ExecuteC(0x2909e20)
2025-01-06T07:07:19.3948911Z 	/.cache/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x344
2025-01-06T07:07:19.3959426Z github.com/spf13/cobra.(*Command).Execute(...)
2025-01-06T07:07:19.3966835Z 	/.cache/mod/github.com/spf13/cobra@v1.8.1/command.go:1041
2025-01-06T07:07:19.3969749Z github.com/siderolabs/talos/cmd/installer/cmd/imager.Execute()
2025-01-06T07:07:19.3972292Z 	/src/cmd/installer/cmd/imager/root.go:238 +0x24
2025-01-06T07:07:19.3974258Z main.main()
2025-01-06T07:07:19.3975720Z 	/src/cmd/installer/main.go:19 +0x6c

@smira smira self-assigned this Jan 13, 2025
smira added a commit to smira/talos that referenced this issue Jan 13, 2025
Fixes siderolabs#8987

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
smira added a commit to smira/talos that referenced this issue Jan 16, 2025
Fixes siderolabs#8987

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
(cherry picked from commit 8de1975)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants