Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

device provision fails to unpack image for Meles & LPi4A #128

Closed
KevinMX opened this issue Apr 9, 2024 · 1 comment · Fixed by #130
Closed

device provision fails to unpack image for Meles & LPi4A #128

KevinMX opened this issue Apr 9, 2024 · 1 comment · Fixed by #130
Labels
area: ruyipkg RuyiSDK Package Management issues bug Something isn't working
Milestone
0.9

Comments

@KevinMX
Copy link

KevinMX commented Apr 9, 2024 •
edited
Loading 

mx @ Phony in ~ |15:37:30  
$ RUYI_DEBUG=X ruyi device provision
debug: [2024-04-09T15:37:32.361209] __file__ = /home/mx/.cache/ruyi/progcache/0.8.0/__main__.py, sys.executable =                          __main__.py:47
                                    /home/mx/.cache/ruyi/progcache/0.8.0/python3, __compiled__ = __nuitka_version__(major=2, minor=1,                    
                                    micro=4, releaselevel='release', containing_dir='/usr/bin', standalone=True, onefile=True,                           
                                    macos_bundle_mode=False, no_asserts=False, no_docstrings=False, no_annotations=False, module=False)                  
debug: [2024-04-09T15:37:32.362643] argv[0] = ruyi, self_exe = /usr/bin/ruyi                                                               __main__.py:54
debug: [2024-04-09T15:37:32.386085] got defaults from system libcrypto libcrypto.so                                                      ssl_patch.py:113
debug: [2024-04-09T15:37:32.386784] X509_get_default_cert_file_env() = SSL_CERT_FILE                                                     ssl_patch.py:114
debug: [2024-04-09T15:37:32.387306] X509_get_default_cert_file() = /usr/lib/ssl/cert.pem                                                 ssl_patch.py:115
debug: [2024-04-09T15:37:32.387856] X509_get_default_cert_dir_env() = SSL_CERT_DIR                                                       ssl_patch.py:116
debug: [2024-04-09T15:37:32.388509] X509_get_default_cert_dir() = /usr/lib/ssl/certs                                                     ssl_patch.py:117
debug: [2024-04-09T15:37:32.473073] PyPI-style version of ruyi: 0.8.0                                                                       version.py:36
debug: [2024-04-09T15:37:32.473798] epoch 0 pre None post None                                                                              version.py:19
debug: [2024-04-09T15:37:32.477866] args=Namespace(func=<compiled_function cli_device_provision at 0x733083e5c640>)                       __init__.py:295

RuyiSDK Device Provisioning Wizard

This is a wizard intended to help you install a system on your device for your
development pleasure, all with ease.

You will be asked some questions that help RuyiSDK understand your device and
your intended configuration, then packages will be downloaded and flashed onto
the device's storage, that you should somehow make available on this host
system beforehand.

Note that, as Ruyi does not run as root, but raw disk access is most likely
required to flash images, you should arrange to allow your user account sudo
access to necessary commands such as dd. Flashing will fail if the sudo
configuration does not allow so.

Continue? (y/N) y

The following devices are currently supported by the wizard. Please pick your device:

  1. Allwinner Nezha D1
  2. Canaan Kendryte K230
  3. Milk-V Duo
  4. Milk-V Duo S
  5. Milk-V Mars
  6. Milk-V Mars CM
  7. Milk-V Meles
  8. Milk-V Pioneer Box
  9. Milk-V Vega
  10. SiFive HiFive Unmatched
  11. Sipeed Lichee RV
  12. Sipeed LicheePi 4A
  13. StarFive VisionFive
  14. StarFive VisionFive2

Choice? (1-14) 7

The device has the following variants. Please choose the one corresponding to your hardware at hand:

  1. Milk-V Meles (4G RAM)
  2. Milk-V Meles (8G RAM)

Choice? (1-2) 1

The following system configurations are supported by the device variant you have chosen. Please pick the one you want to put on the device:

  1. RevyOS for Milk-V Meles (4G RAM)

Choice? (1-1) 1
debug: [2024-04-09T15:37:35.101989] provisioning device variant 'milkv-meles@4g'                                                     provision_cli.py:121

We are about to download and install the following packages for your device:

 * board-image/revyos-milkv-meles
 * board-image/uboot-revyos-milkv-meles-4g

Proceed? (y/N) y
debug: [2024-04-09T15:37:35.942642] about to install for host linux/x86_64: {'board-image/revyos-milkv-meles',                             pkg_cli.py:220
                                    'board-image/uboot-revyos-milkv-meles-4g'}                                                                           
debug: [2024-04-09T15:37:35.952232] checking /home/mx/.cache/ruyi/distfiles/boot-milkv-meles-1.0.0.ext4                                    distfile.py:44
debug: [2024-04-09T15:37:35.952863] file /home/mx/.cache/ruyi/distfiles/boot-milkv-meles-1.0.0.ext4 not existent                           distfile.py:48
info: downloading https://github.com/milkv-meles/meles-images/releases/download/v1.0.0/boot.ext4 to 
/home/mx/.cache/ruyi/distfiles/boot-milkv-meles-1.0.0.ext4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
100  100M  100  100M    0     0  7104k      0  0:00:14  0:00:14 --:--:-- 13.3M
info: extracting boot-milkv-meles-1.0.0.ext4 for package revyos-milkv-meles-1.0.0
debug: [2024-04-09T15:37:50.761703] checking /home/mx/.cache/ruyi/distfiles/root-meles-20231210_134926.ext4.tar.gz                         distfile.py:44
debug: [2024-04-09T15:37:50.762475] file /home/mx/.cache/ruyi/distfiles/root-meles-20231210_134926.ext4.tar.gz not existent                distfile.py:48
info: downloading https://github.com/milkv-meles/meles-images/releases/download/v1.0.0/root-meles-20231210_134926.ext4.tar.gz to 
/home/mx/.cache/ruyi/distfiles/root-meles-20231210_134926.ext4.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:04 --:--:--     0
100 1397M  100 1397M    0     0  25.4M      0  0:00:54  0:00:54 --:--:-- 32.2M
info: extracting root-meles-20231210_134926.ext4.tar.gz for package revyos-milkv-meles-1.0.0
debug: [2024-04-09T15:38:50.657875] about to call tar: argv=['tar', '-x', '-z', '-f',                                                       unpack.py:144
                                    '/home/mx/.cache/ruyi/distfiles/root-meles-20231210_134926.ext4.tar.gz', '--strip-components=1', '-C',               
                                    '/home/mx/.local/share/ruyi/blobs/.revyos-milkv-meles-1.0.0.tmpgwk_55y6']                                            
Traceback (most recent call last):
  File "/home/mx/.cache/ruyi/progcache/0.8.0/__main__.py", line 57, in <module>
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/cli/__init__.py", line 300, in main
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 30, in cli_device_provision
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 110, in do_provision_interactive
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 149, in do_provision_combo_interactive
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/pkg_cli.py", line 244, in do_install_atoms
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/pkg_cli.py", line 365, in do_install_blob_pkg
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/distfile.py", line 104, in unpack_or_symlink
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/unpack.py", line 80, in do_unpack_or_symlink
  File "/home/mx/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/unpack.py", line 30, in do_unpack
OSError: [Errno 39] Directory not empty: '/home/mx/.local/share/ruyi/blobs/.revyos-milkv-meles-1.0.0.tmpgwk_55y6' -> '/home/mx/.local/share/ruyi/blobs/revyos-milkv-meles-1.0.0'

PS: The image is okay when I tried to manually unpack using tar xvf.

Reproduced on: Ubuntu 22.04.4 LTS & Arch Linux x86_64
@weilinfox
Copy link
Contributor

貌似多个文件的镜像都会有类似的问题

$ RUYI_DEBUG=x ./ruyi device provision
debug: [2024-04-09T16:02:35.647820] __file__ = /home/hachi/.cache/ruyi/progcache/0.8.0/__main__.py, sys.executable = /home/hachi/.cache/ruyi/progcache/0.8.0/python3, __compiled__ = __nuitka_version__(major=2, minor=1,     __main__.py:47
                                    micro=4, releaselevel='release', containing_dir='/home/hachi/ruyisdk/ruyi-mugen', standalone=True, onefile=True, macos_bundle_mode=False, no_asserts=False, no_docstrings=False,                        
                                    no_annotations=False, module=False)                                                                                                                                                                     
debug: [2024-04-09T16:02:35.649046] argv[0] = ./ruyi, self_exe = /home/hachi/ruyisdk/ruyi-mugen/ruyi                                                                                                                          __main__.py:54
debug: [2024-04-09T16:02:35.667331] got defaults from system libcrypto libcrypto.so                                                                                                                                         ssl_patch.py:113
debug: [2024-04-09T16:02:35.667880] X509_get_default_cert_file_env() = SSL_CERT_FILE                                                                                                                                        ssl_patch.py:114
debug: [2024-04-09T16:02:35.668402] X509_get_default_cert_file() = /etc/ssl/cert.pem                                                                                                                                        ssl_patch.py:115
debug: [2024-04-09T16:02:35.668953] X509_get_default_cert_dir_env() = SSL_CERT_DIR                                                                                                                                          ssl_patch.py:116
debug: [2024-04-09T16:02:35.669447] X509_get_default_cert_dir() = /etc/ssl/certs                                                                                                                                            ssl_patch.py:117
debug: [2024-04-09T16:02:35.669993] get_default_verify_paths() values differ between bundled and system libssl                                                                                                               ssl_patch.py:45
debug: [2024-04-09T16:02:35.670610] bundled: DefaultVerifyPaths(cafile=None, capath=None, openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/usr/lib/ssl/cert.pem', openssl_capath_env='SSL_CERT_DIR',                     ssl_patch.py:48
                                    openssl_capath='/usr/lib/ssl/certs')                                                                                                                                                                    
debug: [2024-04-09T16:02:35.671377]  system: DefaultVerifyPaths(cafile='/etc/ssl/cert.pem', capath='/etc/ssl/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/ssl/cert.pem',                                 ssl_patch.py:49
                                    openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/ssl/certs')                                                                                                                                     
debug: [2024-04-09T16:02:35.748763] PyPI-style version of ruyi: 0.8.0                                                                                                                                                          version.py:36
debug: [2024-04-09T16:02:35.749408] epoch 0 pre None post None                                                                                                                                                                 version.py:19
debug: [2024-04-09T16:02:35.753535] args=Namespace(func=<compiled_function cli_device_provision at 0x7f65c1d64740>)                                                                                                          __init__.py:295

RuyiSDK Device Provisioning Wizard

This is a wizard intended to help you install a system on your device for your
development pleasure, all with ease.

You will be asked some questions that help RuyiSDK understand your device and
your intended configuration, then packages will be downloaded and flashed onto
the device's storage, that you should somehow make available on this host
system beforehand.

Note that, as Ruyi does not run as root, but raw disk access is most likely
required to flash images, you should arrange to allow your user account sudo
access to necessary commands such as dd. Flashing will fail if the sudo
configuration does not allow so.

Continue? (y/N) y

The following devices are currently supported by the wizard. Please pick your device:

  1. Allwinner Nezha D1
  2. Canaan Kendryte K230
  3. Milk-V Duo
  4. Milk-V Duo S
  5. Milk-V Mars
  6. Milk-V Mars CM
  7. Milk-V Meles
  8. Milk-V Pioneer Box
  9. Milk-V Vega
  10. SiFive HiFive Unmatched
  11. Sipeed Lichee RV
  12. Sipeed LicheePi 4A
  13. StarFive VisionFive
  14. StarFive VisionFive2

Choice? (1-14) 12

The device has the following variants. Please choose the one corresponding to your hardware at hand:

  1. Sipeed LicheePi 4A (8G RAM)
  2. Sipeed LicheePi 4A (16G RAM)

Choice? (1-2) 1

The following system configurations are supported by the device variant you have chosen. Please pick the one you want to put on the device:

  1. openEuler RISC-V (headless) for Sipeed LicheePi 4A (8G RAM)
  2. openEuler RISC-V (XFCE) for Sipeed LicheePi 4A (8G RAM)
  3. RevyOS for Sipeed LicheePi 4A (8G RAM)

Choice? (1-3) 1
debug: [2024-04-09T16:02:42.216698] provisioning device variant 'sipeed-lpi4a@8g'                                                                                                                                       provision_cli.py:121

We are about to download and install the following packages for your device:

 * board-image/oerv-sipeed-lpi4a-headless
 * board-image/uboot-oerv-sipeed-lpi4a-8g

Proceed? (y/N) y
debug: [2024-04-09T16:02:43.764495] about to install for host linux/x86_64: {'board-image/oerv-sipeed-lpi4a-headless', 'board-image/uboot-oerv-sipeed-lpi4a-8g'}                                                              pkg_cli.py:220
debug: [2024-04-09T16:02:43.783696] checking /home/hachi/.cache/ruyi/distfiles/boot-20231130-224203.ext4.zst                                                                                                                  distfile.py:44
debug: [2024-04-09T16:02:43.784871] file /home/hachi/.cache/ruyi/distfiles/boot-20231130-224203.ext4.zst not existent                                                                                                         distfile.py:48
info: downloading https://mirror.iscas.ac.cn/openeuler-sig-riscv/openEuler-RISC-V/preview/openEuler-23.09-V1-riscv64/lpi4a/boot-20231130-224203.ext4.zst to /home/hachi/.cache/ruyi/distfiles/boot-20231130-224203.ext4.zst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  243M  100  243M    0     0  8416k      0  0:00:29  0:00:29 --:--:-- 8637k
info: extracting boot-20231130-224203.ext4.zst for package oerv-sipeed-lpi4a-headless-0.2309.1
debug: [2024-04-09T16:03:13.868316] about to call zstd: argv=['zstd', '-d', '/home/hachi/.cache/ruyi/distfiles/boot-20231130-224203.ext4.zst', '-o', './boot-20231130-224203.ext4']                                            unpack.py:231
/home/hachi/.cache/ruyi/distfiles/boot-20231130-224203.ext4.zst: 524288000 bytes 
debug: [2024-04-09T16:03:14.163059] checking /home/hachi/.cache/ruyi/distfiles/root-20231130-224203.ext4.zst                                                                                                                  distfile.py:44
debug: [2024-04-09T16:03:14.163699] file /home/hachi/.cache/ruyi/distfiles/root-20231130-224203.ext4.zst not existent                                                                                                         distfile.py:48
info: downloading https://mirror.iscas.ac.cn/openeuler-sig-riscv/openEuler-RISC-V/preview/openEuler-23.09-V1-riscv64/lpi4a/root-20231130-224203.ext4.zst to /home/hachi/.cache/ruyi/distfiles/root-20231130-224203.ext4.zst
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  905M  100  905M    0     0  8682k      0  0:01:46  0:01:46 --:--:-- 9760k
info: extracting root-20231130-224203.ext4.zst for package oerv-sipeed-lpi4a-headless-0.2309.1
debug: [2024-04-09T16:05:02.605330] about to call zstd: argv=['zstd', '-d', '/home/hachi/.cache/ruyi/distfiles/root-20231130-224203.ext4.zst', '-o', './root-20231130-224203.ext4']                                            unpack.py:231
/home/hachi/.cache/ruyi/distfiles/root-20231130-224203.ext4.zst: 4294967296 bytes 
Traceback (most recent call last):
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/__main__.py", line 57, in <module>
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/cli/__init__.py", line 300, in main
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 30, in cli_device_provision
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 110, in do_provision_interactive
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/device/provision_cli.py", line 149, in do_provision_combo_interactive
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/pkg_cli.py", line 244, in do_install_atoms
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/pkg_cli.py", line 365, in do_install_blob_pkg
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/distfile.py", line 104, in unpack_or_symlink
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/unpack.py", line 80, in do_unpack_or_symlink
  File "/home/hachi/.cache/ruyi/progcache/0.8.0/ruyi/ruyipkg/unpack.py", line 30, in do_unpack
OSError: [Errno 39] Directory not empty: '/home/hachi/.local/share/ruyi/blobs/.oerv-sipeed-lpi4a-headless-0.2309.1.tmp0ip1ivbg' -> '/home/hachi/.local/share/ruyi/blobs/oerv-sipeed-lpi4a-headless-0.2309.1'

https://github.com/ruyisdk/ruyi/blob/24a701c18375eb4c8f59ecfb220b036a203e8253/ruyi/ruyipkg/unpack.py#L30C9-L30C38

这里 os.replace 会坏掉

@KevinMX KevinMX changed the title device provision fails to unpack image for Meles device provision fails to unpack image for Meles & LPi4A Apr 9, 2024
xen0n added a commit to xen0n/ruyi that referenced this issue Apr 9, 2024
@xen0n
fix(ruyipkg): move atomic unpacking processing one level up
5906f86 
One package could want to unpack multiple distfiles into a single
install root, so the atomicity must be maintained at package level,
instead of individual distfile level.

This is effectively a rewrite of ruyisdk#124.

Fixes: ruyisdk#128
@xen0n xen0n mentioned this issue Apr 9, 2024
Merged
@xen0n xen0n added this to the 0.9 milestone Apr 9, 2024
@xen0n xen0n added bug Something isn't working area: ruyipkg RuyiSDK Package Management issues labels Apr 9, 2024
@xen0n xen0n closed this as completed in #130 Apr 9, 2024
xen0n added a commit that referenced this issue Apr 9, 2024
@xen0n
fix(ruyipkg): move atomic unpacking processing one level up
607623c 
One package could want to unpack multiple distfiles into a single
install root, so the atomicity must be maintained at package level,
instead of individual distfile level.

This is effectively a rewrite of #124.

Fixes: #128
(cherry picked from commit 5906f86)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area: ruyipkg RuyiSDK Package Management issues bug Something isn't working
Projects
None yet
Milestone
0.9
Development

Successfully merging a pull request may close this issue.

Fix unpacking failure involving packages with multiple distfiles xen0n/ruyi
3 participants
@xen0n @KevinMX @weilinfox