Build for linux/arm64 fails

[comfuture]

Description of the bug

Problem

Installing PyMuPDF on linux/arm64 fails while install dependencies on linux/arm64 docker build.
To make multiarch image, I've made a github workflow that builds the image for both amd64 and arm64. The build for amd64 is successful, but the build for arm64 stuck with the following error:

https://github.com/comfuture/glados/actions/runs/9674016697/job/26688926160#step:8:848

#16 37.91   Downloading PyMuPDF-1.24.6.tar.gz (30.4 MB)
#16 39.94      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 30.4/30.4 MB 14.2 MB/s eta 0:00:00
#16 41.66   Getting requirements to build wheel: started
#16 42.61   Getting requirements to build wheel: finished with status 'done'
#16 42.63   Installing backend dependencies: started
#16 57.14   Installing backend dependencies: finished with status 'done'
#16 57.14   Preparing metadata (pyproject.toml): started
#16 117.9   Preparing metadata (pyproject.toml): still running...
            ... (Repeats forever) ...
            ... (Repeats forever) ...

How to reproduce the bug

The project has files as follows:

├──.github
│  └── workflows
│     └── deploy.yaml
├── pymupdf_on_arm64
│   └── __init__.py (empty)
├── main.py
├── Dockerfile
└── pyproject.toml

pyproject.toml is as follows:

[build-system]
requires = ["flit_core >=3.9,<4"]
build-backend = "flit_core.buildapi"

[project]
name = "pymupdf_on_arm64"
requires-python = ">=3.10"

dependencies = [
  "PyMuPDF>=1.24.6",
]

Dockerfile is as follows:

FROM python:3.11

ADD . /app
WORKDIR /app

ENV FLIT_ROOT_INSTALL=1
RUN pip install flit
RUN flit install -s

CMD [ "python", "main.py" ]

The workflow deploy.yaml is as follows:

name: deploy
on:
  push:
    branches:
      - main
jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up QEMU
        uses: docker/setup-qemu-action@v3
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
      - name: build and push
        uses: docker/build-push-action@v5.2.0
        with:
          tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }},ghcr.io/${{ github.repository }}:latest
          context: .
          file: ./Dockerfile
          platforms: linux/amd64,linux/arm64
          push: false
          outputs: type=image,name=target,annotation-index.org.opencontainers.image.description=PyMuPDF Reproduce build error multi-arch image

Then push the project to the repository, and the workflow will be triggered.

PyMuPDF version

1.24.6

Operating system

Linux

Python version

3.11

[julian-smith-artifex-com]

I'm not sure we should assume the build is broken after just [20m] 40m; ARM builds on Github often run very slowly - i've seen PyMuPDF ARM builds take several hours. I think Github may be using an emulator.

Suggest you let it run until Github times out (6h i think).

[vedeht]

My Arm build won't run any version greater than 1.24.5. What do you want us to test so we can give you feedback. I had to update our requirements.txt file to make sure it only uses 1.24.5

[Lanzelot711]

I am running into the same issue on an Mac M1 arm build (local)

[vedeht]

@julian-smith-artifex-com what information are you waiting for? I'm happy to do whatever you need to help resolve this. I am stuck at version 1.24.5 until this is resolved. All our docker containers run on an arm server.

[julian-smith-artifex-com]

@julian-smith-artifex-com what information are you waiting for? I'm happy to do whatever you need to help resolve this. I am stuck at version 1.24.5 until this is resolved. All our docker containers run on an arm server.

The first line of the log at https://github.com/comfuture/glados/actions/runs/9674016697/job/26688926160#step:8:848 is:

2024-06-26T05:22:06.1409863Z Current runner version: '2.317.0'

The end of the log is:

2024-06-26T05:39:00.8320591Z #16 962.0   Preparing metadata (pyproject.toml): still running...
2024-06-26T05:40:01.1951329Z #16 1022.4   Preparing metadata (pyproject.toml): still running...
2024-06-26T05:41:03.5579643Z #16 1084.8   Preparing metadata (pyproject.toml): still running...
2024-06-26T05:42:04.2682315Z #16 1145.5   Preparing metadata (pyproject.toml): still running...
2024-06-26T05:42:08.1229545Z ##[error]The operation was canceled.
2024-06-26T05:42:08.1467241Z Post job cleanup.
...
2024-06-26T05:42:24.2017181Z Cleaning up orphan processes

So it looks like the Github workflow was manually cancelled about 20 min after it started.

So i'd like you to run the build again and leave it to run to completion without cancelling it. [Github will kill it if it hasn't finished after 6 hours]

I'm not familiar with flit. There may be an option to make it show the diagnostics from the underlying build, which would help us to figure out what's happening. With pip one can use -v or -vv for this; maybe there's something similar for flit?

[Lanzelot711]

I think this is independent of flit. I have local poetry as the built system and on version 1.24.5 my example docker image is building in 20 secconds with flag --no-cache while with the 1.24.9 it is stuck forever with preparing metadata as shown in the log.

pyproject.toml

[tool.poetry]
name = "example"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
pymupdf = "1.24.9"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock

# This file is automatically @generated by Poetry 1.8.3 and should not be changed by hand.

[[package]]
name = "pymupdf"
version = "1.24.9"
description = "A high performance Python library for data extraction, analysis, conversion & manipulation of PDF (and other) documents."
optional = false
python-versions = ">=3.8"
files = [
    {file = "PyMuPDF-1.24.9-cp310-none-macosx_10_9_x86_64.whl", hash = "sha256:da5d9699472bfd1de52975de3eb7efaf5190ac5801b9fc6bcccde603afbe6937"},
    {file = "PyMuPDF-1.24.9-cp310-none-macosx_11_0_arm64.whl", hash = "sha256:3d1133983c7ac388a35bbab8dfc4c26a874c05edc47d2038961add2efa4639a8"},
    {file = "PyMuPDF-1.24.9-cp310-none-manylinux2014_x86_64.whl", hash = "sha256:5199567353d1543e6c21c626148f8ac9ebb14ce553f2c434fcb9b00e195e1e52"},
    {file = "PyMuPDF-1.24.9-cp310-none-musllinux_1_2_x86_64.whl", hash = "sha256:c97f0b2fb201c9d9bc0f15a901641174e8896a9ae9fbe0d5bb1a6f2315cc3ced"},
    {file = "PyMuPDF-1.24.9-cp310-none-win32.whl", hash = "sha256:00499b864a56a2168254dce3d0f12048b96e9b3bdd43fecace18a1572342c8d4"},
    {file = "PyMuPDF-1.24.9-cp310-none-win_amd64.whl", hash = "sha256:f074e501e883428e7d5480f732ea6a6bd17146f10ebefb9b84957fd32b79f0d4"},
    {file = "PyMuPDF-1.24.9-cp311-none-macosx_10_9_x86_64.whl", hash = "sha256:caf43ce86790f95049a5849f2802b5c412b865cd368ece89a39a54fc84aa45cd"},
    {file = "PyMuPDF-1.24.9-cp311-none-macosx_11_0_arm64.whl", hash = "sha256:13d06161176e1d4e337f5b5e053b628e4531bab5effb269a83dc38d4deb8e659"},
    {file = "PyMuPDF-1.24.9-cp311-none-manylinux2014_x86_64.whl", hash = "sha256:042ad205c7ef615d9fbab7078f6fa8d14f020ed2dfe3a79d803b6171318565b5"},
    {file = "PyMuPDF-1.24.9-cp311-none-musllinux_1_2_x86_64.whl", hash = "sha256:b4495833bb0300fc885491928f2cbdf96afb569205dcc256bb4c43e3d1fde7cb"},
    {file = "PyMuPDF-1.24.9-cp311-none-win32.whl", hash = "sha256:e53370f3679a7b013c2abb801bb566882dab1fb59646d4b0a717ee0d350c5ab1"},
    {file = "PyMuPDF-1.24.9-cp311-none-win_amd64.whl", hash = "sha256:454932e9c7b9cd3057ee83dfe805f551a1382b9e216e87a32eb44c6d6843f966"},
    {file = "PyMuPDF-1.24.9-cp312-none-macosx_10_9_x86_64.whl", hash = "sha256:93cc4908259f133c9dc88f5e77329c4b2dbc03fca83126b1efffedb67ade0fb9"},
    {file = "PyMuPDF-1.24.9-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:84e1516d4b3e40711b9a6dbaedd30e0a89d6a054ca408a56114ceb5a1461f0d1"},
    {file = "PyMuPDF-1.24.9-cp312-none-manylinux2014_x86_64.whl", hash = "sha256:de8b330900c194efeedeb97adab25520479d101fc9aed50d7323dde08698ae24"},
    {file = "PyMuPDF-1.24.9-cp312-none-musllinux_1_2_x86_64.whl", hash = "sha256:41c92d69993e7614730205b75d7999b21ca0f929d31b2bb86a4b58d3b1b0451a"},
    {file = "PyMuPDF-1.24.9-cp312-none-win32.whl", hash = "sha256:a04af6f3f5f35cb62bc7b3c2e9cfff510aa56c39c53355ecfff40b7cb9773fef"},
    {file = "PyMuPDF-1.24.9-cp312-none-win_amd64.whl", hash = "sha256:e2828a79415ae3dd90c629697ace51db7f1e81f426fc2fc034c2151dbe58be6e"},
    {file = "PyMuPDF-1.24.9-cp38-none-macosx_10_9_x86_64.whl", hash = "sha256:241913d0c76aacb05acdd8a0e82b1105883ffe6ef3bb4d9742b41d3c5e84d2db"},
    {file = "PyMuPDF-1.24.9-cp38-none-macosx_11_0_arm64.whl", hash = "sha256:ff70e26625b6cdd036e2c63b5d6c1897949c0e8b205cd756276f27baadaad340"},
    {file = "PyMuPDF-1.24.9-cp38-none-manylinux2014_aarch64.whl", hash = "sha256:8e29bc817afad511072371f24624c7c3b7485a9e656b6a65dc58fecdf5043b08"},
    {file = "PyMuPDF-1.24.9-cp38-none-manylinux2014_x86_64.whl", hash = "sha256:d17ec6920f91c43b6e777a017f3aaf44b205a3216771db9e8aa46e78a703f8f6"},
    {file = "PyMuPDF-1.24.9-cp38-none-musllinux_1_2_x86_64.whl", hash = "sha256:5cec9d17fdcbd83fa2c90190c22f652a0a51275cf75a29068eea025fff076829"},
    {file = "PyMuPDF-1.24.9-cp38-none-win32.whl", hash = "sha256:4f7b19f5c0026db49b7be17901728ed15761c5aa2031f04b01f9eb2e54f1b50e"},
    {file = "PyMuPDF-1.24.9-cp38-none-win_amd64.whl", hash = "sha256:e4c867f1cde68ff0e9c7889ea27c4c2c67df80e776f82619888bb69d1e1b27cf"},
    {file = "PyMuPDF-1.24.9-cp39-none-macosx_10_9_x86_64.whl", hash = "sha256:b4f85c24050e3778be6c7c1f4d4965fd4385281264798df7b4301b78895053fd"},
    {file = "PyMuPDF-1.24.9-cp39-none-macosx_11_0_arm64.whl", hash = "sha256:4e807010ef4e63cfb70dd88fe1fcd1d7e2b4e62ffa2b1dc53b35bc18bf939d8e"},
    {file = "PyMuPDF-1.24.9-cp39-none-manylinux2014_aarch64.whl", hash = "sha256:5dac888cc16981e385c886c26de6aabf914059215e028d14cd67767ff0c1288c"},
    {file = "PyMuPDF-1.24.9-cp39-none-manylinux2014_x86_64.whl", hash = "sha256:de55817c02e06ff75233ce2487cc5ebcbf585acd694bb69500825ee37789ac79"},
    {file = "PyMuPDF-1.24.9-cp39-none-musllinux_1_2_x86_64.whl", hash = "sha256:49cb22196f11c2327f6345554db48cfb2e31ed4f073ca6a872f21ddc4b0619c1"},
    {file = "PyMuPDF-1.24.9-cp39-none-win32.whl", hash = "sha256:46b1f84816c666e1c82f4249c1e815e92c462633255d72da20751eaad125d0f0"},
    {file = "PyMuPDF-1.24.9-cp39-none-win_amd64.whl", hash = "sha256:4fa45474d63715c707e3c3a6ebeeee75fd7aaa180512b75863e437f6876dfa86"},
    {file = "PyMuPDF-1.24.9.tar.gz", hash = "sha256:3692a5e824f10dc09bbddabab207f7cd5979831e48dd2f4de1be21e441767473"},
]

[package.dependencies]
PyMuPDFb = "1.24.9"

[[package]]
name = "pymupdfb"
version = "1.24.9"
description = "MuPDF shared libraries for PyMuPDF."
optional = false
python-versions = ">=3.8"
files = [
    {file = "PyMuPDFb-1.24.9-py3-none-macosx_10_9_x86_64.whl", hash = "sha256:3c9e694b1fb1bde37a8d3c953fbd0916e7dee8a4650142547d4f832105b17689"},
    {file = "PyMuPDFb-1.24.9-py3-none-macosx_11_0_arm64.whl", hash = "sha256:3fd74ee7969712ab457495465da0a61aab44d8cf9b71b9ef51910a8c6a90ad57"},
    {file = "PyMuPDFb-1.24.9-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:198f6b3713b6f980fa96c1099be0d5459c7d43c593299948f0ba528577e6bf46"},
    {file = "PyMuPDFb-1.24.9-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:ae044ebc8299f5a3ba822a6dfe97285dffd6c66cba194bc39180aa189a2755c9"},
    {file = "PyMuPDFb-1.24.9-py3-none-win32.whl", hash = "sha256:20ea17fd5799dcf7813ec099c0ce303f763e6e4ba8d0f54d5f84e4df90c3a340"},
    {file = "PyMuPDFb-1.24.9-py3-none-win_amd64.whl", hash = "sha256:c6b8adc0b9c91ff0f657440a816ad2130429a808cd53ff273f3e72532e526bdc"},
    {file = "PyMuPDFb-1.24.9.tar.gz", hash = "sha256:5505f07b3dded6e791ab7d10d01f0687e913fc75edd23fdf2825a582b6651558"},
]

[metadata]
lock-version = "2.0"
python-versions = "^3.12"
content-hash = "08949dcff98f79abd27d217e6b983b85b9f0f289b9291648c6eb6903694e2a48"

Dockerfile

FROM python:3.12-bullseye

RUN pip install poetry

    
# Set the working directory
WORKDIR /app

COPY ./pyproject.toml .
COPY ./poetry.lock .

RUN poetry install

ENV PYTHONPATH=/app

# Expose the application port
EXPOSE 8000

[julian-smith-artifex-com]

Linux/aarch64 wheels were missing from pypi.org until a few days ago.

So it's possible that pip or poetry or other Python packaging tools have been trying to build from source, which can take a long time (e.g. hours).

The wheels are now available on pypi.org, so please retry, and report back here with the results.

[Lanzelot711]

That makes sense

it still took 160s instead of 20s with prior version(1.24.5), which is an 8x increase, but it completed the build and worked on my local machine.

for my use case this is fine, thank you

[julian-smith-artifex-com]

Thanks for the update, I'll close this issue.