Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[19.07] Some Python packages are not built due to ValueError: not enough values to unpack #12680

Closed
BKPepe opened this issue Jul 1, 2020 · 13 comments
Closed

[19.07] Some Python packages are not built due to ValueError: not enough values to unpack #12680

BKPepe opened this issue Jul 1, 2020 · 13 comments

Comments

@BKPepe
Copy link
Member

BKPepe commented Jul 1, 2020 •
edited
Loading

Maintainers: @jefferyto , @commodo
Env: all
After update of Python3 in OpenWrt 19.07 some packages like zipp, cryptography, dateutil fails with the following error:

  File "/builder/shared-workdir/build/sdk/staging_dir/hostpkg/lib/python3.7/site-packages/pip/_vendor/packaging/tags.py", line 642, in _linux_platforms
    _, arch = linux.split("_", 1)
ValueError: not enough values to unpack (expected 2, got 1)
Makefile:35: recipe for target '/builder/shared-workdir/build/sdk/build_dir/target-arm_cortex-a9+vfpv3-d16_musl_eabi/pypi/python-zipp-python3/zipp-0.6.0/.built' failed

Faillog: https://downloads.openwrt.org/releases/faillogs-19.07/arm_cortex-a9_vfpv3-d16/packages/python-zipp/python3/compile.txt
@BKPepe
Copy link
Member Author

BKPepe commented Jul 1, 2020

I think that this could be related to this issue pypa/setuptools#1934

@BKPepe
Copy link
Member Author

BKPepe commented Jul 1, 2020

I downgraded setuptools in staging_dir/hostpkg/lib/python3.7/site-packages and installed setuptools-scm by this command: build/staging_dir/hostpkg/bin/pip3.7 install setuptools-scm and then run make package/python-zipp/{cleancompile} it worked.

@jefferyto
Copy link
Member

I think the actual issue is pypa/packaging#259. I'm looking into why we are setting _PYTHON_HOST_PLATFORM=linux2 now, if we can do what Alpine is doing or if we should patch pip's copy of packaging to bypass this.

@commodo
Copy link
Contributor

commodo commented Jul 2, 2020

I think the actual issue is pypa/packaging#259. I'm looking into why we are setting _PYTHON_HOST_PLATFORM=linux2 now, if we can do what Alpine is doing or if we should patch pip's copy of packaging to bypass this.

That setting is pretty old.
I think I just copied it from the old Python ~2.7.3 without giving it much thought.
I think it may have been put there by the first person who packaged Python for OpenWrt (before 2015). In the “oldpackages” repo, if that still exists anywhere these days.

@jefferyto
Copy link
Member

Well...

  • Host pip got the same environment variables as host Python in 4d94943 and 4d0a44d. (Not sure how the exact same change can be merged twice without any conflicts...)
  • Host Python got _PYTHON_HOST_PLATFORM=linux2 when I added python-host.mk in b12d7b6 (this was synced to python3-host.mk in 3af2027). I believe I copied _PYTHON_HOST_PLATFORM=linux2 from python-package.mk.
  • _PYTHON_HOST_PLATFORM="linux-$(ARCH)" was changed to _PYTHON_HOST_PLATFORM=linux2 in python-package.mk in f5f8d83 (synced to python3-package.mk in 8c057f3).
  • _PYTHON_HOST_PLATFORM="linux-$(ARCH)" was added to python-package.mk in 0141057 (added to python3-package.mk in 056e565).

Two things:

  • I think _PYTHON_HOST_PLATFORM should not be set for host pip (and perhaps even host Python), since it is installing/running code for host
  • For building packages for target, I'm still trying to understand if/what value of _PYTHON_HOST_PLATFORM is appropriate

Thoughts?

@commodo
Copy link
Contributor

commodo commented Jul 2, 2020 

* I think `_PYTHON_HOST_PLATFORM` should not be set for host pip (and perhaps even host Python), since it is installing/running code for host

fine by me

* For building packages for target, I'm still trying to understand if/what value of `_PYTHON_HOST_PLATFORM` is appropriate

i think i played with it a bit a few years ago, and just decided to go with linux2 for target-Python [no idea why for host-python]; there might be some auto-detection there that doesn't work well for target-Python, it could be that it felt sane to have it unified for host-python as well;

@KaneGreen
Copy link

KaneGreen commented Jul 3, 2020 •
edited
Loading

I met the same problem these days.

pip-20.1.1 has this file _vendor/packaging/tags.py, while pip-19.2.3 doesn't have.
In fact, this file was introduced since pip-19.3, so pip-19.2.3 is the last version without the file.
Reverting pip to 19.2.3 may help. But it is NOT the ultimate solution.

Now, I am trying to using _PYTHON_HOST_PLATFORM=linux-aarch64 instead of _PYTHON_HOST_PLATFORM=linux2 in three file (python3-host.mk, python3-package.mk, python3/Makefile) to build on x86_84 for my ARM device.
If it succeeds, _PYTHON_HOST_PLATFORM="linux-$(ARCH)" may work.
It failed. We need another solution.

@BKPepe
Copy link
Member Author

BKPepe commented Jul 5, 2020

@jefferyto , @commodo What do you think to revert, I mean postpone update Python of 3.7 in OpenWrt 19.07 until you found workaround/solution? For Turris routers, we froze packages feed before update was introduced.

@jefferyto
Copy link
Member

I will resume working on this tomorrow, hope to find the right fix in a day or two. I would prefer to wait until later this week to decide if a revert is necessary.

@commodo
Copy link
Contributor

commodo commented Jul 6, 2020

My preference would be to revert for now, and see later.
But that's because I don't have enough time to dig into it now.

jefferyto added a commit to jefferyto/openwrt-packages that referenced this issue Jul 8, 2020
@jefferyto
python3: Use default _PYTHON_HOST_PLATFORM
4b0c1f8 
This lets the Python build process set _PYTHON_HOST_PLATFORM instead of
forcing an explicit value.

Also:

* Save the target _PYTHON_HOST_PLATFORM value during Build/InstallDev
  for use when building target Python packages (in python3-package.mk).

* Use the (mostly) default PYTHON_FOR_BUILD value, instead patch
  configure to remove the platform triplet from the sysconfigdata file
  name.

* Remove the "CROSS_COMPILE=yes" make variable (there is no indication
  that this variable is necessary).

* Force host pip to build packages from source instead of downloading
  binary wheels.

  Previously, host pip can download universal (platform-independent)
  wheels but not platform-specific wheels, because of the custom
  _PYTHON_HOST_PLATFORM value. (Packages that do not have universal
  wheels would be compiled from source.)

  With a correct _PYTHON_HOST_PLATFORM, host pip can install
  platform-specific wheels as well. However, the pre-built shared object
  (.so) files in these wheels will have the host's platform triplet in
  their file names. When target Python packages are built (using the
  target's _PYTHON_HOST_PLATFORM), Python will not use these shared
  object files.

  By forcing host pip to build packages from source, the built shared
  object files will not have the platform triplet in their file names.
  (Host Python has been patched to remove the platform triplet from file
  names.) This allows these packages to be used when building target
  Python packages.

  (The net effect of this complete change is that platform-dependent
  packages will continue to be compiled from source, while
  platform-independent packages will now also be compiled from source.)

Fixes openwrt#12680.

Signed-off-by: Jeffery To <jeffery.to@gmail.com>
This was referenced Jul 8, 2020
Merged
Merged
@jefferyto
Copy link
Member

I believe this issue has been fixed.

@KaneGreen
Copy link

My cross-compiling (on x86_64) to aarch64 succeed.

@BKPepe
Copy link
Member Author

BKPepe commented Jul 15, 2020

Yes, I can confirm that this is fixed. Closing.

@BKPepe BKPepe closed this as completed Jul 15, 2020
@jefferyto jefferyto mentioned this issue Oct 16, 2020
Merged
KaneGreen added a commit to KaneGreen/OpenWrt-Builder that referenced this issue Nov 25, 2020
@KaneGreen
try to fix Python 3.7.8 (openwrt/packages#12680)
2aff361
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jefferyto @commodo @BKPepe @KaneGreen