Support for python3.11

[kuutsav]

Any idea when the support for python3.11 will be out?

[taku910]

Binary wheel packages are now built with github actions (actions/setup-python@v3), which don't support python3.11 now.
Basically, we will follow the support status of github actions.

When running pip on 3.11, binary packages are built from source. If you have the proper build environment on Ubuntu, this fallback build will work, but otherwise, e.g., WIndows, it will likely not work as is.

You might want to build the wheel packages manually. There is an instruction page.
https://github.com/google/sentencepiece/tree/master/python

[Omar-Abdul-Azeez]

@taku910 I get a file not found error but string_view.h does exist where it's specified.

[ 46%] Building CXX object src/CMakeFiles/sentencepiece-static.dir/bpe_model.cc.obj
D:\sentencepiece\src\bpe_model.cc:1:0: warning: -fPIC ignored for target (all code is position independent)
 // Copyright 2016 Google Inc.
 ^
In file included from D:\sentencepiece\src\common.h:29:0,
                 from D:\sentencepiece\src\model_interface.h:24,
                 from D:\sentencepiece\src\bpe_model.h:18,
                 from D:\sentencepiece\src\bpe_model.cc:15:
D:/sentencepiece/third_party/absl/strings/string_view.h:31:23: fatal error: string_view: No such file or directory
compilation terminated.
src\CMakeFiles\sentencepiece-static.dir\build.make:495: recipe for target 'src/CMakeFiles/sentencepiece-static.dir/bpe_model.cc.obj' failed
mingw32-make[2]: *** [src/CMakeFiles/sentencepiece-static.dir/bpe_model.cc.obj] Error 1
CMakeFiles\Makefile2:126: recipe for target 'src/CMakeFiles/sentencepiece-static.dir/all' failed
mingw32-make[1]: *** [src/CMakeFiles/sentencepiece-static.dir/all] Error 2
Makefile:154: recipe for target 'all' failed
mingw32-make: *** [all] Error 2

[taku910]

string_view support (C++17 or later) is required to build sentencepiece.
#784

[Omar-Abdul-Azeez]

so I've upgraded the build environment (actually removed mingw and used cygwin) and managed to build sentencepiece but the output is:

Install the project...
-- Install configuration: ""
-- Installing: /cygdrive/d/sentencepiece/build/root/lib/pkgconfig/sentencepiece.pc
-- Installing: /cygdrive/d/sentencepiece/build/root/lib/libsentencepiece.a
-- Installing: /cygdrive/d/sentencepiece/build/root/lib/libsentencepiece_train.a
-- Installing: /cygdrive/d/sentencepiece/build/root/bin/spm_encode.exe
-- Installing: /cygdrive/d/sentencepiece/build/root/bin/spm_decode.exe
-- Installing: /cygdrive/d/sentencepiece/build/root/bin/spm_normalize.exe
-- Installing: /cygdrive/d/sentencepiece/build/root/bin/spm_train.exe
-- Installing: /cygdrive/d/sentencepiece/build/root/bin/spm_export_vocab.exe
-- Installing: /cygdrive/d/sentencepiece/build/root/include/sentencepiece_trainer.h
-- Installing: /cygdrive/d/sentencepiece/build/root/include/sentencepiece_processor.h

Note there is no lib/sentencepiece.lib nor lib/sentencepiece_train.lib
Trying to build the python wheel fails since it looks for those two lib files...

[juliusfrost]

Binary wheel packages are now built with github actions (actions/setup-python@v3), which don't support python3.11 now. Basically, we will follow the support status of github actions.

When running pip on 3.11, binary packages are built from source. If you have the proper build environment on Ubuntu, this fallback build will work, but otherwise, e.g., WIndows, it will likely not work as is.

You might want to build the wheel packages manually. There is an instruction page. https://github.com/google/sentencepiece/tree/master/python

Looking at the latest build wheels workflow it looks like it is already using python 3.11. Looks like there just needs to be a new release for pip.

[Omar-Abdul-Azeez]

As far as I can tell, while 3.11 is used, only 3.6 to 3.10 wheels are built.

[juliusfrost]

As far as I can tell, while 3.11 is used, only 3.6 to 3.10 wheels are built.

Got it. It looks like cibuildwheel package needs to be updated to support 3.11

[juliusfrost]

I created a PR to update cibuildwheel: #819

[juliusfrost]

The lastest CI build succeeded: https://github.com/google/sentencepiece/actions/runs/4228494807
and the artifact download that contains the build wheels is here: https://github.com/google/sentencepiece/suites/11099309556/artifacts/565198247

@taku910 Do you know when the next release is planned to upload to pypi?

[billkiddo]

@juliusfrost the package won't build because of error: "Cannot open include file: 'sentencepiece_processor.h'" python 3.11 on win11 :(

[ZeroBomb]

@billkiddo I was able to build on windows.

1. Use vcpkg to install, remembering to use x64-windows-static
2. Copy the folders (debug, include, lib, share) from vcpkg\installed\x64-windows-static to sentencepiece\build\root
3. Build as normal

[powderluv]

can we please get a pypi release of sentencepiece with 3.11 / Windows ?

[psifertex]

Looks like GH runners support 3.11 now:
actions/runner-images#6459

We've got a project blocked on this not existing as well.

EDIT: Sorry, I missed #810 (comment) previously. Looks like that's known to be resolved.

[liednik]

I had a similar problem with building a docker image and running apt-get install -y cmake resolved the problem, though now the size of the image is pretty bigger than before.

[lsmith77]

I can confirm that the master branch works with python 3.11

[jstoppa]

@billkiddo I was able to build on windows.

1. Use vcpkg to install, remembering to use x64-windows-static
2. Copy the folders (debug, include, lib, share) from vcpkg\installed\x64-windows-static to sentencepiece\build\root
3. Build as normal

Thanks @ZeroBomb this bit was the last piece of the puzzle to finally be able to install sentencepiece

[zetaloop]

I built sentencepiece on windows 11 for python 3.11 and created a pre-compiled whl file for version 0.1.98.
Hope this helps those who want to install it quickly.
pip install sentencepiece-0.1.98-cp311-cp311-win_amd64.whl
sentencepiece-0.1.98-cp311-cp311-win_amd64.zip

[workingprototype]

I built sentencepiece on windows 11 for python 3.11 and created a pre-compiled whl file for version 0.1.98. Hope this helps those who want to install it quickly. pip install sentencepiece-0.1.98-cp311-cp311-win_amd64.whl sentencepiece-0.1.98-cp311-cp311-win_amd64.zip

You legend. This fixed it. Thanks!

[MKlaput]

Awesome, this saved me a lot of trouble! Thanks!

[bitwisexornot]

Still does not build on MinGW (x64)

[crypto-maniac]

i got this error

pip install sentencepiece-0.1.98-cp311-cp311-win_amd64.whl
WARNING: Requirement 'sentencepiece-0.1.98-cp311-cp311-win_amd64.whl' looks like a filename, but the file does not exist
ERROR: sentencepiece-0.1.98-cp311-cp311-win_amd64.whl is not a supported wheel on this platform.

[zetaloop]

i got this error

pip install sentencepiece-0.1.98-cp311-cp311-win_amd64.whl
WARNING: Requirement 'sentencepiece-0.1.98-cp311-cp311-win_amd64.whl' looks like a filename, but the file does not exist
ERROR: sentencepiece-0.1.98-cp311-cp311-win_amd64.whl is not a supported wheel on this platform.

The first error occurs because the specified Python package file cannot be found. Please check if the current working directory is pointing to the location where the downloaded and extracted .whl file is located.

The second error may be due to an incompatible system. This .whl file is only supported on Windows x64 systems, and cannot be used on MacOS or Linux. You may need to build it by yourself.

[markusantonuis]

I built sentencepiece on windows 11 for python 3.11 and created a pre-compiled whl file for version 0.1.98. Hope this helps those who want to install it quickly. pip install sentencepiece-0.1.98-cp311-cp311-win_amd64.whl sentencepiece-0.1.98-cp311-cp311-win_amd64.zip

After hours of trying to get this installed, this is what worked for me on Windows 10, thanks a lot!

[taku910]

Fixed in v0.1.98

[m4ttgit]

Win10. Solved it with python 11.6.

Refer : #945