Support build on native Windows #4661

python3kgae · 2024-08-11T12:34:06Z

🚀 The feature, motivation and pitch

Currently, executorch supports Windows via WSL.
It would be beneficial to enable native Windows builds as well, as this would assist users who do not have WSL installed.

Alternatives

Executorch already supports Windows via WSL.

Additional context

No response

RFC (Optional)

The plan is to create a PowerShell version of install_requirements.sh.

Here is a list of things that need to be updated for building on native Windows:

Resolve build issues in file_data_loader.cpp, which includes <unistd.h>, and invoke ::close on -1.
Get the Windows version of buck2 for Windows build in build/resolve_buck.py.
Check the path based on the build configuration for Windows in setup.py. For example, third-party/flatbuffers/flatc should be third-party/flatbuffers/Debug/flatc.exe for Debug builds.
Avoid generating 'def flat.exe()' which is illegal python in build/pip_data_bin_init.py.in.
Add the Windows version of kernel_link_options in build/Utils.cmake.
Add links of pthreadpool and cpuinfo for custom_ops_aot_lib in extension/llm/custom_ops/CMakeLists.txt.
Define ssize_t for Windows in runtime/core/portable_type/tensor_impl.h and runtime/core/exec_aten/util/tensor_util.h.
[Build] choose_qparams_tensor_out get wrong return type cause build fail on native Windows #4659

iseeyuan · 2024-08-12T16:57:28Z

Hi @python3kgae , thanks for submitting this issue and proposing the list of items. @daveboat and @malfet , does the list make sense to you?

We are limited in resource on Windows. @python3kgae , are you willing to directly contribute to the ET windows build? We are happy to support/unblock you.

Added install_requirements.ps1 for build on native Windows. Also fix issues block build on native Windows. 1. Get the Windows version of buck2 for Windows. 2. Check the path based on the build configuration for Windows in setup.py. 3. Avoid generating 'def flat.exe()' which is illegal python. 4. Add the Windows version of kernel_link_options. 5. Add links of pthreadpool and cpuinfo for custom_ops_aot_lib. 6. Define ssize_t for Windows. Still have one link issue tracked with pytorch#4659. For pytorch#4661

python3kgae · 2024-08-13T03:21:12Z

PR created.
Need some help with #4659.
Not sure where choose_qparams_tensor_out is defined for WSL.

Move the details of install_requirements.sh into install_requirements.py. Then, call install_requirements.py from install_requirements.sh. This will avoid duplication when add Windows version of installing_requirements. This is for issue pytorch#4661.

Move the details of install_requirements.sh into install_requirements.py. Then, call install_requirements.py from install_requirements.sh. This will avoid duplication when adding Windows version of installing_requirements. This is for issue #4661. Release Note: Moved the main install_requirements.sh logic into install_requirements.py so that it can also run on Windows. Test Plan: Tested on a mac. Ran `./install_requirements.sh --pybind xnnpack` under python 3.10, and it completed successfully. To make sure that xnnpack was actually installed, I manually ran the steps from the notebook at https://colab.research.google.com/drive/1qpxrXC3YdJQzly3mRg-4ayYiOjC6rue3, and it passed. When running the same in a clean python 3.8 environment, it failed as expected with ``` % ./install_requirements.sh --pybind xnnpack Collecting packaging Using cached packaging-24.1-py3-none-any.whl.metadata (3.2 kB) Using cached packaging-24.1-py3-none-any.whl (53 kB) Installing collected packages: packaging Successfully installed packaging-24.1 ERROR: ExecuTorch does not support python version 3.8.19: must satisfy ">=3.10" ``` Also tested under WSL and native Windows.

Added install_requirements.ps1 for build on native Windows. Also fix issues block build on native Windows. 1. Get the Windows version of buck2 for Windows. 2. Check the path based on the build configuration for Windows in setup.py. 3. Avoid generating 'def flat.exe()' which is illegal python. 4. Add the Windows version of kernel_link_options. 5. Add links of pthreadpool and cpuinfo for custom_ops_aot_lib. 6. Define ssize_t for Windows. Still have one link issue tracked with pytorch#4659. For pytorch#4661

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

Fix windows build issues in setup.py 1. Build executable name and dynamic lib name based on platform. 2. Set the src directory based on build config for windows. 3. Avoid using os.geteuid() on windows. For pytorch#4661

Move size_t and ssize_t header to runtime/platform/compiler.h. Define ssize_t for windows as ptrdiff_t. For pytorch#4661

There is no direct replacement for libpthread.so on Windows. Link pthreadpool and cpuinfo staticly for windows. For pytorch#4661

Add install_requirements.bat on windows like install_requirements.sh. Also force use ClangCL on windows in install_requirements.py. For pytorch#4661

Windows executable will have .exe suffix. It will cause illegal python code `def flat.exe():` generated. This change fix the issue by remove the suffix. For pytorch#4661

Add msvc version of kernel_link_options. For pytorch#4661

Fix windows build issues in setup.py 1. Build executable name and dynamic lib name based on platform. 2. Set the src directory based on build config for windows. 3. Avoid using os.geteuid() on windows. For #4661

Add install_requirements.bat on windows like install_requirements.sh. Also force use ClangCL on windows in install_requirements.py. For #4661

Add msvc version of kernel_link_options. For #4661

There is no direct replacement for libpthread.so on Windows. Link pthreadpool and cpuinfo staticly for windows. For #4661 Pull Request resolved: #4860

Move size_t and ssize_t header to runtime/platform/compiler.h. Define ssize_t for windows as ptrdiff_t. For #4661 Pull Request resolved: #4859

Two build issues have been addressed in this pull request: 1. Avoid closing when fd_ is -1, as this would cause a crash on Windows. 2. Introduce separate headers for Windows and Unix, enabling Windows builds to utilize a distinct header and implement pread. For pytorch#4661

The binary directory for Windows will have build type like Debug/Release at the end. For pytorch#4661

The basic_string_view constructor: ``` template< class It, class End > constexpr basic_string_view( It first, End last ); ``` requires C++20. To allow the code to compile with C++17, use the basic_string_view constructor: ``` constexpr basic_string_view( const CharT* s, size_type count ); ``` For pytorch#4661

…ble (#5025) The basic_string_view constructor: ``` template< class It, class End > constexpr basic_string_view( It first, End last ); ``` requires C++20. To allow the code to compile with C++17, use the basic_string_view constructor: ``` constexpr basic_string_view( const CharT* s, size_type count ); ``` For #4661

Two build issues have been addressed in this pull request: 1. Avoid closing when fd_ is -1, as this would cause a crash on Windows. 2. Introduce separate headers for Windows and Unix, enabling Windows builds to utilize a distinct header and implement pread. For pytorch#4661

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

Replace clock_gettime with timespec_get which works on Windows as well. For pytorch#4661

Set the src directory based on build configuration for windows. Rename _portable_lib.* to _portable_lib.cp* to avoid _portable_lib.lib is selected on windows. For pytorch#4661

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

python3kgae mentioned this issue Aug 13, 2024

[Build] fix build issues for native Windows. #4681

Open

manuelcandales added module: build Related to buck2 and cmake build triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module labels Aug 14, 2024

manuelcandales assigned dbort Aug 14, 2024

dbort assigned tarun292 and unassigned dbort Aug 14, 2024

python3kgae mentioned this issue Aug 16, 2024

[Build] Implement install_requirements.sh with Python #4748

Merged

python3kgae added a commit to python3kgae/executorch that referenced this issue Aug 23, 2024

[Build] Support windows in resolve_buck.py

948a153

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

python3kgae mentioned this issue Aug 23, 2024

[Build] Support windows in resolve_buck.py #4856

Open

python3kgae mentioned this issue Aug 23, 2024

[Build] Support windows in setup.py #4858

Merged

python3kgae added a commit to python3kgae/executorch that referenced this issue Aug 23, 2024

[Build] Define ssize_t for windows build.

b033d1a

Move size_t and ssize_t header to runtime/platform/compiler.h. Define ssize_t for windows as ptrdiff_t. For pytorch#4661

python3kgae mentioned this issue Aug 23, 2024

[Build] Define ssize_t for windows build. #4859

Merged

python3kgae added a commit to python3kgae/executorch that referenced this issue Aug 23, 2024

[Build] Link pthreadpool and cpuinfo for windows.

e41a062

There is no direct replacement for libpthread.so on Windows. Link pthreadpool and cpuinfo staticly for windows. For pytorch#4661

python3kgae mentioned this issue Aug 23, 2024

[Build] Link pthreadpool and cpuinfo for windows. #4860

Merged

python3kgae mentioned this issue Aug 23, 2024

[Build] Add install_requirements.bat for windows build #4862

Merged

python3kgae mentioned this issue Aug 23, 2024

[Build] Remove .exe suffix on windows #4864

Merged

python3kgae added a commit to python3kgae/executorch that referenced this issue Aug 23, 2024

[Build] add kernel_link_options for windows

e6a65b4

Add msvc version of kernel_link_options. For pytorch#4661

python3kgae mentioned this issue Aug 23, 2024

[Build] add kernel_link_options for windows #4865

Merged

dbort pushed a commit that referenced this issue Aug 23, 2024

[Build] Add install_requirements.bat for windows build (#4862)

ee6e4e9

Add install_requirements.bat on windows like install_requirements.sh. Also force use ClangCL on windows in install_requirements.py. For #4661

dbort pushed a commit that referenced this issue Aug 23, 2024

[Build] add kernel_link_options for windows (#4865)

bc66ff2

Add msvc version of kernel_link_options. For #4661

kirklandsign pushed a commit that referenced this issue Aug 23, 2024

[Build] Link pthreadpool and cpuinfo for windows.

b157e5f

There is no direct replacement for libpthread.so on Windows. Link pthreadpool and cpuinfo staticly for windows. For #4661 Pull Request resolved: #4860

kirklandsign pushed a commit that referenced this issue Aug 23, 2024

[Build] Define ssize_t for windows build.

6d29c1d

Move size_t and ssize_t header to runtime/platform/compiler.h. Define ssize_t for windows as ptrdiff_t. For #4661 Pull Request resolved: #4859

python3kgae mentioned this issue Aug 26, 2024

[Build] fix file_data_loader.cpp build issues for windows #4899

Draft

python3kgae added a commit to python3kgae/executorch that referenced this issue Aug 29, 2024

[Build] Use correct flatcc path for Windows

983c240

The binary directory for Windows will have build type like Debug/Release at the end. For pytorch#4661

python3kgae mentioned this issue Aug 29, 2024

[Build] Set correct flatcc path for Windows #4993

Open

python3kgae mentioned this issue Aug 31, 2024

[Build] Use C++17 Constructor for tiktoken.cpp when C++20 is unavailable #5025

Merged

jsidewhite pushed a commit to jsidewhite/executorch that referenced this issue Sep 7, 2024

[Build] Support windows in resolve_buck.py

31526d2

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

jsidewhite mentioned this issue Sep 7, 2024

[Windows Build] Implement MMAP for mmap_data_loader.cpp #5164

Open

python3kgae added a commit to python3kgae/executorch that referenced this issue Sep 10, 2024

[Build] use timespec_get to get current time

92bac8e

Replace clock_gettime with timespec_get which works on Windows as well. For pytorch#4661

python3kgae mentioned this issue Sep 10, 2024

[Build] use timespec_get to get current time #5252

Open

python3kgae mentioned this issue Sep 15, 2024

[Build] Support windows for pybindings install in setup.py #5382

Open

hans00 mentioned this issue Sep 23, 2024

Support bare-metal Windows #3735

Open

4 tasks

python3kgae added a commit to python3kgae/executorch that referenced this issue Oct 8, 2024

[Build] Support windows in resolve_buck.py

07f5c52

Add Windows support to BUCK_PLATFORM_MAP. Also use urllib.request.urlretrieve directly to work around PermissionError on Windows. For pytorch#4661

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support build on native Windows #4661

Support build on native Windows #4661

python3kgae commented Aug 11, 2024 •

edited

Loading

iseeyuan commented Aug 12, 2024 •

edited

Loading

python3kgae commented Aug 13, 2024

Support build on native Windows #4661

Support build on native Windows #4661

Comments

python3kgae commented Aug 11, 2024 • edited Loading

🚀 The feature, motivation and pitch

Alternatives

Additional context

RFC (Optional)

iseeyuan commented Aug 12, 2024 • edited Loading

python3kgae commented Aug 13, 2024

python3kgae commented Aug 11, 2024 •

edited

Loading

iseeyuan commented Aug 12, 2024 •

edited

Loading