Setuptools #48301

risemeup1 · 2022-11-23T17:54:36Z

PR types

others

PR changes

others

Describe

在paddle支持setuptools打包工具，新增setup.py文件主要分为两个功能，1 封装cmake和make（暂不支持ninja加速，等ninja编译完成之后可支持）；2 使用setuptools工具打包python目录；3 本地同时支持原来的cmake -> make -> pip install .whl逻辑和现在使用python setupt.py install的逻辑；4 现在将GPUps流水线换成了现在python setup.py install逻辑，其它的流水线里的有些逻辑或者流水线插件里需要依赖.whl包做操作，需要后续讨论下再替换；5 目前已经跑通了py3，GPUps，CINN，Build流水线里所有的单测

To Do：
1逐步在所有流水线上替换成python setup.py install命令
2 后期支持ninja加速编译

使用方法：
1 使用方法,如在gpu环境下：
原来：cmake .. -DPY_VERSION=3.7 -DWITH_GPU=ON ，然后make -j ，pip install .whl
现在：export PY_VERSION=3.7 WITH_GPU=ON，然后python setup.py develop或者install
若要设置make -j的核数，如可export MAX_JOBS=23

2 常用命令：
2.1python setup.py install:
将包直接安装到python环境中，install 会将文件复制到 site-packages python安装文件夹下，如将包paddlepaddle_gpu-0.0.0-py3.7-linux-x86_64.egg 直接安装到/py3.7_env/lib/python3.7/site-packages

2.2 python setup.py develop:
develop 在 site-packages 中创建一个 .egg-link 文件> 目录，该方法不会真正的安装包，而是在系统环境中创建一个软链接指向包实际所在目录。即Creating /py3.7_env/lib/python3.7/site-packages/paddlepaddle-gpu.egg-link (link to build/python)，这样在修改包之后不用再安装就能生效，便于调试

2.3 python setup.py cmake-only :
仅仅camke-only,cmake后即退出，然后执行ccmake build可以调整build选项，或者python setup.py或者install命令

2.4 python setup.py rerun-cmake：
重新cmake

3 其他功能后面陆续完善

paddle-bot · 2022-11-23T17:54:39Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

zhangbo9674 · 2022-12-02T03:46:30Z

setup.py

+        subprocess.check_call(cmake_args)
+
+
+def make_run(args, build_path, envrion_var):


suggest build_run

zhangbo9674 · 2022-12-02T03:47:15Z

setup.py

+
+
+def get_package_data_and_package_dir():
+


delete empyt lines

zhangbo9674 · 2022-12-02T03:50:37Z

setup.py

+
+
+def get_headers():
+


delete empty lines

zhangbo9674 · 2022-12-02T03:57:10Z

python/CMakeLists.txt

@@ -63,9 +68,9 @@ if(WIN32)
    OUTPUT ${PADDLE_PYTHON_BUILD_DIR}/.timestamp
    COMMAND
      ${CMAKE_COMMAND} -E copy_directory ${PADDLE_SOURCE_DIR}/python/paddle
-      ${PADDLE_BINARY_DIR}/python/paddle/
+      ${PADDLE_BINARY_DIR}/python/paddle/ if (NOT WITH_SETUP)


为什么写在了一行？

这个是pre-commit自动修改的

zhangbo9674 · 2022-12-02T03:58:34Z

python/CMakeLists.txt

@@ -22,8 +22,13 @@ set(SETUP_LOG_FILE "setup.py.log")

 set(FLUID_CORE_NAME "libpaddle")

-configure_file(${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in
-               ${CMAKE_CURRENT_BINARY_DIR}/setup.py)
+if(WITH_SETUP)


Maybe INSTALL_WITH_SETUP is clearer?
Where this environment variable is set？

是否都最好保持用WITH开头好一些呢？

zhangbo9674 · 2022-12-02T04:02:20Z

paddle/scripts/paddle_build.sh

@@ -917,6 +917,160 @@ set +x
 set -ex
    fi
 }
+
+function run_linu_cpu_test_by_setup(){


linu -> linux

zhangbo9674 · 2022-12-02T04:03:58Z

paddle/scripts/paddle_build.sh

@@ -917,6 +917,160 @@ set +x
 set -ex
    fi
 }
+
+function run_linu_cpu_test_by_setup(){


这个函数只是删除了pip install *.whl的逻辑吧，需要重新写一个函数么？

我是觉得后期统一删除？？先留着？？

相同逻辑太多，不易于维护，如果改动很小，尽量复用原逻辑

zhangbo9674 · 2022-12-04T00:54:28Z

setup.py

+    )
+else:
+    if os.getenv("PY_VERSION") is None:
+        print("export PY_VERSION = %s" % platform.python_version())


这个print如果是为了在编译日志中提示用户，可以优化一下描述

zhangbo9674 · 2022-12-04T00:55:01Z

setup.py

+
+IS_WINDOWS = os.name == 'nt'
+
+


delete empty line

zhangbo9674 · 2022-12-04T00:55:48Z

setup.py

+IS_WINDOWS = os.name == 'nt'
+
+
+# filter args of setup.py


建议删除该注释，与函数名无区别，该注释无意义

zhangbo9674 · 2022-12-04T00:56:21Z

setup.py

+        filter_args_list.append(arg)
+    return cmake_and_build, only_cmake, rerun_cmake, filter_args_list
+
+


delete empty line

zhangbo9674 · 2022-12-04T00:57:32Z

setup.py

+    print("Start executing %s" % dist.script_name)
+    # get args of setup.py
+    dist.script_args = sys.argv[1:]
+    print("args of setup.py:%s" % dist.script_args)


print("Start executing %s" % dist.script_name)、print("args of setup.py:%s" % dist.script_args)
这两个print建议合并一个来打印解析结果

已改成print("Start executing python {} {}".format(dist.script_name,dist.script_args))

zhangbo9674 · 2022-12-04T01:17:31Z

paddle/scripts/paddle_build.sh

 function main() {
    local CMD=$1
    local parallel_number=$2
    init
    case $CMD in
      build_only)
-        cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        #cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}


delete unused code

zhangbo9674 · 2022-12-04T01:19:09Z

paddle/scripts/paddle_build.sh

 function main() {
    local CMD=$1
    local parallel_number=$2
    init
    case $CMD in
      build_only)
-        cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        #cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        run_setup ${PYTHON_ABI:-""} ${parallel_number}


run_setup函数中如何指定的parallel_number？没有看到指定逻辑

应该把run_setuo后边的第二个参数parallel_number直接删掉，我们是在setup.py中设置的这部分逻辑

这意味着用户无法设置编译并发度么？

zhangbo9674 · 2022-12-04T01:19:38Z

paddle/scripts/paddle_build.sh

@@ -3491,6 +3715,7 @@ function main() {
      build_and_check_cpu)
        set +e
        cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        #run_setup ${PYTHON_ABI:-""} ${parallel_number}


delete unused code

zhangbo9674 · 2022-12-04T01:19:52Z

paddle/scripts/paddle_build.sh

@@ -3665,7 +3890,9 @@ function main() {
        build_mac
        ;;
      cicheck_py37)
-        cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        # cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        # run_linux_cpu_test ${PYTHON_ABI:-""} ${PROC_RUN:-1}


delete unused code

zhangbo9674 · 2022-12-04T01:20:10Z

paddle/scripts/paddle_build.sh

@@ -3678,7 +3905,8 @@ function main() {
        parallel_test
        ;;
      build_gpubox)
-        cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}
+        #cmake_gen_and_build ${PYTHON_ABI:-""} ${parallel_number}


delete unused code

zhangbo9674

LGTM

zhiqiu

LGTM

XiaoguangHu01

LGTM

luotao1 · 2022-12-05T07:37:21Z

paddle/scripts/paddle_build.sh

+                export PYTHON_EXECUTABLE=/Library/Frameworks/Python.framework/Versions/3.6/bin/python3
+                export PYTHON_INCLUDE_DIR=/Library/Frameworks/Python.framework/Versions/3.6/include/python3.6m/
+                export PYTHON_LIBRARY=/Library/Frameworks/Python.framework/Versions/3.6/lib/libpython3.6m.dylib
+                pip3.6 install --user -r ${PADDLE_ROOT}/python/requirements.txt


@gsq7474741 这里新引入的python 3.6内容也需要清理下

* test * test * test * test * test * suport setuptools for paddle * modify paddle_build.sh * modify paddle_build.sh * modify paddle_build.sh * modify paddle_build.sh * modify paddle_build.sh * test * modify setup.py * modify build_options * modify build_options * modify paddle_build.sh * modify setup.py * modify paddle_build.sh * modify setup.py * modify setup.py * modify setup.py * modify setup.py * modfiy paddle_build.sh * debug * debug * debug * dddd * debug * debug * debug * debug * debug * debug * debug * debug * debug * fix bug that no version.py * debug * debug * debug * debug * debug * debug * Delete .pre-commit-config.yaml * debug * support ninja * support ninja * debug * debug * debug * support setuptools for paddle * modify code style * debug * debug * debug * debug * 取消make clean * 取消make clean * debug * debug * debug * debug for py3 * debug * debug * debug * 将mkdir_and_copy_file单独封装一个函数 * modify paddle_build.sh * modify setup.py after zhangbo reviewd

risemeup1 added 5 commits November 23, 2022 17:22

test

2e38890

test

3f09d26

test

f71773c

test

a813c41

test

e88579a

risemeup1 added 24 commits November 24, 2022 01:17

suport setuptools for paddle

b3ec07b

modify paddle_build.sh

a987634

modify paddle_build.sh

23239ed

modify paddle_build.sh

2be7708

modify paddle_build.sh

a1c3489

modify paddle_build.sh

840b868

test

452b4ee

modify setup.py

47d3cad

modify build_options

67c6e31

modify build_options

985f554

modify paddle_build.sh

168d6b0

modify setup.py

4f3549f

modify paddle_build.sh

e23879e

modify setup.py

a1c4996

modify setup.py

16c510b

modify setup.py

edf6997

modify setup.py

3a6474c

modfiy paddle_build.sh

da2429d

debug

9a0d21e

debug

61d2719

debug

a8a10a2

dddd

99fef47

debug

7fcbd65

debug

9a545a8

zhangbo9674 reviewed Dec 2, 2022

View reviewed changes

risemeup1 added 15 commits December 2, 2022 04:46

modify code style

d583144

debug

0084159

debug

c88fefd

debug

1d2c3b3

debug

66490b4

取消make clean

354c226

取消make clean

d9cfd3e

取消make clean

065ca7d

debug

e41a0bf

debug

c9992b0

debug

86bfaf1

debug for py3

18c6abf

debug

71d1f8f

debug

861728b

debug

93ad2c4

zhangbo9674 reviewed Dec 4, 2022

View reviewed changes

risemeup1 added 3 commits December 4, 2022 02:48

将mkdir_and_copy_file单独封装一个函数

e6fdcd7

modify paddle_build.sh

df90690

modify setup.py after zhangbo reviewd

41efa7f

zhangbo9674 approved these changes Dec 5, 2022

View reviewed changes

zhiqiu approved these changes Dec 5, 2022

View reviewed changes

XiaoguangHu01 approved these changes Dec 5, 2022

View reviewed changes

phlrain self-requested a review December 5, 2022 06:20

phlrain approved these changes Dec 5, 2022

View reviewed changes

zhangbo9674 merged commit 9913da0 into PaddlePaddle:develop Dec 5, 2022

SigureMo mentioned this pull request Dec 5, 2022

[CodeStyle][py36] update pypi doc #48640

Merged

luotao1 reviewed Dec 5, 2022

View reviewed changes

risemeup1 deleted the setuptools branch April 23, 2023 07:13

		subprocess.check_call(cmake_args)


		def make_run(args, build_path, envrion_var):

		filter_args_list.append(arg)
		return cmake_and_build, only_cmake, rerun_cmake, filter_args_list

Setuptools #48301

Setuptools #48301

Conversation

risemeup1 commented Nov 23, 2022 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Nov 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangbo9674 left a comment

Choose a reason for hiding this comment

zhiqiu left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

risemeup1 commented Nov 23, 2022 •

edited

Loading