Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Supports offline compilation of Paddle third-party libraries #53744

Merged
merged 40 commits into from
May 17, 2023

Conversation

risemeup1
Copy link
Contributor

@risemeup1 risemeup1 commented May 12, 2023

PR types

Others

PR changes

Others

Description

card-70106
背景:
现在Paddle编译第三方库通过ExternalProject_Add命令对第三方库下载,当编译到某个第三方库的时候才会下载第三方库,然后编译,这种方式会存在很多问题,如下:

  • 一边编译一边下载,当编译到某个第三方库的时候开始下载,导致git clone第三方库下载频率增加,如果网络或者代理不稳定,任何一次git clone失败就会导致编译失败,太过于依赖代理和网络的稳定性;

  • 删除build目录后重新编译就需要重新下载这些第三方库,又要重新git clone第三方库,编译时间会增加很多,实际上第三方库是可以复用的;

  • 外部用户需要外网能访问github,经常会因为git clone而下载失败,次PR合入之后,没有网络的用户,可以将第三方库拷贝到third_party目录下,就可以编译;

解决方案:此PR完成第一部分工作

  • 将不打开任何编译选项,即cmake ..需要下载的第三方库(zlib gflags glog eigen threadpool dlpack xxhash warpctc warprnnt utf8proc lapack protobuf gloo crypotopp)通过git submodule作为Paddle的子模块,编译的时候不再需要下载第三方库

  • (to do)将常用编译选项下(如WITH_GPU,WITH_DISTRIBUTE等)且比较小的第三方库作为Paddle的子模块

  • (to do)其它的第三方库根据不同的编译选项进行判断,在cmake时下载,在make -j时准备好所有的第三方库

@paddle-bot
Copy link

paddle-bot bot commented May 12, 2023

你的PR提交成功,感谢你对开源项目的贡献!
请关注后续CI自动化测试结果,详情请参考Paddle-CI手册
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

@paddle-bot
Copy link

paddle-bot bot commented May 12, 2023

❌ The PR is not created using PR's template. You can refer to this Demo.
Please use PR's template, it helps save our maintainers' time so that more developers get helped.

Copy link
Contributor

@jiahy0825 jiahy0825 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@risemeup1 risemeup1 changed the title Offline compile Supports offline compilation of Paddle third-party libraries May 16, 2023
@@ -52,11 +52,11 @@ CMakeSettings.json
Makefile
.test_env/
.cache/
third_party/
build/third_party/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build 目录下的还需要添加么?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

对,build整个目录好像已经添加了,加不加这个不影响应该

Copy link
Contributor

@XiaoguangHu01 XiaoguangHu01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@pangyoki pangyoki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@pangyoki pangyoki left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jzhang533
Copy link
Contributor

  • 引入submodule的方式,对开发的习惯会有改变。除了这个PR之外,还需要看看还有哪些地方需要做出说明或者更新。我暂时想到的:
    • 要有一份简短的材料,说明一下,本地进行开发的时候,需要做哪些事情。(如git clone--recursive 选项, 执行git submoudle 等)。
    • 更新所有涉及源码编译的文档。
    • CI里是不是也要做更新?
  • submodule在本地做了更新后,可能会不小心在提交PR的时候,带上了对submodule的更新,除了在review的时候注意外,不确定有没有什么方式做保护。

@risemeup1
Copy link
Contributor Author

  • 引入submodule的方式,对开发的习惯会有改变。除了这个PR之外,还需要看看还有哪些地方需要做出说明或者更新。我暂时想到的:

    • 要有一份简短的材料,说明一下,本地进行开发的时候,需要做哪些事情。(如git clone--recursive 选项, 执行git submoudle 等)。
    • 更新所有涉及源码编译的文档。
    • CI里是不是也要做更新?
  • submodule在本地做了更新后,可能会不小心在提交PR的时候,带上了对submodule的更新,除了在review的时候注意外,不确定有没有什么方式做保护。

后面会更改官方文档,CI CLONE流水线也会加--recursive

@risemeup1 risemeup1 merged commit 734dc44 into PaddlePaddle:develop May 17, 2023
@zhiqiu
Copy link
Contributor

zhiqiu commented May 17, 2023

  • submodule在本地做了更新后,可能会不小心在提交PR的时候,带上了对submodule的更新,除了在review的时候注意外,不确定有没有什么方式做保护。

把third_party目录放到ignore是不是就可以

@risemeup1
Copy link
Contributor Author

  • submodule在本地做了更新后,可能会不小心在提交PR的时候,带上了对submodule的更新,除了在review的时候注意外,不确定有没有什么方式做保护。

把third_party目录放到ignore是不是就可以

是的

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants