-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Supports offline compilation of Paddle third-party libraries #53744
Conversation
你的PR提交成功,感谢你对开源项目的贡献! |
❌ The PR is not created using PR's template. You can refer to this Demo. |
1f82730
to
d65868b
Compare
… offline_compile
… offline_compile
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@@ -52,11 +52,11 @@ CMakeSettings.json | |||
Makefile | |||
.test_env/ | |||
.cache/ | |||
third_party/ | |||
build/third_party/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
build 目录下的还需要添加么?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
对,build整个目录好像已经添加了,加不加这个不影响应该
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
|
后面会更改官方文档,CI CLONE流水线也会加--recursive |
把third_party目录放到ignore是不是就可以 |
是的 |
PR types
Others
PR changes
Others
Description
card-70106
背景:
现在Paddle编译第三方库通过ExternalProject_Add命令对第三方库下载,当编译到某个第三方库的时候才会下载第三方库,然后编译,这种方式会存在很多问题,如下:
一边编译一边下载,当编译到某个第三方库的时候开始下载,导致git clone第三方库下载频率增加,如果网络或者代理不稳定,任何一次git clone失败就会导致编译失败,太过于依赖代理和网络的稳定性;
删除build目录后重新编译就需要重新下载这些第三方库,又要重新git clone第三方库,编译时间会增加很多,实际上第三方库是可以复用的;
外部用户需要外网能访问github,经常会因为git clone而下载失败,次PR合入之后,没有网络的用户,可以将第三方库拷贝到third_party目录下,就可以编译;
解决方案:此PR完成第一部分工作
将不打开任何编译选项,即cmake ..需要下载的第三方库(zlib gflags glog eigen threadpool dlpack xxhash warpctc warprnnt utf8proc lapack protobuf gloo crypotopp)通过git submodule作为Paddle的子模块,编译的时候不再需要下载第三方库
(to do)将常用编译选项下(如WITH_GPU,WITH_DISTRIBUTE等)且比较小的第三方库作为Paddle的子模块
(to do)其它的第三方库根据不同的编译选项进行判断,在cmake时下载,在make -j时准备好所有的第三方库