logger manager #45909

sljlp · 2022-09-09T03:56:24Z

PR types

Others

PR changes

Others

Describe

uniform logger manager in FleetAPI.
hidde API under distributed/utils which users don't need.

log samples:

paddle-bot · 2022-09-09T03:56:37Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

gongweibao

logger does not belong to Fleet class.

gongweibao · 2022-09-13T02:05:26Z

python/paddle/distributed/fleet/fleet.py

@@ -1335,17 +1329,13 @@ def _minimize_impl(self,
                                                        no_grad_set=no_grad_set)

        if meta_optimizer:
-            # print("before minimize program id:", id(loss.block.program))


logger.debug()?

gongweibao · 2022-09-13T02:05:36Z

python/paddle/distributed/fleet/fleet.py

            optimize_ops, params_grads = meta_optimizer.minimize(
                loss, startup_program, parameter_list, no_grad_set=no_grad_set)
-            # print("after minimize program id:", id(loss.block.program))


Same as before

gongweibao · 2022-09-13T02:07:21Z

python/paddle/distributed/utils/__init__.py

+# limitations under the License.
+
+__all__ = [  #noqa
+    'get_host_name_ip',


Are all these codes for the users?

gongweibao

Small comments.

gongweibao · 2022-09-14T06:55:29Z

python/paddle/distributed/fleet/fleet.py

@@ -1463,3 +1479,41 @@ def _minimize_losses_impl(self,
        fleet.util._set_strategy(context["valid_strategy"])

        return optimize_ops, params_grads
+
+    def setLogLevel(self, level):


Move to log_util

gongweibao · 2022-09-14T06:59:16Z

python/paddle/distributed/utils/__init__.py

+
+__all__ = []
+
+from .log_utils import get_logger


Remove them in the future PR.

gongweibao

LGTM

gongweibao · 2022-09-14T11:26:01Z

python/paddle/distributed/fleet/__init__.py

@@ -90,3 +91,6 @@
 shrink = fleet.shrink
 get_hybrid_communicate_group = fleet.get_hybrid_communicate_group
 distributed_scaler = distributed_scaler
+set_log_level = log_util.set_log_level
+get_log_level_code = log_util.get_log_level_code


get_log_level one is enough!

gongweibao

LGTM

… fleet_logger

…e/Paddle into fleet_logger

* fix apis global scatter/gather * update

XiaoguangHu01

LGTM

XiaoguangHu01

LGTM

… fleet_logger

XiaoguangHu01

LGTM

jzhang533

如果API是设计为只限内部调用的，建议API的名称有特殊的标识，常见的做法是用_开头命名。
API不公开，不代表API不需要写文档。

jzhang533 · 2022-09-20T03:02:55Z

python/paddle/distributed/utils/log_utils.py

+import logging
+
+
+def get_logger(log_level, name="root"):


why we need another 'get_logger'? there are multiple versions of this tiny utility.
$ grep -irn 'def get_logger' python
python/paddle/distributed/utils/log_utils.py:18:def get_logger(log_level, name="root"):
python/paddle/distributed/auto_parallel/utils.py:31:def get_logger(log_level, name="auto_parallel"):
python/paddle/distributed/fleet/utils/http_server.py:30:def get_logger(name, level, fmt):
python/paddle/distributed/fleet/launch_utils.py:267:def get_logger(log_level=20, name="root"):
python/paddle/fluid/tests/unittests/auto_checkpoint_utils.py:44:def get_logger():
python/paddle/fluid/incubate/fleet/utils/http_server.py:25:def get_logger(name, level, fmt):
python/paddle/fluid/log_helper.py:22:def get_logger(name, level, fmt=None):

This should be resolved later.

jzhang533 · 2022-09-20T03:16:57Z

python/paddle/distributed/utils/__init__.py

+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__all__ = []


APIs are only hidden from here. But APIs are still accessible by

from paddle.distributed.utils.launch_utils import get_cluster, get_gpus, get_cluster_from_args

Considering to add a prefix '_'.

下划线的意思是模块内部，也就是一个文件内部。
加不加两可之间。我还没有找到这个方面的标准。
但是不能主动暴露给用户。

uniform logger manager in FleetAPI. hidde API under distributed/utils which users don't need.

gongweibao requested changes Sep 13, 2022

View reviewed changes

gongweibao requested changes Sep 14, 2022

View reviewed changes

gongweibao previously approved these changes Sep 14, 2022

View reviewed changes

gongweibao reviewed Sep 14, 2022

View reviewed changes

sljlp dismissed gongweibao’s stale review via 2ece969 September 14, 2022 12:15

gongweibao previously approved these changes Sep 15, 2022

View reviewed changes

manage fleet logger and ut for loger

0bcd885

sljlp dismissed gongweibao’s stale review via 0bcd885 September 15, 2022 03:07

add ut

f47549e

fuyinno4 previously approved these changes Sep 15, 2022

View reviewed changes

fix

9f950fb

liuTINA0907 previously approved these changes Sep 15, 2022

View reviewed changes

sljlp dismissed stale reviews from liuTINA0907 and fuyinno4 via 9f950fb September 15, 2022 11:52

sljlp mentioned this pull request Sep 15, 2022

[Cherry pick] fleet logger #46087

Merged

sljlp and others added 5 commits September 15, 2022 22:57

Merge branch 'develop' into fleet_logger

7d62cb3

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

3d8efd6

… fleet_logger

fix static_check error when compile twice

059fe59

Merge commit 'refs/pull/46140/head' of https://github.com/PaddlePaddl…

ee5ebcc

…e/Paddle into fleet_logger

rm build/python/paddle in buld_pr_dev

9bd1d60

fuyinno4 previously approved these changes Sep 18, 2022

View reviewed changes

sljlp and others added 2 commits September 18, 2022 12:50

fix apis global scatter/gather

d2c3f39

Fix fleet logger (#4)

8bf0175

* fix apis global scatter/gather * update

sljlp dismissed fuyinno4’s stale review via 8bf0175 September 18, 2022 05:06

XiaoguangHu01 previously approved these changes Sep 18, 2022

View reviewed changes

rm global_scatter & global_gather in utils

92de171

sljlp dismissed XiaoguangHu01’s stale review via 92de171 September 18, 2022 12:36

XiaoguangHu01 previously approved these changes Sep 19, 2022

View reviewed changes

fuyinno4 previously approved these changes Sep 19, 2022

View reviewed changes

liuTINA0907 previously approved these changes Sep 19, 2022

View reviewed changes

sljlp added 2 commits September 19, 2022 14:23

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

e99726f

… fleet_logger

revert fix

667c094

sljlp dismissed stale reviews from liuTINA0907, fuyinno4, and XiaoguangHu01 via 667c094 September 19, 2022 06:35

sljlp added 3 commits September 19, 2022 15:56

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

63e2102

… fleet_logger

recompute

d12a4ab

update

4057027

fuyinno4 approved these changes Sep 19, 2022

View reviewed changes

sljlp requested review from gongweibao, XiaoguangHu01 and liuTINA0907 September 19, 2022 23:25

XiaoguangHu01 approved these changes Sep 20, 2022

View reviewed changes

liuTINA0907 approved these changes Sep 20, 2022

View reviewed changes

jzhang533 reviewed Sep 20, 2022

View reviewed changes

sljlp merged commit 264ad20 into PaddlePaddle:develop Sep 20, 2022

fuyinno4 pushed a commit that referenced this pull request Sep 22, 2022

logger manager (#45909) (#46087)

7eb046c

uniform logger manager in FleetAPI. hidde API under distributed/utils which users don't need.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

logger manager #45909

logger manager #45909

sljlp commented Sep 9, 2022 •

edited

Loading

paddle-bot bot commented Sep 9, 2022

gongweibao left a comment

gongweibao Sep 13, 2022

sljlp Sep 13, 2022

gongweibao Sep 13, 2022

sljlp Sep 13, 2022

gongweibao Sep 13, 2022

sljlp Sep 13, 2022 •

edited

Loading

gongweibao left a comment

gongweibao Sep 14, 2022

sljlp Sep 14, 2022

gongweibao Sep 14, 2022

gongweibao left a comment

gongweibao Sep 14, 2022 •

edited

Loading

sljlp Sep 18, 2022

gongweibao left a comment

XiaoguangHu01 left a comment

XiaoguangHu01 left a comment

XiaoguangHu01 left a comment

jzhang533 left a comment

jzhang533 Sep 20, 2022

sljlp Sep 20, 2022

jzhang533 Sep 20, 2022

sljlp Sep 20, 2022

gongweibao Sep 20, 2022

logger manager #45909

logger manager #45909

Conversation

sljlp commented Sep 9, 2022 • edited Loading

PR types

PR changes

Describe

paddle-bot bot commented Sep 9, 2022

gongweibao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sljlp Sep 13, 2022 • edited Loading

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

gongweibao Sep 14, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gongweibao left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

jzhang533 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sljlp commented Sep 9, 2022 •

edited

Loading

sljlp Sep 13, 2022 •

edited

Loading

gongweibao Sep 14, 2022 •

edited

Loading