Cannot find the stdout/stderr log of a long run job

[zheng-ningxin]

Organization Name: MS internal usage

Short summary about the issue/question:
Hi, I met a problem when I used the OpenPAI to run the training jobs, and I think it may confuse other users, so I raise this issue.
I submitted a task, after the task is completed, I want to check the stdout of this task, unexpectedly found that the output of the task was empty.

Later, it was found that the output of the task was compressed into the log folder, and the website page no longer displayed the output of the task.

I think this is confusing for users, so I raise this issue.
Brief what process you are following:

How to reproduce it:

OpenPAI Environment:

• OpenPAI version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Hardware (e.g. core number, memory size, storage size, GPU type etc.):
• Others:

Anything else we need to know:

[fanyangCS]

@zheng-ningxin , please share us the job url.

[zheng-ningxin]

Hi~ the url is at: https://ne.openpai.org/job-detail.html?username=nizhen&jobName=nizhen___mobilenet_v2_l1_07_0001

[fanyangCS]

@Binyang2014 , pls take a look.

[Binyang2014]

This is caused by log-rotate . Current log-rotate config is:

{
    nomail
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
    maxsize 300M
    maxage 30
}

It will auto compress the log if size is more than 300M or exist more than one week. This work start at 7.28, which is one-week ago.
So log is rotated.

[fanyangCS]

This is caused by log-rotate . Current log-rotate config is:

{
    nomail
    weekly
    rotate 4
    compress
    missingok
    notifempty
    copytruncate
    maxsize 300M
    maxage 30
}

It will auto compress the log if size is more than 300M or exist more than one week. This work start at 7.28, which is one-week ago.
So log is rotated.

We need a complete design for log management. including view, archive, etc.

[zheng-ningxin]

Yes~ I understand that log-rotate is necessary. However, I think, after the log-rotation, the website should still show part of the output log, or inform the users that their output has been compressed.

[Binyang2014]

According to https://manpages.debian.org/jessie/logrotate/logrotate.8.en.html
log will be rotated at first day of week. So every Sunday, the old log will be rotated.