Improve ocrd_network logging

[MehmedGIT]

This PR improves the logging mechanism of the ocrd_network module.

Additions:

• A separate logging dir tree structure for the modules (processing servers, processing workers, processor servers, mets servers, processing jobs). Configurable with an env variable.
• Processing job-level logging - each job is logged into a separate file with format {job_id}.log
• Processing job-level logging file paths are added to the Job models and preserved in the database.
• The ocrd_network logging is based on the format provided in ocrd_utils
• Support env variables for setting the root directory for logging/sockets
• An endpoint for getting the log file of a processing job of a processor

Feel free to recommend suggestions.

[kba]

LGTM, just a quick read-through, I'll contribute ideas once I have some more time for it.

[joschrew]

I have tested and it seems to work fine. I think it is a huge improvement, all structured now and much better to find the right logfile now.
What imho could be added is:

• make logfile and and mets-server basedir configurable with env. For example when I test locally I use /data for the mets-server (this is better for my local setup with docker-compose) and when this was configurable it would be easier for me to set it up. And I could use /data in the docker-compose-setup as well and would not have to volume mount two directories (now I am mounting /data and /tmp).
• add a logging endpoint for the processors: as now we have the log-file-path in the database it would be very easy to add and endpoint to the processing server to retrieve the log of a processor through http

[MehmedGIT]

make logfile and and mets-server basedir configurable with env. For example when I test locally I use /data for the mets-server (this is better for my local setup with docker-compose) and when this was configurable it would be easier for me to set it up. And I could use /data in the docker-compose-setup as well and would not have to volume mount two directories (now I am mounting /data and /tmp).

Resolved with 464f8ea

add a logging endpoint for the processors: as now we have the log-file-path in the database it would be very easy to add and endpoint to the processing server to retrieve the log of a processor through HTTP

Resolved with c81ffcc

[kba]

Looks good, much cleaner and being able to get the log via the API is excellent.

Ready for release IMHO. Can you summarize the changes for the release notes?

[MehmedGIT]

Can you summarize the changes for the release notes?

It's on top. I have updated the initial post.

[MehmedGIT]

@kba, I forgot to update the ocrd --help section - thanks for adding that.

Considering this, I was wondering why should we add the environment variables manually to the help section. I think it would be possible to iterate over all variables inside the OcrdEnvConfig and print their descriptions. Based on prefix values we could also decide what to print, e.g., OCRD_NETWORK prefix would print only those environment variables.

[kba]

Considering this, I was wondering why should we add the environment variables manually to the help section. I think it would be possible to iterate over all variables inside the OcrdEnvConfig and print their descriptions. Based on prefix values we could also decide what to print, e.g., OCRD_NETWORK prefix would print only those environment variables.

The main reasons I did the --help section manually is order of the output and that I found it difficult to get the \b escapes right. It is a source of inconsistency though, I'll think about finding a better way.