Generate log files for Python Profiling cli

[nartal1]

This fixes #1363

In PR-#1269, support was added to generate stderr log files for qualification tool and missed profiling tool. This PR adds support to generate log files for profiling tool as well. While doing so, refactor was done to distinguish between profiling and qualification log.

Before this change:

2024-10-01 14:12:00,954 ERROR rapids.tools.submit.onpremLocal: Error removing temporary log4j properties file: remove: path should be string, bytes or os.PathLike, not NoneType

After this change:
New file created rapids_4_spark_profile_stderr.log under rapids_4_spark_profile/ directory

2024-10-01 14:15:10,345 INFO rapids.tools.submit.onpremLocal: Temporary log4j properties file removed: /tmp/tmps69jwl4g.properties

[amahussein]

Interesting that no change on the scala side. I remember the log file was created by the QualificationOutputWriter. If that's the case, it means that we cannot generate log file by simply running profiling jar cmd. Or do I miss something?

[nartal1]

Interesting that no change on the scala side. I remember the log file was created by the QualificationOutputWriter. If that's the case, it means that we cannot generate log file by simply running profiling jar cmd. Or do I miss something?

We can generate log file by providing the log4jc configuration in java command. Since the sample log4j.properties is checked in, we can modify the output file path, output file name and other parameters before running the java command.

java -Dlog4j.configuration= /home/dev/spark-rapids-tools/user_tools/src/spark_rapids_pytools/resources/dev/log4j.properties -cp rapids-4-spark-tools_2.12-24.08.3-SNAPSHOT.jar:$SPARK_HOME/jars/* com.nvidia.spark.rapids.tool.profiling.ProfileMain /home/dev/tmp_dir/testing_eventlog

[parthosa]

Thanks @nartal1. Tested this. LGTME

[amahussein]

Thanks @nartal1 !
LGTME