Cannot get the main process in vizviewer

[MarcoMagl]

Hi everyone,

I would like to use viztracer to better understand why my code does not scale well with multiprocessing.
To do so, I started with a simple example.

Here is the code:

def my_function(a):
    inside_circle = 0
    num_samples = 0

    start_time = time.time()  # Capture the start time
    duration = 3  # Duration in second

    while True:
        x, y = random.random(), random.random()
        if x**2 + y**2 <= 1:
            inside_circle += 1
        num_samples += 1
        ellapsed = time.time() - start_time
        if ellapsed > duration:
            break    

    pid = os.getpid()
    print(f"Running the function for the {a}th time - Process ID: {pid} took {ellapsed} seconds\n")

    return 4 * inside_circle / num_samples

@log_sparse
def main():
    with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
        futures = [executor.submit(my_function, i) for i in range(5)]
        _ = [f.result() for f in futures]

If I put the @log_sparse decorator above the main function, I do not see the processes from the pool of processes that have been created.

On the contrary, if I put the log_sparse decorator on top of my_function, I do not see what the main process is doing and what happens in between the call of that function and this is what would be really interesting to me!

I launch the profiling from the CLI with:

viztracer --pid_suffix --output_dir ./outputDir --log_sparse test.py

What am I missing?

Thanks in advance for your help :)

[MarcoMagl]

Another couple of questions which come to my mind:

Where can I found information on what vizviewer shows?
I would like to understand:

• what can we see on top (black blocks)?
• what does the green blocks represent? Indeed the function lasts approximately 3 seconds (as I have annotated) so I do not know to what corresponds what we can see below.

Thanks in advance!

[gaogaotiantian]

First of all, if you want to understand your program, you probably should not use log_sparse. That's for cases where you are only interested in the time spent on a specific function. log_sparse by default does not log children functions. So if you put that to main(), it'll only log main(), not any other function calls. That's what sparse means.

As for the UI, it's perfetto so you can check their docs out. I believe all the colors correspond to some CPU usage. The black bar is probably the number function calls starting at that time period. The green one on the process is normally the frequency of function calls as well. Because you only have too few calls, they might not quite make sense. They will be if your trace file is more comprehensive.

[MarcoMagl]

Thanks for the reply.

I used log_sparse in order to limit the size of the output.

Even on a toy example without log_sparse, I launched simply with vitracer, I was gettting this kind of output:

Total Entries: 1000006                                                          

Circular buffer is full, you lost some early data, but you still have the most recent data.
    If you need more buffer, use "viztracer --tracer_entries <entry_number>"
    Or, you can try the filter options to filter out some data you don't need
    use --quiet to shut me up

Use the following command to open the report:

Now I changed the way I launched to ignore all functions that are frozen and it's better:

viztracer --ignore_c_function --ignore_frozen test_calculate_demand.py --parallel-module none

Thank you!