Process.run freezes if the subprocess generates a lot of output

[Akirathan]

When invoking System.Process.run with a command that prints a lot to the stdout, it freezes. After some investigation, I have found out that this is not Enso-specific, but this can be reproduced by a simple Java program:

import java.io.IOException;
import java.lang.ProcessBuilder.Redirect;
import java.nio.file.Path;
import java.util.List;

public class Main {

  public static void main(String[] args) throws IOException, InterruptedException {
    var bldr = new ProcessBuilder();
    var lineCount = 20000;
    var cmd = List.of(
        "/usr/bin/python",
        "/home/pavel/scripts/generate_output.py",
        Integer.toString(lineCount)
    );
    var path = Path.of("/tmp/tmp.txt");
    bldr.command(cmd);
    //var redirect = Redirect.to(path.toFile());
    var redirect = Redirect.PIPE;
    bldr.redirectOutput(redirect);
    var before = System.currentTimeMillis();

    var process = bldr.start();
    var retCode = process.waitFor();
    if (retCode == 0) {
      System.out.println("Process finished successfully");
    } else {
      System.out.println("Process failed with code " + retCode);
    }
    var after = System.currentTimeMillis();
    var is = process.getInputStream();
    var bytes = is.readAllBytes();
    System.out.println("Time taken: " + (after - before) + "ms");
    System.out.println("Output length: " + bytes.length + " bytes");
  }
}

With generate_output.py (just prints lines of length 1024 of random characters):

import sys
import random
import string

LINE_LEN = 1024

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python generate_output.py <len>")
        sys.exit(1)
    num_lines = int(sys.argv[1])
    # print num_out random characters to the stdout
    for i in range(num_lines):
        print(''.join(random.choices(string.ascii_uppercase + string.digits, k=LINE_LEN)))

In Java, this process hangs forever. This is because of the pipe redirection. If we change the redirect to a file, then the subprocess exits almost immediately.

Partially explained in:

• https://stackoverflow.com/a/12893987/4816269
• https://stackoverflow.com/questions/71510714/how-to-improve-performance-of-java-process

I have encountered this issue when I was trying to create a subprocess that runs the following command:

tools/performance/engine-benchmarks/bench_download.py -v -s engine --since 2024-01-01 --create-csv

And noticed that when I remove the -v flag, it does not freeze.