[Bug]: FileSystems have a high risk of leaking resources due to the repeated invocation of FileSystems.setDefaultPipelineOptions

[mosche]

What happened?

The underlying issue here is the repeated initialization of filesystems triggered in SerializablePipelineOptions (#18430).
Every call to FileSystems.setDefaultPipelineOptions() will silently replace any existing previous file systems without cleaning up resources.

In the case of S3FileSystem this can leak threads if the file system was already in use.
The simple test below shows the every increasing number of threads in the system.

public class S3FileSystemTest {
  

  @Test
  public void testSetDefaultPipelineOptions() throws Exception {
    S3Options s3Options = s3Options();
    s3Options.setAwsCredentialsProvider(StaticCredentialsProvider.create(AwsBasicCredentials.create("key", "secret")));

    for (int i = 0; i < 1000; i++) {
      FileSystems.setDefaultPipelineOptions(s3Options);
      FileSystems.match(ImmutableList.of("s3://bucket/key*"));
      System.out.println("Number of threads " + Thread.activeCount());
    }
  }
}  

This is particularly bad with some runners, e.g. see #17645:

A normal Go Flink ValidatesRunner execution would contain the logging line from this class nearly 80,000 times, making it much more difficult to read the logs.

Issue Priority

Priority: 3 (minor)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner