GH-15233: [C++] Correct thread contention in s3fs

[EpsilonPrime]

Avoids thread contention which can occur when using CopyFiles to copy files from a filesystem other than S3 to the S3FS. This is accomplished by not trying to complete the merging of a potentially multipart file asynchronously while already in an I/O thread (instead that work is completed in the already active thread).

• Closes: [Python] pyarrow.fs.copy_files hangs indefinitely #15233

[github-actions]

• Closes: [Python] pyarrow.fs.copy_files hangs indefinitely #15233

[github-actions]

⚠️ GitHub issue #15233 has been automatically assigned in GitHub to PR creator.

[westonpace]

I think CloseAsync is expected to act as an async method. By pushing the writes to the foreground it is no longer acting as an async method.

This is similar to the issues we ran into on the read path. The solution was to make everything async by default and the only time we wrap async with sync is at the API layer (e.g. on that first call from the user to a sync method). This allows us to keep one async path for all the "internals" and only mirror the two paths at the edges.

Instead, could we change CopyFile (and CopyFiles) to use async methods? Or create CopyFileAysnc and CopyFilesAsync (and change CopyFiles to return CopyFilesAsync.result()?

[amol-]

@EpsilonPrime are you still interested in moving this forward? It looks like there are some failing checks and some review feedbacks to address

[EpsilonPrime]

@EpsilonPrime are you still interested in moving this forward? It looks like there are some failing checks and some review feedbacks to address

@amol- Yes, I'm still working on this but my alternate version of the solution (using futures throughout) is not yet working.

[zeroshade]

@EpsilonPrime Just checking back in on this, are you still working on this and we should keep it open? Do you need any assistance in getting it working?

[EpsilonPrime]

Solved by another PR.