Add folder-based partitioning for s3 scan source

[graytaylor0]

Description

This change adds support for the s3 scan source to partition based on folders instead of objects. This means that, unlike object partitions, folder partitions will never be marked as complete, and each node / worker that grabs a folder partition will list and process objects in that folder, then give it up. The use case for this is order guarantee within S3 folders, as well as aggregations within S3 folders, as the same worker / node will process objects from the same S3 folder. To enable this mode, one must set the following configuration.

source:
  s3:
    scan:
      folder_partitions:
        # The depth of the folders to partition on. For example, given object keys "folder-1/folder-3/folder-4/"  and "folder-1/folder-2/folder-5/", a depth of 1 would result in one partition of "folder-1/", while a depth of 2 would result in 2 partitions, "folder-1/folder-3/" and "folder-1/folder-2/". If an object does not contain enough depth, it will not be included in the scan.
        depth: 2 # Defaults to 1
        
        # The max number of objects to process before giving up the folder partition. If the number of objects in the folder is less than this maximum, then all of the objects in the folder will be processed before giving up the partition
        max_objects_per_ownership: 10 # Defaults to 50

Check List

• New functionality includes testing.
• New functionality has a documentation issue. Please link to it in this PR.
  • New functionality has javadoc added
• Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[oeyh]

Minor: we should probably be consistent on the naming. I see both prefix partition and folder partition being used.

[graytaylor0]

good catch this was an outdated message where I initially had a mode of PREFIX or FULL_OBJECT_KEY

[dinujoh]

Need to bump this up.

[dinujoh]

This is duplicate. Can be moved to beginning to the method.

[dinujoh]

This method can be split into smaller methods for readability. We can move this while loop to different method.