too many folders in root of bucket

[bunnybones1]

During a quick inspection with cyberduck, the root bucket took a long time to list (with warnings):

It might be prudent to chop our prefixes into 2 character chunks, inspired by how git organizes objects:

[pkieltyka]

alternatively, instead of making it a prefix hash, we could make it a suffix hash on the filename/ext...

cc: @dencoded @c2h5oh

[c2h5oh]

@pkieltyka

• Suffix is no-go: S3 separates files into shards based on file path prefix and is not smart about it - it splits into equal character ranges, not equal file count.
  Each shard supports up to 100 iops.

• using 1-2 cards of file hash as top level directory does, but remember it's base64 and not hex, so 1 char = 64 directories, 2 chars = 4096 directories.

[dencoded]

btw, they claimed to fix this issue with path prefix/shard capacity but people are still complaining so I wouldn't trust 100% they fixed it

[c2h5oh]

@dencoded when? I know that 14 months ago that was still the case (I was doing a large s3 migration with almost 16M bucket objects).