Separation of shard and cache directories by endpoint.

[hoytak]

This PR separates out the shard and cache directories by endpoint, ensuring that shards and chunks do not mix between endpoints (E.g. production and development, etc.).

[assafvayner]

I'm slightly worried about the chance we get different directories in the case of huggingface.co vs hf.co since both are valid, but we'll see if that causes an issue down the line.

[hoytak]

I'm slightly worried about the chance we get different directories in the case of huggingface.co vs hf.co since both are valid, but we'll see if that causes an issue down the line.

True, though I think that could be handled by a mapping layer on the python side.

[seanses]

I'm slightly worried about the chance we get different directories in the case of huggingface.co vs hf.co since both are valid, but we'll see if that causes an issue down the line.

@assafvayner The endpoint here is actually CAS endpoint not the Hub endpoint.