Prune data in meta store for deleted shards

[corylanou]

Problem Statement

Currently, when a shard is deleted, it is simply marked with a timestamp in the DeletedAt field.

For long running instances, this can cause a build up of shard infos in the data store that create unnecessary garbage when being cloned as well as unnecessary cycles during searches of the shards.

Proposal

• Each data node that successfully drops the shard will remove itself from the owner group of the shard.
• A periodic service that runs every 30 minutes will prune all shards from the meta store that have no owners, or the DeletedAt is more than 2 weeks old.

[jwilder]

Sounds good. I think the periodic service should be the current the retention service that already already does most of the other tasks.