Add prefix restrictions on caches

[Saartank]

Description

This PR introduces a mechanism to define restrictions on the prefixes a cache is allowed to serve. The list of allowed prefixes can now be configured in the custom_fields column of the registry database as a JSON list, for example:

{"AllowedPrefixes": ["/ns1", "/ns1/ns2"]}

• The cache will only serve requests for the specified prefixes in the AllowedPrefixes list.
• If the AllowedPrefixes key is absent in custom_fields, the default behavior remains unchanged (i.e., the cache will serve all prefixes).
• Note: If /ns1 is in AllowedPrefixes, it does not imply that the cache is allowed to serve /ns1/ns2.

---

Implementation Details

1. New Registry Endpoint:

   • A new endpoint, /caches/allowedPrefixes, is added to the registry.
   • This endpoint returns a mapping of cache hostnames to their respective allowed prefixes.

2. In-Memory Data Management:

   • A new goroutine is implemented in Director to maintain the allowed prefixes data for caches in memory by periodically querying the new registry endpoint.
   • The in-memory data has a 15-minute timeout. The goroutine:
     • Queries the registry API at a configurable interval (Director.RegistryQueryInterval).
     • Falls back to a 1-second interval if the data is uninitialized or expired, switching back to the regular interval upon successful retrieval.

3. Prefix Filtering in Server Ads:

   • While processing a server ad in the director, the allowed prefixes are filtered using the in-memory data.
   • Fail-safe behaviors:
     • Fail-Close Mechanism: Reject the server advertisement if the in-memory data is outdated (older than 15 minutes).
     • Sync Grace Period: If the in-memory data is uninitialized, wait up to 3 seconds for the registry and director to sync before rejecting the advertisement.

---

Additional Information

• This implementation introduces a backward compatibility issue between central services. The new director with this mechanism will not work with an older registry that does not include the /caches/allowedPrefixes endpoint.

• A new endpoint, /caches/allowedPrefixes, has been created in the registry. The top-level endpoint in the registry is defined as /*wildcard, which is used to retrieve information about namespaces. Due to this change, if a cache in the federation is named allowedPrefixes, it would not work. While this is not seen as a significant issue, middleware can be implemented to address it if necessary.

• After merging this PR, I will create a new issue to enable users to update the caches data in the registry through its UI, allowing them to add the list of allowed prefixes for a cache to the database.

[Saartank]

This PR does not implement a mechanism for inputting values for prefixes that certain caches should be allowed to serve. The idea is for an admin to configure this through the UI, which will be implemented in a future PR.

For now, to test this feature, the SQLite database itself needs to be modified. Use the following SQLite command to add custom fields:

UPDATE namespace
SET custom_fields = json_set(COALESCE(custom_fields, '{}'), '$.AllowedPrefixes', json_array('/ns1', '/ns2/ns3'))
WHERE prefix = "/caches/cache-hostname";

With this added, the cache with hostname cache-hostname will only be allowed to serve prefixes /ns1 and /ns2/ns3.

[jhiemstrawisc]

I know we discussed a few other things offline that might generate more work (like the fact that we won't have a way to configure this via our pelican.yaml if we rely solely on the web UI), but I decided to finish the review so you can incorporate feedback in the next round.

[jhiemstrawisc]

LGTM.