Rejected Advertisement Metric

[patrickbrophy]

This PR addresses issue #1774. This metric is a simple counter that tracks the number of times the director has rejected an advertisement from an origin or cache.

[jhiemstrawisc]

I'm a bit worried about this as is. Assume we set up an origin that, for some reason, hasn't yet been approved by the federation administrators. The origin is hardcoded to advertise with the Director once every minute, which will cause this metric to start ticking up pretty fast, but with no way of telling:
a) what server is having advertisement issues
b) whether that server eventually succeeded but another server started failing at the same time

Is there a way you can restructure the metric such that we'd have an easy way to answer "Which servers, identified by their hostname, produced the most advertisement failures in the last week?" The challenge here would be finding a way to treat the hostnames safely, since those are an arbitrary part of some JSON we get.

Thoughts

[patrickbrophy]

For part a, I think that the server name could be a label for this metric. This can be sourced from the advertisement. For part b, I think this could be achievable through a query. For the latter part of your comment:

Is there a way you can restructure the metric such that we'd have an easy way to answer "Which servers, identified by their hostname, produced the most advertisement failures in the last week?"

I think you could achieve this via a query like the following:
topk(10, sum_over_time(pelican_director_rejected_advertisements_total[1w]) by (hostname))