[feature] Add meta robots tag; allow robots to index profile card if user is Discoverable

[tsmethurst]

This PR adds more granular control over which endpoints robots are allowed to index.

By default, all web pages now have the following tag added to them:

<meta name="robots" content="noindex, nofollow">

This restricts search engines to not being able to index anything on the page.

However, if a user is Discoverable, then some limited indexing will be allowed of their profile page (but not their statuses or anyone else's statuses either).

In this case, the returned tag will be:

<meta name="robots" content="nofollow, noarchive, nositelinkssearchbox, max-image-preview:standard">

This allows the user's profile to be indexed, but not archived, and links will not be followed.

Currently, this won't make much difference for users, since by default users are not Discoverable, and there's no way of changing this right now aside from manually changing a value in the database. But in future it'll come in handy when users actually want their profiles to be discoverable by Google etc etc.

To support this, the robots.txt response has been reworked to Disallow all api/auth/well-known/file/s2s/panel stuff, instead of a blanket disallow of everything.

[tsmethurst]

(converted this to a draft cuz we should merge it some time after 0.5.0 is out, not immediately)