catalog-bsp site url is indexed by search engine

[FuhuXia]

catalog-bsp.data.gov is not supposed to be a public site. However, we started to see Google is crawling catalog site at this url.

How to reproduce

Go to https://www.google.com/search?q=site%3Acatalog-bsp.data.gov

Expected behavior

No results should be returned

Actual behavior

About 42,600 results returned.

[adborden]

@FuhuXia do you have ideas on the best way to fix? Since we don't actually want to serve anything at catalog-bsp.data.gov and essentially only use it as a CNAME for the BSP origin, I'm thinking we should just have CKAN restrict Host to catalog.data.gov via Apache config. That way CKAN will return 400s for any requests with Host: catalog-bsp.data.gov, but any valid requests (Host: catalog.data.gov) will respond 200 as usual.

Then there is a one-time cleanup to tell search engines to purge their indexes.

[FuhuXia]

Yes, that will work. Still, using host http://localhost should be allowed so we don't have to spoof ip for local testing. So the change can happen on port 443 conf only, or we do allowing anything except catalog-bsp.data.gov instead of blocking anything except catalog.data.gov.

[adborden]

I prefer whitelists over blacklists for this kind of thing. It's usually more secure. So only allow localhost and catalog.data.gov. Otherwise folks can get creative in using our app to serve arbitrary hosts which is problematic.

For port 443, I wouldn't tackle that yet. There's an outstanding issue to move CKAN instances to serve exclusively on HTTPS, but that involves some BSP configuration #570

[FuhuXia]

Cool. Whitelisting catalog.data.gov sounds good to me as long as localhost is allowed.

[adborden]

Just for reference, we already implemented this for nginx hosts and define the whitelist in a single variable.

[mogul]

A short-term fix would be to add a robots.txt that only appears for catalog-bsp.data.gov to tell search engines not to index.
https://support.google.com/webmasters/answer/6062608?hl=en

[mogul]

Looks like we'd have to add a noindex directive as well.
https://developers.google.com/search/reference/robots_meta_tag

[adborden]

Using robots.txt or a meta tag would require some changes to CKAN. Setting the Host whitelist in apache is theoretically a one-line change to the apache site config.

[mogul]

We can't set up a directive in Apache to serve it?

[FuhuXia]

I think configuring apache to restrict hostname to catalog is good solution and low level of effort to implement. Using robots, or noindex directive, either in meta or apache, are same amount of work, if not more. Also there is a risk that some crawlers might not follow the rule of directives.

[adborden]

That's fair, it's the same amount of work to update the apache config. There is no content hosted at catalog-bsp.data.gov, so having a robots.txt when there's no content seems unnecessary unless it helps search engines to remove content. I believe that returning a 400 will have the same effect though.

Even if we add robots.txt, we still want to whitelist the domains since its also a potential security risk to accept arbitrary Hosts.