websites: 301-based websites blocking

[yanmaani]

(I'm not sure if this is an issue with the lists or with the explorer, but this MAY be a duplicate of citizenlab/test-lists#940.)

If a domain is accessed over HTTP, and it responds by 301'ing to a different domain, that seems like some indication of blocking, or at least of the site owner wanting to fake it (see: Daily Mail incident).

I think it may be prudent to detect cases where a domain, accessed over HTTP, responds by a 301/302 to a different domain that isn't a subdomain of the initial origin, and flag those in some way:

Here's two examples:
NL - GET http://rt.com -> GET https://freedom.nl/landingpage/russia-today-en-sputnik-zijn-geblokkeerd-op-last-van-het-om
UA - GET http://rt.com -> GET https://cdn.kyivstar.ua/sites/default/files/lock_sites/index.html

Neither freedom.nl nor cdn.kyivstar.ua are subdomains of rt.com.

[bassosimone]

Thank you for letting us know about this! I am going to flag this issue as a data quality issue and add it to the list of actions to be considered while redesigning our network measurement methodology.

[hellais]

I believe we capture this already in our new 0.5 methodology, but if not we can reopen a new issue.