Change sub-resource referrer policy to report 3p origin

[pes10k]

Currently Brave does a number of things to prevent privacy loss through the referrer header

https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)#modified-features-and-functionality

This has significant webcompat cost, which we try to mitigate with an exception list.

Since Brave made this decision, Chromium has gotten tighter with its handling of the referrer header. Basically, they have a default of strict-origin-when-cross-origin; sites who want the previous, more permissive behavior need to opt in.

To reduce the webcompat cost, but still aggressively protect user privacy, we should:

1. Use Chromium's behavior as the default
2. Respect when sites specify a more restrictive policy (e.g. no-referrer)
3. Treat less restrictive policies as strict-origin-when-cross-origin
4. Never send referrer for insecure requests

E.g.

1. Set effective policy to strict-origin-when-cross-origin.
2. Is this a cross-site navigation?
   • GET / HEAD?
     • If yes, change effective policy to no-referrer
   • Otherwise:
     • Requested policy of origin?
       • If yes, change effective policy to strict-origin
     • Requested policy of no-referrer, same-origin, or strict-origin?
       • If yes, change effective policy to the requested policy.
3. Is this a sub-resource or iframe request?
   • Requested policy of origin?
     • If yes, change effective policy to strict-origin
   • Requested policy of no-referrer, same-origin, or strict-origin?
     • If yes, change effective policy to the requested policy.

(previously expressed as)
* if request is HTTP -> no-referrer
* elif {no-referrer-when-downgrade,unsafe-url, origin, origin-when-cross-origin} -> strict-origin-when-cross-origin
* elif {no-referrer, same-origin, strict-origin} -> no change
* else strict-origin-when-cross-origin

Interaction with Shields
This intervention should be tied to the cookie setting toggle in shields. When shields are down, or the cookies setting for the top level URL is "All cookies allowed", Brave should default to Chromium behavior. When the cookie setting in shields is "Cross-site cookies blocked" or "All cookies blocked", the policy described in this issue should apply.

[pes10k]

Note, re privacy confab discussion, there would be no changes to current policy for navigations.

There is also a medium-to-long term goal of building crawls to build a known-deny list, distinct from this change

[fmarier]

These pages will need to be updated once this is fixed:

• https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)#modified-features-and-functionality
• https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections#privacy-protection-enabled-regardless-of-whether-fingerprinting-protection-mode-is-on
• https://github.com/cookie-status/cookie-status-dev/blob/master/content/brave/_index.md
• https://github.com/cookie-status/cookie-status-dev/blob/master/content/_index.md

[pes10k]

Also, once this is complete, revisit #7933 to see if it still needs an independent fix

[fmarier]

With the change to top-level POST navigations, #5075 and #9470 might get resolved.

[pes10k]

Manual tests for QA https://dev-pages.bravesoftware.com/referrer/index.html and https://dev-pages.brave.software/index.html (same tests, but both origins are needed for some of the tests)

[fmarier]

I updated my test pages at https://fmarier.github.io/brave-testing/referrer-spoofing.html (including
new form test cases), https://referrer.fmarier.org/redirects.html, and https://referrer.fmarier.org/samesite.html.

I also confirmed that all of the test cases on https://dev-pages.brave.software/referrer/index.html and https://fmarier.com/referrer/strict-origin.html work.

[fmarier]

I updated the following wiki pages:

• https://github.com/brave/brave-browser/wiki/Deviations-from-Chromium-(features-we-disable-or-remove)#modified-features-and-functionality
• https://github.com/brave/brave-browser/wiki/Fingerprinting-Protections

[LaurenWags]

Marking as QA/Blocked for same reason as #6657 (comment)

[pes10k]

@LaurenWags which issues / tests on https://dev-pages.brave.software/referrer/index.html and https://fmarier.com/referrer/strict-origin.html are not working for you?

[LaurenWags]

1. On https://fmarier.github.io/brave-testing/referrer-spoofing.html I don't know that some of the referrer stuff is working as expected:
   a. under Form-submission, when I click on the same-origin submission button, i get a 405 not allowed message

b. under iframe navigations the cross-origin navigation and cross-origin navigation (redirected via meta) initially show as expected. However, if I first click on cross-origin navigation link under Top-level navigations, and I navigate back to https://fmarier.github.io/brave-testing/referrer-spoofing.html they do not show expected result until I refresh the page. is this ok?

2. On https://referrer.fmarier.org/redirects.html (which I see now isn't in the PR but is listed in Change sub-resource referrer policy to report 3p origin #8696 (comment)) the links at the top do not show blank, I see Referrer: https://referrer.fmarier.org/ on each page, expected? (boxes at the bottom of the page all work as expected)

[fmarier]

a. under Form-submission, when I click on the same-origin submission button, i get a 405 not allowed message

Now fixed. This was due to GitHub apparently starting to block POST requests.

b. under iframe navigations the cross-origin navigation and cross-origin navigation (redirected via meta) initially show as expected. However, if I first click on cross-origin navigation link under Top-level navigations, and I navigate back to https://fmarier.github.io/brave-testing/referrer-spoofing.html they do not show expected result until I refresh the page. is this ok?

Partially fixed (the first test case) by switching away from whatismyreferrer.com. I believe we may have an underlying bug here ("cross-origin navigation (redirected via meta)").

On https://referrer.fmarier.org/redirects.html (which I see now isn't in the PR but is listed in #8696 (comment)) the links at the top do not show blank, I see Referrer: https://referrer.fmarier.org/ on each page, expected? (boxes at the bottom of the page all work as expected)

The links at the top are out-of-scope for this PR. The boxes, which work fine, is all that we should be testing against this PR.

[LaurenWags]

Thank you @fmarier! I'll get this all re-checked and log a follow-up issue 😄

[LaurenWags]

Verified passed with

Brave | 1.12.108 Chromium: 84.0.4147.105 (Official Build) (64-bit)
-- | --
Revision | a6b12dfad6663f13a7e16e9a42a6a4975374096b-refs/branch-heads/4147@{#943}
OS | macOS Version 10.14.6 (Build 18G3020)

• Verified test plan from 8696: Change referrer blocking in Brave brave-core#5613
• Confirmed tests on https://fmarier.github.io/brave-testing/referrer-spoofing.html work as expected. Logged [Desktop] sometimes referrer doesn't display properly until page is reloaded - follow up to 8696 #11119 as follow up noted in Change sub-resource referrer policy to report 3p origin #8696 (comment).
• Confirmed tests on https://referrer.fmarier.org/samesite.html work as expected.
• Confirmed tests on https://fmarier.com/referrer/strict-origin.html work as expected.
• Confirmed tests on https://referrer.fmarier.org/redirects.html (bottom boxes only per Change sub-resource referrer policy to report 3p origin #8696 (comment)) work as expected.
• Confirmed tests on https://dev-pages.brave.software/referrer/index.html work as expected.

---

Verification passed on

Brave	1.12.108 Chromium: 84.0.4147.105 (Official Build) (64-bit)
Revision	a6b12dfad6663f13a7e16e9a42a6a4975374096b-refs/branch-heads/4147@{#943}
OS	Ubuntu 18.04 LTS

• Verified test plan from 8696: Change referrer blocking in Brave brave-core#5613
• Confirmed tests on https://fmarier.github.io/brave-testing/referrer-spoofing.html work as expected. Encountered [Desktop] sometimes referrer doesn't display properly until page is reloaded - follow up to 8696 #11119
• Confirmed tests on https://referrer.fmarier.org/samesite.html work as expected.
• Confirmed tests on https://fmarier.com/referrer/strict-origin.html work as expected.
• Confirmed tests on https://referrer.fmarier.org/redirects.html (bottom boxes only per Change sub-resource referrer policy to report 3p origin #8696 (comment)) work as expected.
• Confirmed tests on https://dev-pages.brave.software/referrer/index.html work as expected.

---

Verification passed on

Brave | 1.12.108 Chromium: 84.0.4147.105 (Official Build) (64-bit)
-- | --
Revision | a6b12dfad6663f13a7e16e9a42a6a4975374096b-refs/branch-heads/4147@{#943}
OS | Windows 10 OS Version 1903 (Build 18362.959)

• Verified test plan from 8696: Change referrer blocking in Brave brave-core#5613
• Confirmed tests on https://fmarier.github.io/brave-testing/referrer-spoofing.html work as expected. Encountered [Desktop] sometimes referrer doesn't display properly until page is reloaded - follow up to 8696 #11119
• Confirmed tests on https://referrer.fmarier.org/samesite.html work as expected.
• Confirmed tests on https://fmarier.com/referrer/strict-origin.html work as expected.
• Confirmed tests on https://referrer.fmarier.org/redirects.html (bottom boxes only per Change sub-resource referrer policy to report 3p origin #8696 (comment)) work as expected.
• Confirmed tests on https://dev-pages.brave.software/referrer/index.html work as expected.

[srirambv]

Verification passed on OnePlus 6T with Android 10 running 1.12.111 x64 build

• Verified test plan from 8696: Change referrer blocking in Brave brave-core#5613
• Verified tests on https://fmarier.github.io/brave-testing/referrer-spoofing.html and works as expected
• Verified tests on https://referrer.fmarier.org/samesite.html and works as expected
• Verified tests on https://fmarier.com/referrer/strict-origin.html and works as expected
• Verified tests on https://referrer.fmarier.org/redirects.html and works as expected
• Verified tests on https://dev-pages.brave.software/referrer/index.html and works as expected

---

Verification passed on Samsung Tab A with Android 10 running 1.12.111 x64 build

• Verified test plan from 8696: Change referrer blocking in Brave brave-core#5613
• Verified tests on https://fmarier.github.io/brave-testing/referrer-spoofing.html and works as expected
• Verified tests on https://referrer.fmarier.org/samesite.html and works as expected
• Verified tests on https://fmarier.com/referrer/strict-origin.html and works as expected
• Verified tests on https://referrer.fmarier.org/redirects.html and works as expected
• Verified tests on https://dev-pages.brave.software/referrer/index.html and works as expected