Manual test run on macOS (Intel) for 1.21.x Release #1 #14199

kjozwiak · 2021-02-17T05:00:16Z

Installer

About pages

Verify that both chrome:// and about:// forward to brave:// (run through several internal pages)

Importing

Verify that you can import History, Favorites/Bookmarks and Passwords from Google Chrome
Verify that you can import History, Favorites/Bookmarks, Passwords, Search Engines and Autofill from Firefox
Verify that you can import Favorites/Bookmarks from Microsoft Edge
Verify that importing bookmarks using Bookmark HTML File retains the folder structure on a clean profile

Context menus

Verify Block element via selector removes a CSS element from page without reloading
Verify Clear all CSS rules for this sites removes the blocked element after page reload
Verify Clear all CSS rules for all sites removes the rule and loads all elements for all pages

Extensions/Plugins

Verify pdfium, Torrent viewer extensions are installed automatically on fresh profile and cannot be disabled (they don't show up in brave://extensions)
Verify older version of an extension gets updated to new version via Google server
Verify older version of an extension gets updated to new version via Brave server
Verify that magnet links and .torrent files correctly open WebTorrent and you're able to download the file(s)
- Tip: Free torrents available via https://webtorrent.io/free-torrents

Chrome Web Store (CWS)

Verify that installing https://chrome.google.com/webstore/detail/adblock-plus-free-ad-bloc/cfhdojbkjhnklbpkdaibdccddilifddb from CWS displays the Brave has not reviewed the extension. warning via the "Add Extension" modal
Verify that installing https://chrome.google.com/webstore/detail/lastpass-free-password-ma/hdokiejnpimakedhajhdlcegeplioahd from CWS doesn't display the Brave has not reviewed the extension. warning via the "Add Extension" modal

PDF

Test that you can print a PDF
Test that PDF is loaded over HTTPS at https://basicattentiontoken.org/BasicAttentionTokenWhitePaper-4.pdf
Test that PDF is loaded over HTTP at http://www.pdf995.com/samples/pdf.pdf
Test that https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.105.6357&rep=rep1&type=pdf opens without issues

Widevine

Verify Widevine Notification is shown when you visit Netflix for the first time
Test that you can stream on Netflix on a fresh profile after installing Widevine
Verify Widevine Notification is shown when you visit HBO Max for the first time
Test that you can stream on HBO Max on a fresh profile after installing Widevine

Geolocation

Check that https://browserleaks.com/geo works and shows correct location
Check that https://html5demos.com/geo/ works but doesn't require an accurate location

Crash Reporting

Check that loading brave://crash & brave://gpucrash causes the new tab to crash
Check that brave://crashes lists the Uploaded Crash Report ID once the report has been submitted
Verify the crash ID matches the report on Backtrace using _rxid equal [ value ]

Bravery settings

Verify that HTTPS Everywhere works by loading http://https-everywhere.badssl.com/
Turning HTTPS Everywhere off and Shields off both disable the redirect to https://https-everywhere.badssl.com/
Verify that toggling Ads and trackers blocked works as expected
Visit https://testsafebrowsing.appspot.com/s/phishing.html, verify that Safe Browsing (via our Proxy) works for all the listed items
Visit https://www.blizzard.com and then turn on script blocking, page should not load.
Test that 3rd party storage results are blank at https://jsfiddle.net/7ke9r14a/9/ when 3rd party cookies are blocked and not blank when 3rd party cookies are unblocked
Test that https://mixed-script.badssl.com/ shows up as grey not red (no mixed content scripts are run)

Fingerprint Tests

Visit https://jsfiddle.net/bkf50r8v/13/, ensure 3 blocked items are listed in Shields. Result window should show got canvas fingerprint 0 and got webgl fingerprint 00
Test that audio fingerprint is blocked at https://audiofingerprint.openwpm.com/ only when Block all fingerprinting protection is on
Test that Brave browser isn't detected via user-agent, on https://www.whatismybrowser.com. The site will say "Looks like Brave on [Windows/Linux/macOS]", and yet, "But it's announcing that it's Chrome [version] on [platform]"
Test that https://diafygi.github.io/webrtc-ips/ doesn't leak IP address when Block all fingerprinting protection is on

Brave Ads

Rewards

Social-media blocking settings

Verify individual Social media blocking buttons works as intended when enabled/disabled by visiting https://fmarier.github.io/brave-testing/social-widgets.html
visit brave://settings/privacy -> Site and Shields Settings -> Cookies and site data and ensure that
- both https://[*.]firebaseapp.com & https://accounts.google.com are added into Sites that can always use cookies when Allow Google login buttons on third party sites is enabled
- both https://[*.]firebaseapp.com & https://accounts.google.com are removed from Sites that can always use cookies when Allow Google login buttons on third party sites is disabled
ensure that you can log in into https://www.expensify.com while Allow Google login buttons on third party sites is enabled
ensure that once Allow Google login buttons on third party sites has been disabled, you can't log in into https://www.expensify.com

Sync

Tor Tabs

Visit https://check.torproject.org in a Tor window, ensure it shows a success message for using a Tor exit node
Visit https://check.torproject.org in a Tor window, note down exit node IP address. Do a hard refresh (Ctrl+Shift+R/Cmd+Shift+R), ensure exit IP changes after page reloads
Visit https://check.torproject.org in a Tor window, note down exit node IP address. Click New Tor Identity for this site in app menu, ensure the exit node IP address changes after page is reloaded
Visit https://protonirockerxow.onion & https://brave5t5rjjg3s6k.onion/ in a Tor window and ensure both pages resolve
Visit https://browserleaks.com/geo in a Tor window, ensure location isn't shown
Verify Torrent viewer doesn't load in a Tor window
Ensure you are able to download a file in a Tor window. Verify all Download/Cancel, Download/Retry and Download works in Tor window

Cookie and Cache

Go to http://samy.pl/evercookie/ and set an evercookie. Check that going to prefs, clearing site data and cache, and going back to the evercookie site does not remember the old evercookie value

Chromium/Brave GPU

Verify that brave://gpu (Brave) matches chrome://gpu (Chrome) when using the same Chromium version

Components

Remove the following component folders and ensure that they're being re-downloaded after restarting the browser:
- afalakplffnnnlkncjhbmahjfjhmlkal: AutoplayWhitelist.dat, ExtensionWhitelist.dat, ReferrerWhitelist.json and Greaselion.json
- CertificateRevocation
- cffkpbalmllkdoenhmdmpbkajipdjfam: rs-ABPFilterParserData.dat & regional_catalog.json (AdBlock)
- gccbbckogglekeggclmmekihdgdpdgoe: (Sponsored New Tab Images)
- jicbkmdloagakknpihibphagfckhjdih: speedreader-updater.dat
- oofiananboodjbbmdelgdommihjbkfag: HTTPSE
- Safe Browsing
Restart the browser, load brave://components, wait for 8 mins and verify that no component shows any errors

Note: Always double check brave://components to make sure there's no errors/missing version numbers

Session storage

Temporarily move away your browser profile and test that a new profile is created on browser relaunch
- macOS - ~/Library/Application\ Support/BraveSoftware/
- Windows - %userprofile%\appdata\Local\BraveSoftware\
- Linux(Ubuntu) - ~/.config/BraveSoftware/
Test that both windows and tabs are being restored, including the current active tab
Ensure that tabs are being lazy loaded when a previous session is being restored

Upgrade

The text was updated successfully, but these errors were encountered:

kjozwiak · 2021-02-17T05:07:07Z

Please don't start the above until we get C89 merged into 1.21.x.

stephendonner · 2021-03-02T00:41:48Z

Build used:

Brave	1.21.70 Chromium: 89.0.4389.69 (Official Build) (x86_64)
Revision	6df9174a477a0bea2118cb4669954d517be16104-refs/branch-heads/4389@{#1332}
OS	macOS Version 11.2.2 (Build 20D80)

I skipped:

Visit https://jsfiddle.net/bkf50r8v/13/, ensure 3 blocked items are listed in Shields. Result window should show got canvas fingerprint 0 and got webgl fingerprint 00 (defunct test)
evercookie (couldn't get it to work)
most Uphold tests (knowledge gap)
Verify that brave://gpu (Brave) matches chrome://gpu (Chrome) when using the same Chromium version (see screenshot; my minor versions didn't match)

Test that https://diafygi.github.io/webrtc-ips/ doesn't leak IP address when Block all fingerprinting protection is on (couldn't get it to work with/without Shields)
Verify older version of an extension gets updated to new version via Brave server (defunct)

LaurenWags · 2021-03-02T16:20:02Z

Using

Brave | 1.21.71 Chromium: 89.0.4389.72 (Official Build) (x86_64)
-- | --
Revision | 3f345f156bfd157bd1bea06310e55f3fb2490359-refs/branch-heads/4389@{#1393}
OS | macOS Version 10.15.7 (Build 19H512)

the following sections were verified:

Cookie and Cache
Chromium/Brave GPU (compared Brave version above with Google Chrome | 89.0.4389.72 (Official Build) beta (x86_64)
parts of Rewards (Uphold cases, wallet restore)

stephendonner · 2021-03-02T16:41:46Z

Thanks, @LaurenWags - I forgot that we have to check the cookie in Preferences, not its value on the page :-( Was going to confirm that with you/team today.

LaurenWags · 2021-03-02T16:50:54Z

also need to put shields down for that page to work correctly unfortunately :(

stephendonner · 2021-03-02T19:40:41Z

Closing out; verified PASSED using

Brave	1.21.70 Chromium: 89.0.4389.69 (Official Build) (x86_64)
Revision	6df9174a477a0bea2118cb4669954d517be16104-refs/branch-heads/4389@{#1332}
OS	macOS Version 11.2.2 (Build 20D80)

Many thanks again, @LaurenWags, for noticing and picking up my slack 👍

stephendonner · 2021-03-02T19:42:21Z

Closing out; verified PASSED using
Brave 1.21.70 Chromium: 89.0.4389.69 (Official Build) (x86_64)
Revision 6df9174a477a0bea2118cb4669954d517be16104-refs/branch-heads/4389@{#1332}
OS macOS Version 11.2.2 (Build 20D80)

https://github.com/brave/brave-browser/releases/tag/v1.21.71 is newer, but we'll be re-spinning for #14428 anyway.

kjozwiak added tests OS/macOS QA/Yes release-notes/exclude OS/Desktop labels Feb 17, 2021

kjozwiak added this to the 1.21.x - Beta milestone Feb 17, 2021

kjozwiak mentioned this issue Feb 17, 2021

1.21.x Release #1 brave/qa-resources#227

Closed

stephendonner self-assigned this Feb 26, 2021

stephendonner closed this as completed Mar 2, 2021

LaurenWags added the QA Pass-macOS label Mar 2, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manual test run on macOS (Intel) for 1.21.x Release #1 #14199

Manual test run on macOS (Intel) for 1.21.x Release #1 #14199

kjozwiak commented Feb 17, 2021 •

edited by LaurenWags

Loading

kjozwiak commented Feb 17, 2021

stephendonner commented Mar 2, 2021

LaurenWags commented Mar 2, 2021 •

edited

Loading

stephendonner commented Mar 2, 2021

LaurenWags commented Mar 2, 2021

stephendonner commented Mar 2, 2021

stephendonner commented Mar 2, 2021

Manual test run on macOS (Intel) for 1.21.x Release #1 #14199

Manual test run on macOS (Intel) for 1.21.x Release #1 #14199

Comments

kjozwiak commented Feb 17, 2021 • edited by LaurenWags Loading

Installer

About pages

Importing

Context menus

Extensions/Plugins

Chrome Web Store (CWS)

PDF

Widevine

Geolocation

Crash Reporting

Bravery settings

Fingerprint Tests

Brave Ads

Rewards

Social-media blocking settings

Sync

Tor Tabs

Cookie and Cache

Chromium/Brave GPU

Components

Session storage

Upgrade

kjozwiak commented Feb 17, 2021

stephendonner commented Mar 2, 2021

LaurenWags commented Mar 2, 2021 • edited Loading

stephendonner commented Mar 2, 2021

LaurenWags commented Mar 2, 2021

stephendonner commented Mar 2, 2021

stephendonner commented Mar 2, 2021

kjozwiak commented Feb 17, 2021 •

edited by LaurenWags

Loading

LaurenWags commented Mar 2, 2021 •

edited

Loading