Support for custom filter lists in Shields

[fmarier]

Description

Some popular adblocking extensions help users create filter lists to supplement the default ones. This can for example allow someone to hide parts of a website that are not deemed useful or desirable but that would otherwise be out of scope for EasyList/EasyPrivacy.

This is an advanced use case for sure, and we should focus on shipping the best default-on adblocker, but I thought I'd mention it since it might allow some users to avoid installing an additional extension and have everything they need built-in.

Designs

Add functionality to subscribe to filter lists via URL on brave://adblock

Insert the following designs below “Additional Filters” and above “Custom Filters”:

After the user clicks on "Add filter list via URL", a form field appears with "Submit" and "Cancel":

Built-in error validation for the field specifies entry must start with https://

Submitting the filter list URL will add the filter list to a table with a toggle to enable/disable and available actions:

• Update now
• View website
• View source
• Unsubscribe

If the user enters a filter list that is invalid or unavailable, the table entry shows the filter as Download failure with no toggle available:

Assets

Figma: https://www.figma.com/file/tLXWGCpNoiJxDZDdpfordj/Desktop-Settings?node-id=49%3A2359

[rebron]

cc: @pes10k @ryanbr Is this a dupe?

[pes10k]

No, this is different. Currently Brave users can add additional rules in the text box on brave://adblock (soon to be brave://shields/filter-rules I think). Whats being asked for here, I believe, is the ability to subscribe to new lists, by URL.

[ryanbr]

Another way could be added, via ahref abp:subscribe? to streamline custom filter additions (supported by ABP and uBO)

Used on https://www.filterlists.com/ and on my own site https://secure.fanboy.co.nz/
Sample:
abp:subscribe?location=https://secure.fanboy.co.nz/fanboy-problematic-sites.txt&title=Fanboy Problematic Sites

If this suggestion is out of the scope of this, I can create a separate report.

[pes10k]

I think that would be better to open as an issue in brave-core, and note in it that it would need to come after this issue is closed

[karenkliu]

Designs added!

[intrnl]

I think the next thing would be the ability to disable/enable some of the default filter lists as well 👍

[antonok-edm]

@intrnl if you don't mind sharing, it'd be great to hear your use-case (although, probably in a separate issue).

If there's something in our default lists causing breakage then we'd like to get the lists fixed upstream as much as possible.

[Furax-31]

This feature is still not available?

[stephendonner]

Verified PASSED using

Brave	1.31.77 Chromium: 94.0.4606.71 (Official Build) beta (64-bit)
Revision	1d32b169326531e600d836bd395efc1b53d0f6ef-refs/branch-heads/4606@{#1256}
OS	Linux

Steps:

1. Visited brave://adblock
2. Verified that a "Subscribe to filter lists" section exists between the "Additional Filters" and "Custom Filters" sections, including an "Add filter list via URL" button.

3. Clicked on the button to create a new subscription. A textbox appeared, along with a disabled "Submit" button and a "Cancel" button.

4. Confirmed the "Submit" button remains disabled as long as the text box does not include a valid URL. Pressing enter from the textbox also has no effect whenever the button is disabled.
5. Entered http://failed.url and confirmed it has Update Now and Unsubscribe buttons. Pressing Update Now yields a Download failure error message, with a timestamp on hover.

example	example

6. Entered https://austinhuang.me/0131-block-list/list.txt and confirmed it has Update now, View source, and Unsubscribe buttons. Pressing Update Now updated the list with a new timestamp. View source opens the filter-list source code in a new tab. And pressing Unsubscribe removes the filter URL from the list.
7. Confirmed the information in the table remained consistent across different brave://adblock tab instances, and also across browser restarts.
8. Confirmed I'm unable to add duplicate filter-list URL entries (the Submit button is disabled).

9. visited https://example.com after subscribing to https://antonok.com/tmp/list.txt, and confirmed https://example.com is mostly red now, changed by the filter list. Unchecked https://antonok.com/tmp/list.txt, reloaded, and confirmed that http://example.com was no longer red.

example	example	example

Verified timestamps were accurate after modifying them via ~/Users/{user-dir}/Library/Application\ Support/Brave Software/Brave-Browser-Beta/Local State:

example	example	example	example

---

Verification passed on

Brave | 1.31.66 Chromium: 94.0.4606.61 (Official Build) beta (64-bit)
-- | --
Revision | 418b78f5838ed0b1c69bb4e51ea0252171854915-refs/branch-heads/4606@{#1204}
OS | Windows 10 Version 21H1 (Build 19043.1237)

Steps:

1. Visited brave://adblock

2. Verified that a "Subscribe to filter lists" section exists between the "Additional Filters" and "Custom Filters" sections, including an "Add filter list via URL" button.
   

3. Clicked on the button to create a new subscription. A textbox appeared, along with a disabled "Submit" button and a "Cancel" button.
   

4. Confirmed the "Submit" button remains disabled as long as the text box does not include a valid URL. Pressing enter from the textbox also has no effect whenever the button is disabled.

5. Entered http://failed.url and confirmed it has Update Now and Unsubscribe buttons. Pressing Update Now yields a Download failure error message, with a timestamp on hover.

Before update now	After update now

6. Entered https://austinhuang.me/0131-block-list/list.txt and confirmed it has Update now, View source, and Unsubscribe buttons. Pressing Update Now updated the list with a new timestamp. View source opens the filter-list source code in a new tab. And pressing Unsubscribe removes the filter URL from the list.

example	example	example

7. Confirmed the information in the table remained consistent across different brave://adblock tab instances, and also across browser restarts.

8. Confirmed I'm unable to add duplicate filter-list URL entries (the Submit button is disabled)
   

9. visited https://example.com after subscribing to https://antonok.com/tmp/list.txt, and confirmed https://example.com is mostly red now, changed by the filter list. Unchecked https://antonok.com/tmp/list.txt, reloaded, and confirmed that http://example.com was no longer red.

example	example	example

[stephendonner]

@antonok-edm I verified this issue because the rest worked, but I was unable to get two things working at the time:

1. unable to produce the Download failure since... condition
2. unable to consistently see auto-refresh of the timestamp, on relaunch/clock changes; I did happen to see same tab self-update at 4:46-4:47pm PDT, so not sure what's right.

Can you take a look and clarify if I've done something -- or haven't -- to make those two not reproducible? Thanks!

Local State's timestamps, after auto-updating around 4:46-4:47pm PDT

:

"brave":{
      "ad_block":{
         "checked_default_region":true,
         "custom_filters":"",
         "list_subscriptions":{
            "http://invalid.url/":{
               "enabled":true,
               "last_successful_update_attempt":"0",
               "last_update_attempt":"13275529233046704"
            },
            "https://antonok.com/tmp/list.txt":{
               "enabled":true,
               "last_successful_update_attempt":" 13274924252360775",
               "last_update_attempt":" 13274924252360775"
            },
            "https://austinhuang.me/0131-block-list/list.txt":{
               "enabled":true,
               "last_successful_update_attempt":"13275529052348655",
               "last_update_attempt":"13275529052348655"
            }
         }604800000000
      },

[antonok-edm]

@stephendonner

1. I tried this locally and discovered that the download remains pending if the network is not available at all. I think that's fine, but sadly it makes it more difficult to test this behavior! I think we need to actually simulate a server that becomes unavailable for a second download attempt. You could try setting up a local server with a filter list (URLs on e.g. http://localhost:8000 should work fine here), or potentially muck around with /etc/hosts to trick the browser into thinking some valid domain actually resolves to 127.0.0.1. Or, worst case we can coordinate testing on my own server, I'll just remove the file for the second download attempt.
2. The brave://adblock UI should show About 7 days ago immediately at launch if the timestamps were updated correctly in Local State. If not, there's a few things that might have gone wrong:

• I assume you edited the correct Local State file, given that you found the list subscriptions in the first place.

• Did you close the browser before editing Local State? I believe it may get rewritten before the browser is closed, otherwise.

  Even then, it may take more than a minute to update, depending on how loaded the computer is, whether or not you're on a metered network, battery status - but from what I can see in your screenshots I think it shouldn't be more than 5 minutes.

[stephendonner]

1. I've now reproduced the Download failure since... status-text update; thanks!
2. I set up a quick-and-dirty server at https://localhost:8000 using python3 -m http.server, on my Desktop, with your https://antonok.com/tmp/list.txt copied locally, and it worked 👍

New question: example.com has your stylesheet rules applied to it now, even though I've terminated my server and the fetch for https://localhost:8000/filter.txt fails (host down); do we cache last-known good rules? (i.e. I'm asking if it should stay red or return to its default color, since it can't fetch the latest filter from my localhost).

[antonok-edm]

@stephendonner correct, the last successful download should be cached and used even if more recent downloads are unsuccessful (you should still be able to see that source using the View source button in the triple dot menu). That appears to be working correctly from your screenshot 🎉

[lazymonkey2]

does custom lists automatically update?

[bridiver]

@Tortino the default is weekly unless you manually update

[lazymonkey2]

thank you for the info.
I know some lists are updated daily, so it seems to me that a week is a little to long.
maybe this settings could be user configurable?

[stephendonner]

thank you for the info.
I know some lists are updated daily, so it seems to me that a week is a little to long.
maybe this settings could be user configurable?

Yes; that would be the "non-default update intervals" mentioned in #8107 (comment)

[ryanbr]

Ideally adhering to Expires / Expiring rather than non-default intervals. Server/BW load is real. Weekly is fine as a stopgap

[GeetaSarvadnya]

@antonok-edm When I add/update any valid URL to filter list, I get some Timestamp displayed in the Last updated column. Any idea what timestamp should be displayed in the Last updated column? It's not showing up system time (IST), not UTC also.

[antonok-edm]

@GeetaSarvadnya to be honest, I'm not sure what timezone that's supposed to be - I would have assumed UTC? At any rate, it's the default serialization for the base::Time structure in Chromium, and I wouldn't worry too much about that being incorrect. Should be fine as long as it's within the right day and consistent relative to other downloads and across browser restarts.

I think it can be revisited when the rest of the brave://adblock UI gets redesigned. I'd consider this a UI/UX concern, of which the page doesn't have much of to begin with 😂

[srirambv]

Verification passed on Samsung Tab A with Android 10 running 1.31.84 x64 build

• Verified Subscribe to filter list option is available between Additional filters and Custom filters
• Verified text box and submit button is shown when you click on Add filter list via URL button. Submit button only enables when a URL is being entered

• Verified adding an invalid list(https://failed.url) shows Download failure message. Still shows Update Now and Unsubscribe buttons
  

• Verified adding a valid list (https://austinhuang.me/0131-block-list/list.txt) also shows Update Now, View Source and Unsubscribe options.

• Verified clicking on view source tries to open the source list

• Verified Unsubscribe removes the URL from the list
  

• Verified unable to add a duplicate entry for an existing list

• Verified subscribing to https://antonok.com/tmp/list.txt, and visiting https://example.com loads the page in red

• Verified unchecking the list and reloading the page loads it normally

---

Verification passed on Oppo Reno 5 with Android 10 running 1.31.84 x64 build

• Verified Subscribe to filter list option is available between Additional filters and Custom filters
• Verified text box and submit button is shown when you click on Add filter list via URL button. Submit button only enables when a URL is being entered

• Verified adding an invalid list(https://failed.url) shows Download failure message. Still shows Update Now and Unsubscribe buttons
  

• Verified adding a valid list (https://austinhuang.me/0131-block-list/list.txt) also shows Update Now, View Source and Unsubscribe options.

• Verified clicking on view source tries to open the source list

• Verified Unsubscribe removes the URL from the list
  

• Verified unable to add a duplicate entry for an existing list

• Verified subscribing to https://antonok.com/tmp/list.txt, and visiting https://example.com loads the page in red

• Verified unchecking the list and reloading the page loads it normally