Add official mirrors for Nordic Filters and EasyList Lithuania

[DandelionSprout]

Prerequisites

• I verified that this is not a filter issue
  • Filter issues MUST be reported at filter issue tracker
• This is not a support issue or a question
  • Support issues and questions are handled at /r/uBlockOrigin
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue
  • Your issue may already be reported.
• I tried to reproduce the issue when...
  • uBlock Origin is the only extension
  • uBlock Origin with default lists/settings
  • using a new, unmodified browser profile
• I am running the latest version of uBlock Origin (1.34.1rc2)
• I checked the documentation to understand that the issue I report is not a normal behavior

Description

Directly inspired by a reporter on Slack saying that his Indian ISP is blocking raw.githubusercontent.com (which I believe him on, considering it's India we're talking about), I've set up some mirrors for lists that I have maintainer rights for.

A specific URL where the issue occurs

https://github.com/gorhill/uBlock/blob/master/assets/assets.json

Steps to Reproduce

N/A

Expected behavior:

• Add https://gitlab.com/DandelionSprout/adfilt/-/raw/master/NorwegianList.txt as the 2nd content URL for Dandelion Sprout's Nordic Filters.
• Add https://gitlab.com/DandelionSprout/easylist_lithuania/-/raw/master/easylistlithuania.txt as the 2nd content URL for EasyList Lithuania.
• It's probably fine to delete the repo.or.cz mirror of Dandelion Sprout's Nordic Filters from the assets list, as it was deleted several months ago.

Actual behavior:

Neither Dandelion Sprout's Nordic Filters nor EasyList Lithuania currently have functional non-CDN mirrors by default in uBO's assets list.

Your environment

• uBlock Origin version: 1.34.1rc2
• Browser Name and version: Chrome 90.0.4430.85 x64
• Operating System and version: Windows 10 20H2

[gwarser]

Not a good idea to use gitlab as a mirror... #1285

???

[DandelionSprout]

I lack any other reliable sites with easy mirroring functions to use for this.

[uBlock-user]

https://statically.io/convert/
https://combinatronics.com/

@DandelionSprout

[DandelionSprout]

Those two sites give me the following output, which I presume are meant to be used in the "cdnURLs" paragraph for Nordic Filters instead of in "contentURL":

https://cdn.statically.io/gh/DandelionSprout/adfilt/master/NorwegianList.txt
https://combinatronics.com/DandelionSprout/adfilt/master/NorwegianList.txt
https://cdn.statically.io/gl/DandelionSprout/adfilt/master/NorwegianList.txt

If I've understood this correctly, I can do the same with EasyList Lithuania with ease.

[DandelionSprout]

Considering I still lack PR rights for the uBlock Origin repo, this is my current estimate guideline at which mirrors to add and how: DandelionSprout/uBlock@cf47bf3

[ghajini]

okay i have contacted my isp which is no1 isp(virtue of more subscribers base /low pricing/high competition created) in india, entry of that in indian market resulted in more and more mobile internet users and using their wifi enabled devices,
they have resolved that the request to block raw.githubusercontent.com is receieved from department of telecommunication india, they as isp can't do anything than obey DOT compliance....

once some url is captured by DOT list, its hard to expect them to remove block considering how illiterate they are?

these lists are at least affected,these lists have some cdn urls but ublock doesn't update from them once it detect network error while updating...

(i do connected github support also but they as a big organisation instead of reaching DOT, they said me to connect to my isp, i can contact DOT but that will be unsuccessful attempt as a normal user)

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badlists.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/resource-abuse.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/annoyances.txt

i have disabled default lists and added mapx- ones urls provided....

[DandelionSprout]

they have resolved that the request to block raw.githubusercontent.com is receieved from department of telecommunication india, they as isp can't do anything than obey DOT compliance....

That is most certainly something I expected that India would do at random.

[gorhill]

ublock doesn't update from them once it detect network error while updating...

uBO will keep trying with other URLs if any, but if raw.githubusercontent.com is blocked country-wide -- which I consider a serious issue -- then I will add more fallback URLs for default lists.

[DandelionSprout]

It has dawned on me that I forgot to ask about this originally:

Are there any known calculations for when GitLab will decide to refuse to offer raw list versions? If we assume the uBO version of Nordic Filters has 800,000 weekly users, ! Expires: 1 day, and the GitLab link is 2nd in the priority order, would GitLab be okay with that?

[DandelionSprout]

Even more critically, uBlockOrigin/uAssets#10010 (comment) confirms that my GitLab mirror(s) is synchronisable from PR-China, a fact that should by all accounts place this issue report on fasttrack.

[DandelionSprout]

Okay, so looking into why it could take 6 months for this report to get resolved, I googled around and found https://docs.gitlab.com/ee/administration/instance_limits.html#by-raw-endpoint. With what at least seems to be a limit of 300 raw file queries per minute per repo, it seems like gwarser was correct after all about GitHub being a bad idea for major lists.

So, well, what should I do now? Because the repo.or.cz link should definitely not continue to be in assets.json, and I feel a moral need to accomodate users in authoritarian countries such as large parts of Asia.

[gorhill]

There is a lot of issues in the issue tracker which I still need to do something about. This one here I didn't spend time on it because I am not aware that there is any fetch issues for users of those lists -- so I don't see this as something which I need to fix asap.

[gorhill]

You can always submit a carefully crafted pull request if you want some movement on the issue here.

[DandelionSprout]

Sure thing. Give me 15min.

[DandelionSprout]

gorhill/uBlock@master...DandelionSprout:patch-3 seems good to me, provided I understand uBO's CDN system correctly. I lack PR rights as well, seemingly, so I suppose you'll need to validate the PR somehow.

[gorhill]

No need to have two URLs to https://cdn.statically.io/, if that server goes down, both URLs will be down. Also frankly, I think this is going overboard to have so many mirrors for minor regional lists -- let's agree that there should be no more mirrors than uBlock's own filter lists. I suggest you pick three for cdnURLs, which means 4 mirrors total when including the main https://raw.githubusercontent.com/... one in contentURL.

[DandelionSprout]

Okay, sure thing. Give me 10min.

[DandelionSprout]

gorhill/uBlock@master...DandelionSprout:patch-4 is limited to 4 URLs in the CDN sections, as requested.

[gorhill]

You can remove the https://raw.githubusercontent.com/EasyList-Lithuania/easylist_lithuania/master/easylistlithuania.txt from cdnURLs, it will be copied from contentURL when picking a server, so there should be no more than 3 in cdnURLs. I know there are other places where contentURLs is duplicated, and this is something I need to cleanup.

[DandelionSprout]

Ah, okay, sure thing. I'll append that suggested fix to patch-4 fairly quick.

[gorhill]

Can you git merge --squash patch-4 into your master branch? This way I will be able to pull your changes as a single commit properly attributed.

[DandelionSprout]

I've done something that seem to have the same effect: Have the DandelionSprout/uBlock/master branch fetch upstream, then copy assets.json from patch-4 to master. gorhill/uBlock@master...DandelionSprout:master

[gorhill]

I ended up creating a pull request from your changes, I thought this would make you a contributor but it did not -- originally I was trying to pull directly from your repo using command line, this would have made you contributor.

What you could do is to replace the array of one string with the string itself: DandelionSprout/uBlock@26e97d9#diff-867c526785ad7b2296cb4d7ad5a069232add7ad772faf27beba1ce9570f288abR516-R518

Then I will pull from your master branch and that will make you contributor, removing the friction in the future to submit pull requests.

[DandelionSprout]

Looking into it, I curiously seem to have obtained the right to submit PRs, despite lacking the Contributor label.

The blue box in the below screenshot (taken from a quick test), used to be yellow before I gained rights.

[gorhill]

Alright then, maybe the contributors page just takes a while to update when something changes.