Add queryjump to redirect url (urlskip)

[Yuki2718]

Prerequisites

• I verified that this is not a filter list issue. Report any issues with filter lists or broken website functionality in the uAssets issue tracker.
• This is NOT a YouTube, Facebook or Twitch report. These sites MUST be reported by clicking their respective links.
• This is not a support issue or a question. For support, questions, or help, visit /r/uBlockOrigin.
• I performed a cursory search of the issue tracker to avoid opening a duplicate issue.
• The issue is not present after disabling uBO in the browser.
• I checked the documentation to understand that the issue I am reporting is not normal behavior.

I tried to reproduce the issue when...

• uBO is the only extension.
• uBO uses default lists and settings.
• using a new, unmodified browser profile.

Description

Mentioned in #760 (comment) #2476 (comment) and now it should be possible to implement as a trusted network filter modifier. It's similar to $uritransform except that this should redirect to different domain and complements href-sanitizer. Maybe $to or something can be combined to restrict the domain to redirect rather than always blindly trust the url in parameter. Though the original context was about shortners, I rather want to use this to automate skip of strict-block page which is currently possible only by manual click.

A specific URL where the issue occurs.

`https://ouo.io/s/BulJXu78?s=http://hackstore.link/alg2i`

Steps to Reproduce

NA

Expected behavior

NA

Actual behavior

NA

uBO version

1.57.0

Browser name and version

Not relevant

Operating System and version

Windows 10

[gorhill]

Test cases:

---

Filters:

||example.com^$doc
||example.com/path/to/tracker$urlskip=?url

Navigate to https://example.com/path/to/tracker?url=https://example.org/

Navigated to: https://example.org/

---

Filter:

||example.com/path/to/tracker$urlskip=?url ?to

Navigate to https://example.com/path/to/tracker?url=https%3A%2F%2Fexample.org%2Fpath%2Fto%2Ftracker%3Fto%3Dhttps%253A%252F%252Fgithub.com%252F

Navigated to: https://github.com/

---

Filter:

||example.com/path/to/tracker$urlskip=?url ?to

Navigate to https://example.com/path/to/tracker?url=https%3A%2F%2Fexample.org%2Fpath%2Fto%2Ftracker

Navigated to https://example.com/path/to/tracker?url=https%3A%2F%2Fexample.org%2Fpath%2Fto%2Ftracker

Note: No skip occurred since not all extraction directives could be fulfilled

[uBlock-user]

For https://ouo.io/s/BulJXu78?s=http://hackstore.link/alg2i
Filter - ||ouo.io/s/*$urlskip=?s

[krystian3w]

Spaces before "?" are needed?

[gorhill]

Spaces before "?" are needed?

Yes. Commit message:

The syntax is urlskip=[steps], where steps is a space-separated
list of extraction directives detailing what action to perform on
the current URL.

In the future we might want to add base64-decoding or regex extraction (=?url base64), so a separator is needed for the sake of extending the syntax in the future, a space is a good choice since it's never meant to appear in a URL.

[hurgir]

Hi, when I want to reach https://www.evekonomisi.net/git/11770, it redirects to this address

https://rdr.btrck.com/aff_c?offer_id=17&aff_id=1&source=55754&aff_sub=www.evekonomisi.net&aff_sub2=&aff_sub3=hepsiburada.com/philips-ca6700-10-espresso-makinesi-kirec-temizleme-solusyonu-p-HBCV00002YNJSF&aff_sub4=inf-www.evekonomisi.net-7324&aff_sub5=inf&url=https%3a%2f%2f7t4g.adj. st%2fproduct%3fsku%3dHBCV00002YNJSF%26adj_t%3d16v6nskj_16n3rgcj%26publisher_id%3d{transaction_id}%26offer_id%3d{offer_id}%26campaign%3dwinfluenced%26adgroup%3d{aff_sub}{source}%26creative%3d-aff_sub3- %26adj_deep_link%3dhbapp%253a%252f%252fproduct%253fsku%253dHBCV00002YNJSF%2526adj_t%253d16v6nskj_16n3rgcj%2526utm_source%253dwinfluenced%2526utm_medium%253dinfluencer%2526utm_campaign%253dwinfluenced%2526utm_content%253d- aff_sub3-%2526utm_term%253dinfluencer%2526wt_inf%253d{source}%26adj_install_callback%3dhttps%253a%252f%252fwinfluenced-pixel-bs56cx2ajq-ey. a.run. app%252fapi%252fpixel%252fcallback%252finstall%253fevent_type%253dInstall%2526linkid%253d95O2L4G8A2%2526ltoken%253d16v6nskj_16n3rgcj%2526app_id%253d%7Bapp_id%7D%2526offer_id%253d17%2526advertiser_id%253d{advertiser_id}%2526click_id%253d{transaction_id}%2526app_name%253d%7Bapp_name%7D%2526app_version%253d%7Bapp_version%7D%2526store%253d%7Bstore%7D%2526installed_at%253d%7Binstalled_at%7D%2526ip_address%253d%7Bip_address%7D%2526country%253d%7Bcountry%7D%2526device_name%253d%7Bdevice_name%7Dc%2526device_type%253d%7Bdevice_type%7D%2526os_name%253d%7Bos_name%7D%2526partner_parameters%253d%7Bpartner_parameters%7D%2526debug%253dtrue%26adj_event_callback_m2ril4_wapyen%3dhttps%253a%252f%252fwinfluenced- pixel-bs56cx2ajq-ey. a.run. app%252fapi%252fpixel%252fcallback%252fconversion%253fevent_type%253dConversion%2526linkid%253d95O2L4G8A2%2526ltoken%253d16v6nskj_16n3rgcj%2526etoken%253dm2ril4_wapyen%2526offer_id%253d%257bofferId%257d%2526advertiser_id%253d{advertiser_id}%2526app_id%253d%7Bapp_id%7D%2526click_id%253d{transaction_id}%2526ip_address%253d%7Bip_address%7D%2526country%253d%7Bcountry%7D%2526partner_parameters%253d%7Bpartner_parameters%7D%2526revenue_float%253d%7Brevenue_float%7D%2526currency%253d%7Bcurrency%7D%2526reporting_revenue%253d%7Breporting_revenue%7D%2526reporting_currency%253d%7Breporting_currency%7D%2526os_name%253d%7Bos_name%7D%2526debug%253dtrue

I wrote a filter rule as ||rdr.btrck.com/aff_c*$urlskip=?offer_id aff_id source aff_sub aff_sub2 aff_sub3 and I want to reach hepsiburada.com/philips-ca6700-10-espresso-machine-kirec-temizleme-solusyonu-p-HBCV00002YNJSF

[stephenhawk8054]

I think the parameter value should be a valid string to use with new URL(). The string hepsiburada.com/philips-ca6700-10-espresso-makinesi-kirec-temizleme-solusyonu-p-HBCV00002YNJSF is not a valid string for URL, i.e

new URL("hepsiburada.com/philips-ca6700-10-espresso-machine-kirec-temizleme-solusyonu-p-HBCV00002YNJSF")

will return error, so no, you can't use urlskip in this case.

[hurgir]

So what kind of filter rule should I write for this?

[gorhill]

||rdr.btrck.com/aff_c*$urlskip=?offer_id aff_id source aff_sub aff_sub2 aff_sub3

That is not a valid urlskip filter, it doesn't match what the documentation says how it works. Looks like you want to extract the URL from the aff_sub3 parameter; offer_id, aff_id, source, aff_sub, aff_sub2 are not valid extraction directives. This would be the proper filter:

||rdr.btrck.com/aff_c?$urlskip=?aff_sub3

However as @stephenhawk8054 pointed out, the extracted string is not a valid URL as per URL API, so it does not work in the end.

I do think it should work though, so I will see what can be done about it.

[gorhill]

In the next dev build, this should work:

||rdr.btrck.com/aff_c?$urlskip=?aff_sub3 +https

[Catman-232]

Sorry to bump this but I really didn't think this was worth making a whole new issue for, I know this was added in the past couple weeks so I haven't been able to find any documentation or anyone asking about this case scenario: How do I use $urlskip if the hyperlink URL doesn't even have a property after the ? ? From what I can tell the URL API then just considers the whole URL as a property instead of a value.

For convenience I was using this to successfully skip Pixiv's kinda redundant outbound link capture page https://www.pixiv.net/jump.php since on user profiles they have social media icons for Twitter etc which can only actually be customised with an account name, not any potentially malicious URL. Those links use url=, so that's not a problem to catch with a rule.

However, on gallery pages the links in the description actually do not have url=, they are just
https://www.pixiv.net/jump.php?https%3A%2F%2Fx.com%2FExample_Username

Obviously I can't make a rule like ||pixiv.net/jump.php$urlskip=? since that's "Invalid filter: Bad value assigned to a valid option". I was using uBO for this since I figured it would be more direct and performant than making some Tampermonkey script to do any kind of instant redirect or href edits on all the hyperlinks, convenient coincidence that this static network modifier filter was recently added and can do what I was hoping for, as long as the URL has a property to target.

[gorhill]

How do I use $urlskip if the hyperlink URL doesn't even have a property after the ?

The initial version of urlskip= is limited and deal with the obvious ?url= way.

People can then report cases which cannot be handled by the current urlskip=, then we will consider adding support. For example, see gorhill/uBlock@e81e70937f.

So what you are reporting here is a case where the URL to skip to is encoded as a parameter name. I will think if and what could be done about this.

---

By the way, in general to make me more motivated to work on this, having actual cases I can reproduce immediately on my side helps a lot. If I cannot immediately reproduce, I typically lose interest and move on to do other stuff. Best way is to provide a URL I just have to navigate to and reproduce the issue without further complication like having to create an account etc.

[Catman-232]

Wow that was a fast response, thanks. Sorry about the example link, I didn't realise it wouldn't work, I see that if the link is put straight into the address bar it just takes you to an error page. The a tags in the HTML use href="/jump.php?https...etc which I guess means it only works in-site. I've found an example for you that I confirmed worked while not logged in (using a private window) https://www.pixiv.net/en/artworks/122985712 - Fanbox ones don't have it because Pixiv runs Fanbox themselves.
I didn't even know they had an advert on that in-between page hah.

I was starting a bit of a ramble about the properties in URL objects but after seeing your post edit I figured you probably would already know about this anyway:

the start of the ramble

So I was still tinkering with this and tried doing `new URL` in the dev tools console like stephenhawk8054 showed, and the URL object has the destination URL as a `search` property containing the encoded chars and the question mark at the beginning. It also has a `searchParams` property containing a `URLSearchParams` object with one entry ...

[gorhill]

https://www.pixiv.net/en/artworks/122985712

Where do I click on that page to trigger the navigation to the cloaked URL?

[uBlock-user]

https%3A%2F%2Fx.com%2FExample_Username

That's percent encoding. Support for it will have to be added and doesn't exist yet.

[Catman-232]

I suppose I could do with a lesson in being directly specific, sorry. It's these links down below the main image in the page, you can click "Continue Reading" to expand it to show the full description which has a lot of hyperlinks in it but most are just Fanbox and Patreon links. Actually looks like all of them except the mangamaterials.fanbox.cc ones are prepended with https://www.pixiv.net/jump.php? as I see in the hover url

https%3A%2F%2Fx.com%2FExample_Username

That's percent encoding. Support for it will have to be added and doesn't exist yet.

It was what it said in the address bar on the in-between page so I just copied that, the socials links on profiles, which use ?url= also show the encoding in the address bar and the $urlskip rule worked fine on them.

[gorhill]

This should work with 1.60.1b6 and above:

||pixiv.net/jump.php?$urlskip=&1

[Catman-232]

It works, I just missed the ? moving to before $urlskip= the first time I tested it and had to look a little closer to see that.

[sertonix]

Is it possible to use use urlskip when the link is part of the path?
For example converting https://www.podtrac.com/pts/redirect.mp3/dovetail.prxu.org/7057/1b6a25b9-d8fe-48cd-aab1-cec94c3ae137/darknet-diaries-ep150-mobman-2.mp3 into https://dovetail.prxu.org/7057/1b6a25b9-d8fe-48cd-aab1-cec94c3ae137/darknet-diaries-ep150-mobman-2.mp3

[gorhill]

Is it possible to use use urlskip when the link is part of the path?

Yes, I expected such case would be reported eventually. On which website did you encounter such link?

[sertonix]

On which website did you encounter such link?

https://darknetdiaries.com/episode/150/

When you click on the download button (cloud icon) and then click on mp3.

[gorhill]

This should work in the dev build 1.60.1b11 and above:

||podtrac.com/pts/redirect.mp3/$urlskip=/\/redirect\.mp3\/(.*?\.mp3\b)/ +https

I will add the filter to uBO's privacy list.

[gorhill]

already the exact URL

Sorry, didn't realize this.

the redirection still doesn't occur for https://dts.podtrac.com/redirect.mp3/hozana.si/mp3/berila/794.mp3 and it's still blocked despite -blocked directive was used.

Now I am questioning that we really need -blocked. The main use was to not block certain strict-blocked documents, but since now users are provided the information about the incoming redirects, I don't see the need anymore about this directive.

Should a case like so arise:

||dts.podtrac.com/redirect.mp3/hozana.si/mp3
||podtrac.com^*redirect.mp3/$urlskip=-blocked /\/redirect\.mp3\/(.*?\.mp3\b)/ +https

Then this can already be taken care of without -blocked:

||dts.podtrac.com/redirect.mp3/hozana.si/mp3
||podtrac.com^*redirect.mp3/$urlskip=/\/redirect\.mp3\/(.*?\.mp3\b)/ +https
||dts.podtrac.com/redirect.mp3/hozana.si/mp3$badfilter

---

An argument to keep -blocked is that the request will stay blocked should there be no matching urlskip= filter.

[stephenhawk8054]

@gorhill In this link

http://go.redirectingat.com/?id=355X561&test=off&xcust=94d11aa0976a11efb985b213d4e1265c0INT&url=https%3A%2F%2Fwww.menards.com%2Fmain%2Fstorage-organization%2Fgarage-outdoor-storage%2Fbike-storage%2Ftailgate-bike-rack-carrier-protection-pad-5-bike%2Frmbp500%2Fp-1642874308595068-c-12651.htm

Should we wrap the full link of redirected link in case it's too long?

[gorhill]

On mobile, not preventing multiline pushes the Back/Proceed buttons below the viewport and I think this would be an issue for a lot of people who would think they have to click the link as the only choice.

The link is not in code tags because it's likely to be safe since we added a filter specifically for this. It's the same for links revealed by the magnifier.

[gorhill]

I can let the link wrap but I will limit the height on mobile.

[gorhill]

why ever the landing link have to be clickable

The page is telling me it's going to navigate there, as a user I would be wondering why not just make the link clickable then.

easy copyable for whatever needs

Right-click, "Copy link" / "Copy link address"

[stephenhawk8054]

@gorhill I see the domains are marked bold in the strict-blocked page, is it intended?

[injust]

I saw that https://github.com/gorhill/ublock/wiki/static-filter-syntax has this example:

||example.com/path/to/tracker$urlskip=-blocked ?url

Is there any specific reason for -blocked to be placed before the ?url directive in the example? It still works if placed after ?url.

---

This got me thinking: If you specify multiple ?name directives, how are the -base64, -uricomponent, and -blocked directives applied?

Consider ?foo ?bar -base64. What will this do?

1. Extract the value of foo, then extract the value of bar and base64 decode it
2. Extract the value of foo and base64 decode it, then extract the value of bar and base64 decode it

I haven't tried this and haven't run into any real-life examples, but I wonder if the order of the directives should be formalized a bit.

[stephenhawk8054]

@gorhill Sorry, is it possible to automatically use https for queries (? and &) cases? Because I don't really know when a redirection domain uses http only instead. For example in this case: uBlockOrigin/uAssets@d8457a1 I need to add +https to make it work.

[gorhill]

You mean if the protocol is present and is http:, to automatically upgrade to https:?

[stephenhawk8054]

Yeah exactly.

[stephenhawk8054]

@gorhill In this link:

https://theatre2.prospect2.com/Prod/link-tracker?redirectUrl=aHR0cCUzQSUyRiUyRnRoZWF0cmUyLm9yZyUyRnN1YnNjcmliZSUzRnV0bV9zb3VyY2UlM0RBY3RpdmVDYW1wYWlnbiUyNnV0bV9tZWRpdW0lM0RlbWFpbCUyNnV0bV9jb250ZW50JTNEVGhlYXRyZVNxdWFyZWQlMkJwcm9kdWN0aW9uJTJCb2YlMkJUaWdlciUyQlN0eWxlJTI1MjElMkJiZWdpbnMlMkJGZWJydWFyeSUyQjIzJTI2dXRtX2NhbXBhaWduJTNEUFIlMjUzQSUyQlRpZ2VyJTJCU3R5bGU=&sig=8MyAZqfWi4D4y6jQ74HofpnnUiWyDBP7zh1Mhb1PDywm&iat=1644533531&a=%7C%7C1001253958%7C%7C&account=theatre2%2Eactivehosted%2Ecom&email=x5iTFbWD27OamnRX5Z9MOqSyIXu03Akc1Z8hSpIr818%3D&s=3949e035518f7fd77b4f62360256cda3&i=368A382A2A4505

The URL inside redirectUrl parameter after base64-decode is

http%3A%2F%2Ftheatre2.org%2Fsubscribe%3Futm_source%3DActiveCampaign%26utm_medium%3Demail%26utm_content%3DTheatreSquared%2Bproduction%2Bof%2BTiger%2BStyle%2521%2Bbegins%2BFebruary%2B23%26utm_campaign%3DPR%253A%2BTiger%2BStyle

which needs another layer of uri-decode. Could there be any ways to address this?

Reference: brave/adblock-lists#993

[gorhill]

-uricomponent doesn't work?

[stephenhawk8054]

-uricomponent doesn't work?

I don't think so. I tested

/Prod/link-tracker?redirectUrl=$doc,urlskip=?redirectUrl -base64 -uricomponent

but it doesn't work.

[stephenhawk8054]

Oh wait, sorry my bad. The destination link is http, not https, so this works

/Prod/link-tracker?redirectUrl=$doc,urlskip=?redirectUrl -base64 -uricomponent +https

[stephenhawk8054]

@gorhill Sorry, currently in the result links of Bing search, the URLs look like this:

https://www.bing.com/ck/a?!&&p=ef7ef125927f4fd6e7fd462510d6c90eccceb80d05a10a1250756826aa14611dJmltdHM9MTczMTk3NDQwMA&ptn=3&ver=2&hsh=4&fclid=26de3a35-cd30-618a-182b-2f08cce2603e&psq=lenovo+thinkpad+z16&u=a1aHR0cHM6Ly93d3cubGVub3ZvLmNvbS9qcC9qYS9wL2xhcHRvcHMvdGhpbmtwYWQvdGhpbmtwYWQtei1zZXJpZXMvdGhpbmtwYWQtejE2LSgxNi1pbmNoLWFtZCkvbGVuMTAxdDAwMzc_bXNvY2tpZD0yNmRlM2EzNWNkMzA2MThhMTgyYjJmMDhjY2UyNjAzZQ&ntb=1

I tried

||bing.com/ck/a*^u=a1aHR0c$doc,urlskip=/[?&]u=a1(aHR0c[^&#]+)/ -base64 +https

but not succeeded. Can you reproduce?

[gwarser]

@stephenhawk8054 there is an error or some special encoding in base64 string, around the _.

[stephenhawk8054]

Ah you're right, I forgot to check the atob, it indeed shows error. Not sure what https://www.base64decode.org/ uses since it doesn't show error.

---

Ah ok, it replaces these special values with other values

value.replace(/\s+/g, '').replace(/\-/g, '+').replace(/\_/g, '/')

[gwarser]

_ -> /

---

base64.urlsafe_b64encode(s)

Encode bytes-like object s using the URL- and filesystem-safe alphabet, which substitutes - instead of + and _ instead of / in the standard Base64 alphabet, and return the encoded bytes. The result can still contain =.

https://docs.python.org/3/library/base64.html#base64.urlsafe_b64encode

[freezer2022]

Information: a related subthread: #3499 with a discussion related to added support for a step to extract hash (#) part of a URL. (by the way, there was no need to create a separate thread for it, as it just causes unnecessary fragmentation of the discussion).