-
-
Notifications
You must be signed in to change notification settings - Fork 14k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
curl-impersonate: init at 0.5.4 and replace curl-impersonate-bin #194310
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is very cool and it does stuff I didn't know about :)
-
This built and ran fine on my x86_64 NixOS master, and I didn't see CA issues with either
-ff
or-chrome
. -
I didn't notice anything wrong in
default.nix
, but someone else should take a look. -
Shell completions (I tested zsh) seem to be missing for
curl-impersonate-ff
andcurl-impersonate-chrome
, but they don't seem important enough to block this. -
The update script did not work for me on NixOS master:
nixpkgs/pkgs/tools/networking/curl-impersonate# ./update.py
Traceback (most recent call last):
File "/home/at/trash/nixpkgs/pkgs/tools/networking/curl-impersonate/./update.py", line 86, in <module>
pkgpath = findpath(attr)
File "/home/at/trash/nixpkgs/pkgs/tools/networking/curl-impersonate/./update.py", line 35, in findpath
path = json.loads(subprocess.run(
File "/nix/store/ay0kw1fy63pwa8n06wvm89d4p30jsajy-python3-3.10.7/lib/python3.10/subprocess.py", line 524, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['nix', '--extra-experimental-features', 'nix-command', 'eval', '--json', '.#curl-impersonate.meta.position']' returned non-zero exit status 1.
d86788f
to
d0b08b7
Compare
Yeah I had to dive deep into the nixpkgs code to even figure out how to do some of this stuff... Also I've moved some things around based on the output of nixpkgs-hammering, and I set
I just added a Do completions work for you now?
Oops I think I forgot a flag. Can you try it again with the changes I just force-pushed? |
d0b08b7
to
6a559e5
Compare
{ | ||
"curl-7.84.0.tar.xz" = fetchurl { | ||
url = "https://curl.se/download/curl-7.84.0.tar.xz"; | ||
hash = "sha256-LRGLQ/VHv+W66AbY1HtOWW6lslpsHwgK70n7zYF8Xbg="; | ||
}; | ||
|
||
"brotli-1.0.9.tar.gz" = fetchurl { | ||
url = "https://github.com/google/brotli/archive/refs/tags/v1.0.9.tar.gz"; | ||
hash = "sha256-+ejYHQQFumbRgVKa9CozVPg4yTkJX/mZMNpqqc32/kY="; | ||
}; | ||
|
||
"nss-3.77.tar.gz" = fetchurl { | ||
url = "https://ftp.mozilla.org/pub/security/nss/releases/NSS_3_77_RTM/src/nss-3.77-with-nspr-4.32.tar.gz"; | ||
hash = "sha256-j6UEIeWcTk5x0Z8j3WV3uK+xwT4vm38OyxpVrjd1KV8="; | ||
}; | ||
|
||
"boringssl.zip" = fetchurl { | ||
url = "https://github.com/google/boringssl/archive/3a667d10e94186fd503966f5638e134fe9fb4080.zip"; | ||
hash = "sha256-HsDIkd1x5IH49fUF07dJaabMIMsQygW+NI7GneULpA8="; | ||
}; | ||
|
||
"nghttp2-1.46.0.tar.bz2" = fetchurl { | ||
url = "https://github.com/nghttp2/nghttp2/releases/download/v1.46.0/nghttp2-1.46.0.tar.bz2"; | ||
hash = "sha256-moKXjIcAcbdp8n0riBkct3/clFpRwdaFx/YafhP8Ryk="; | ||
}; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we vendor those downloads instead of using ours?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would worry if we changed the versions too much, the TLS fingerprints may no longer match the browsers it is supposed to be impersonating or the patches from the project may not apply cleanly. Admittedly I did not test that theory, so it is possible using our dependencies could be fine
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi, I just noticed this thread (I'm the author of curl-impersonate). I believe you can use whatever nghttp2 and brotli versions you'd like without it affecting the TLS signatures. I didn't see a case yet where it mattered (when I started curl-impersonate I didn't know that hence the fixed versions). Regarding nss and boringssl, it is important to use known working versions, otherwise the TLS signature might change (new TLS extensions, different default ciphers, and so on).
@@ -0,0 +1,104 @@ | |||
#!/usr/bin/env nix-shell |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are we doing anything complicated in here that would be not that easy with bash, curl and jq?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you could write this in bash, but when I started trying to parse the makefile in bash the code got large and annoying real fast (so I switched to Python). If it's really desired, I can probably rewrite it in bash when I have time
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now that we don't need to fetch other libraries the default nix-update $name might already be enough.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do need to fetch at least BoringSSL and NSS, as @lwthiker mentioned it being sensitive to those versions, and we would still have to parse the makefile to know where to name the sources for nghttp2 and brotli so that the build script can configure them
I'm not sure trying to avoid vendoring those two dependency source URLs is worth it either given it seems to require building them from source anyway, and if we use all of the vendored deps rather than some, automation is somewhat easier for package updates and maintenance burden is lower
I've rewritten the updater script in bash now, though, if that helps
Any thoughts @SuperSandro2000? (thanks you again for the reviews!)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We want to use as little vendored libraries as possible because over time they are usually neglected and they never have nix specific patches which might be very important.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I get that, but to avoid vendoring we would need to also seprately build compatible libs (with some of the curl-impersonate patches for at least boringssl) and bypass the curl-impersonate makefile entirely to write our own build logic (since it expects to build libs with custom options itself)
I'll assess how much work it would be sometime in the next few days to get an idea of what it would take to implement (including upfront difficulty in rewriting all of the relevant source files and in continued maintenance costs)
Also yeah as it is now the static build of NSS in this package is missing the patches to set an NSS_LIB_DIR
(even if that doesn't really provide any value to curl-impersonate). I don't remember other Nix-specific patches this would be missing off the top of my head, but there may be one in boringssl to set a NIX_SSL_DIR
, idk
80d3dd7
to
bdf2f75
Compare
It still wasn't working for me, but with diff --git a/pkgs/tools/networking/curl-impersonate/update.py b/pkgs/tools/networking/curl-impersonate/update.py
index 34fc3e91f34..09e777f812f 100755
--- a/pkgs/tools/networking/curl-impersonate/update.py
+++ b/pkgs/tools/networking/curl-impersonate/update.py
@@ -35,9 +35,12 @@ def findpath(attr):
path = json.loads(subprocess.run(
['nix', '--extra-experimental-features', 'nix-command flakes', 'eval', '--json', f'.#{attr}.meta.position'],
capture_output=True, check=True).stdout).split(':', 1)[0]
- outpath = json.loads(subprocess.run(
+ out = subprocess.run(
['nix', '--extra-experimental-features', 'nix-command flakes', 'eval', '--json', '--impure', '--expr', 'builtins.fetchGit ./.'],
- capture_output=True).stdout)
+ capture_output=True)
+ print(f"stdout is: {out.stdout}")
+ print(f"stderr is: {out.stderr}")
+ outpath = json.loads(out.stdout)
if outpath:
path = path.replace(outpath, os.getcwd())
I saw
and figured out that it does work if you change directory to the nixpkgs root:
|
Yes, thanks! Tested zsh again and they are present for all the binaries. |
Ah, yeah it expects to be run from the root of the repo. I'll see if I can make an adjustment later when I'm at my computer again to fix that and make the script slightly more robust. Thank you for testing it! Edit: I had a minute so I implemented some updates to it for robustness -- it should be good now when run from anywhere in the repo |
2cab60b
to
9a4d2c2
Compare
I've done a little bit more of refactoring/cleaning up things to ensure the little details work (e.g. getting correct rpath if someone tries to use I think with that this is in good shape now that all 4 OfBorg builds pass, including the NixOS test which verifies TLS fingerprints from PCAPs against known browser signatures that passes on the Linux architectures One question I have: Is there a preference for attrs when doing passthru for the firefox/chrome derivations whether they are like (the current) Or alternatively, should those inner passthru derivations be hoisted to the top-level package set with an Also thank you both for the reviews so far, I greatly appreciate them! :) |
9a4d2c2
to
e517379
Compare
I decided on the latter option (hoisting I've successfully tested this on macOS now, but |
Do you want me to put a check for it in the update script so that r-ryantm can handle it? I don't think I can set
Applying the patches to our own versions would require even more custom and fragile makefile parsing against upstream and would require manual intervention for each update, but I suppose it can be done -- the problems below still would matter though
Yes there are several technical reasons, I'll condense everything brought up in reviews and comments above into a numbered list here:
@SuperSandro2000, I really do understand your concerns, but I do not know of a way or am not skilled enough to address these in a way that doesn't make this package unmaintainable immediately after being merged If that just means this does not get merged into nixpkgs and we stick with a prebuilt binary that has similar problems instead (but not tests), then so be it I suppose since your concerns are valid and I don't know what we can do about them Also, I do not want this response to come off as antagonistic, as I do genuinely appreciate all of your reviews and insights on my PRs, so I apologize if it does come off that way. There just seems to be critical miscommunications about what the issue here is. Thank you again for reviewing and continuing to discuss this stuff ❤️ |
Just merge this already. It's a complicated package and it has tests. |
dc31c9f
to
e1c78a8
Compare
That last push was just a minor nit in the NixOS test, but functionally nothing is different. I only changed Edit: I forget that |
e1c78a8
to
bdbf788
Compare
a85a37f
to
bc45899
Compare
I pushed once to use |
bc45899
to
314f2b1
Compare
314f2b1
to
c5cdbb4
Compare
Well, as the point of the package is fingerprint falsification, without comprehensive fingerprint tests we don't know if a no-downside (in the cooperative settings) improvement in a dependency breaks the illusion. (This, though, also makes availability of a binary package useful — even though it's great to have a source build too) |
@Janik-Haag When are you planning to review this? Because I think I'll merge this soon. |
@ofborg eval I have this in my personal overlay and it runs the tests so I know those still pass, but I'll request a fresh eval from ofborg just in case there's some other issue, since the last ofborg eval is fairly stale now |
@7c6f434c feel free to merge |
{ fetchurl }: | ||
|
||
{ | ||
"curl-7.84.0.tar.xz" = fetchurl { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This version has 17 known CVEs https://curl.se/docs/security.html
Are we just going to ignore them or do we want to mark them in meta.knownVulnerabilties?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess it is safe to discard most of them based on the use case (only the protocols the imitated browsers support have any relevance, and I'd argue the use case is even outside the prime usefulness of HSTS support), but indeed there are some unfortunate things, like today's CVE-2023-32001 (well it also applies to our main curl
, we'll see how long it takes for the fix to reach the channel).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm fine with adding meta.knownVulnerabilities
if they are relevant to the build configuration options curl-impersonate uses. I'll investigate and add any relevant ones today or tomorrow
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The only relevant ones for how curl is built that are worth a meta.knownVulnerabilties
that I could find from https://curl.se/docs/vuln-7.84.0.html are:
- CVE-2023-32001 - fopen TOCTOU, CVSSv3 base score 5.5
- CVE-2022-43551 - HSTS bypass, CVSSv3 base score 7.5
- CVE-2022-42916 - HSTS bypass, CVSSv3 base score 7.5
I'll push in a minute to update this PR for this and the removal of pytest
from PYTHONPATH
nativeBuildInputs = [ unzip ]; | ||
|
||
proxyVendor = true; | ||
}).go-modules; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
IIRC this go renamed on staging to goModules
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Checked: that's now in staging-next
even. Unfortunate timing indeed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good to know, I missed that change
We can either merge as-is and I open a new PR to staging-next after master gets merged into it to update the usage here, or we can wait for staging and then I rebase
I don't have much preference either way
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would prefer the first option with merge as-is.
c5cdbb4
to
e28c49d
Compare
(Looks like master→staging-next merge today was after this merge) |
Staging-next followup in #245070 |
Description of changes
This replaces the
curl-impersonate-bin
package with a from-source build that should work on more than just x86_64-linuxNotes:
(fetchGoModules { ... }).go-modules
to get at the internal fetcher and grab the boringssl deps but still letting curl-impersonate's build process actually do the buildingTheoretically this can build on Darwin, but I haven't tried it and I am not certain ca certs would get picked up correctlyEdit: It looks like only curl-impersonate-chrome built on Darwin in ofborg and only on aarch64-darwin (due to old sdk on x86_64), and curl-impersonate-ff failed because libtool is missing-static
on both for some reasoncc: @deliciouslytyped @ivan
Closes #169150
Things done
sandbox = true
set innix.conf
? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD"
. Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/
)nixos/doc/manual/md-to-db.sh
to update generated release notes