TLSFingerprint is lost on all requests after the first #290

adaml881 · 2023-09-26T14:56:36Z

I've noticed 2 issues relating to TLS Fingerprints. I hope someone could advise if it's a bug or if it's something I'm doing wrong.

When using SetTLSFingerprintChrome() on the client It's only effective for the first request that the client makes

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://client.tlsfingerprint.io"

type tlsFingerprintResponse struct {
	NormID string `json:"norm_id"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
}

func main() {
	client := req.C().EnableTraceAll()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, _ := client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)
}

Gives the output

ffa2ee96ff7b42d4  # Good Chrome fingerprint
0b61c64a14c94ede  # Bad Go default fingerprint
0b61c64a14c94ede  # Bad Go default fingerprint

2 When you use EnableForceHTTP2() it prevents the SetTLSFingerprintChrome from working on all requests

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://client.tlsfingerprint.io"

type tlsFingerprintResponse struct {
	NormID string `json:"norm_id"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
}

func main() {
	client := req.C().EnableTraceAll().EnableForceHTTP2()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, _ := client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)
}

0b61c64a14c94ede  # Bad Go default fingerprint
0b61c64a14c94ede  # Bad Go default fingerprint
0b61c64a14c94ede  # Bad Go default fingerprint

The text was updated successfully, but these errors were encountered:

packman80 · 2023-09-28T10:03:47Z

same issue

imroc · 2023-09-30T08:55:47Z

I will take the time to verify and research the solution

imroc · 2023-09-30T09:11:39Z

The first one, it's because client reuse connection by default, no tls handshake after the first request. If you disable keep alive:

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://client.tlsfingerprint.io"

type tlsFingerprintResponse struct {
	NormID string `json:"norm_id"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
}

func main() {
	client := req.C().EnableTraceAll().DisableKeepAlives()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, _ := client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)
}

will got:

ffa2ee96ff7b42d4 # Good Chrome fingerprint
ffa2ee96ff7b42d4 # Good Chrome fingerprint
ffa2ee96ff7b42d4 # Good Chrome fingerprint

imroc · 2023-09-30T09:15:13Z

The second one is a real issue, will fix it in the next release.

adaml881 · 2023-09-30T09:29:32Z

Thank you @imroc

adaml881 · 2023-09-30T11:12:03Z

@imroc

Regarding issue 1. I've traced it back a little to try to find out why a keep alive was causing the bad tls fingerprint on requests after the first.

It appears to be sending a standard crypto/tls handshake even when it reuses a connection?

I added a Println to utls/u_conn.go and crypto/tls/handshake_client_tls13.go

I also noticed for these requests, the trace tells us that the connection was not reused. Perhaps the connection was reused but the TLS layer was closed?

Response

UConn.clientHandshake -> utls u_conn.go

&{[1 0 1 252 3 3 247 136 222 46 12 221 162 19 205 153 6 77 79 79 120 147 34 98 114 121 89 134 250 147 83 127 43 151 177 99 250 79 32 12 204 129 8 27 72 205 113 27 140 235 210 148 4 138 53 137 43 187 245 20 234 31 72 110 94 87 211 61 205 11 194 0 32 154 154 19 1 19 2 19 3 192 43 192 47 192 44 192 48 204 169 204 168 192 19 192 20 0 156 0 157 0 47 0 53 1 0 1 147 250 250 0 0 0 23 0 0 0 0 0 29 0 27 0 0 24 99 108 105 101 110 116 46 116 108 115 102 105 110 103 101 114 112 114 105 110 116 46 105 111 0 13 0 18 0 16 4 3 8 4 4 1 5 3 8 5 5 1 8 6 6 1 68 105 0 5 0 3 2 104 50 0 11 0 2 1 0 255 1 0 1 0 0 43 0 7 6 58 58 3 4 3 3 0 10 0 10 0 8 26 26 0 29 0 23 0 24 0 35 0 0 0 5 0 5 1 0 0 0 0 0 16 0 14 0 12 2 104 50 8 104 116 116 112 47 49 46 49 0 27 0 3 2 0 2 0 51 0 43 0 41 26 26 0 1 0 0 29 0 32 235 97 224 36 255 161 25 42 232 233 222 180 0 12 149 234 156 251 222 115 27 224 39 179 50 115 25 100 79 141 39 84 0 18 0 0 0 45 0 2 1 1 10 10 0 1 0 0 21 0 191 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] 771 [247 136 222 46 12 221 162 19 205 153 6 77 79 79 120 147 34 98 114 121 89 134 250 147 83 127 43 151 177 99 250 79] [12 204 129 8 27 72 205 113 27 140 235 210 148 4 138 53 137 43 187 245 20 234 31 72 110 94 87 211 61 205 11 194] [39578 4865 4866 4867 49195 49199 49196 49200 52393 52392 49171 49172 156 157 47 53] [0] client.tlsfingerprint.io true [6682 29 23 24] [0] true [] [1027 2052 1025 1283 2053 1281 2054 1537] [] true [] true [h2 http/1.1] true [14906 772 771] [] [{6682 [0]} {29 [235 97 224 36 255 161 25 42 232 233 222 180 0 12 149 234 156 251 222 115 27 224 39 179 50 115 25 100 79 141 39 84]}] false [1] [] [] [] false}

Fingerprint: ffa2ee96ff7b42d4
IsConnReused false


Running clientHandshakeStateTLS13.handshake() crypto/tls/handshake_client_tls13.go

&{[1 0 1 25 3 3 75 18 27 173 216 71 51 38 141 47 239 129 225 123 230 104 231 137 19 26 4 53 2 219 59 78 69 205 136 42 16 94 32 60 198 205 28 42 110 175 208 122 181 134 117 212 31 196 32 61 177 164 9 175 119 69 28 126 20 138 250 122 62 244 183 0 38 192 43 192 47 192 44 192 48 204 169 204 168 192 9 192 19 192 10 192 20 0 156 0 157 0 47 0 53 192 18 0 10 19 1 19 2 19 3 1 0 0 170 0 0 0 29 0 27 0 0 24 99 108 105 101 110 116 46 116 108 115 102 105 110 103 101 114 112 114 105 110 116 46 105 111 0 5 0 5 1 0 0 0 0 0 10 0 10 0 8 0 29 0 23 0 24 0 25 0 11 0 2 1 0 0 13 0 26 0 24 8 4 4 3 8 7 8 5 8 6 4 1 5 1 6 1 5 3 6 3 2 1 2 3 255 1 0 1 0 0 16 0 14 0 12 8 104 116 116 112 47 49 46 49 2 104 50 0 18 0 0 0 43 0 5 4 3 4 3 3 0 51 0 38 0 36 0 29 0 32 135 210 75 56 226 132 193 26 22 129 231 205 84 79 135 147 197 127 238 226 23 171 140 140 0 238 13 119 56 196 213 79] 771 [75 18 27 173 216 71 51 38 141 47 239 129 225 123 230 104 231 137 19 26 4 53 2 219 59 78 69 205 136 42 16 94] [60 198 205 28 42 110 175 208 122 181 134 117 212 31 196 32 61 177 164 9 175 119 69 28 126 20 138 250 122 62 244 183] [49195 49199 49196 49200 52393 52392 49161 49171 49162 49172 156 157 47 53 49170 10 4865 4866 4867] [0] client.tlsfingerprint.io true [29 23 24 25] [0] false [] [2052 1027 2055 2053 2054 1025 1281 1537 1283 1539 513 515] [] true [] [http/1.1 h2] true [772 771] [] [{29 [135 210 75 56 226 132 193 26 22 129 231 205 84 79 135 147 197 127 238 226 23 171 140 140 0 238 13 119 56 196 213 79]}] false [] [] []}

Fingerprint: ceedf21a052638a0
IsConnReused false


Running clientHandshakeStateTLS13.handshake() crypto/tls/handshake_client_tls13.go

&{[1 0 1 25 3 3 255 145 17 150 152 37 23 172 9 2 154 146 160 184 109 249 128 228 158 191 141 223 105 127 234 141 253 34 238 155 65 209 32 27 190 164 121 109 163 180 24 60 140 144 45 166 135 11 45 127 62 153 196 170 240 176 255 73 23 60 234 202 103 236 122 0 38 192 43 192 47 192 44 192 48 204 169 204 168 192 9 192 19 192 10 192 20 0 156 0 157 0 47 0 53 192 18 0 10 19 1 19 2 19 3 1 0 0 170 0 0 0 29 0 27 0 0 24 99 108 105 101 110 116 46 116 108 115 102 105 110 103 101 114 112 114 105 110 116 46 105 111 0 5 0 5 1 0 0 0 0 0 10 0 10 0 8 0 29 0 23 0 24 0 25 0 11 0 2 1 0 0 13 0 26 0 24 8 4 4 3 8 7 8 5 8 6 4 1 5 1 6 1 5 3 6 3 2 1 2 3 255 1 0 1 0 0 16 0 14 0 12 8 104 116 116 112 47 49 46 49 2 104 50 0 18 0 0 0 43 0 5 4 3 4 3 3 0 51 0 38 0 36 0 29 0 32 167 251 198 8 140 33 83 72 28 131 198 89 114 164 41 178 113 106 237 16 108 136 175 233 244 4 165 168 183 59 233 64] 771 [255 145 17 150 152 37 23 172 9 2 154 146 160 184 109 249 128 228 158 191 141 223 105 127 234 141 253 34 238 155 65 209] [27 190 164 121 109 163 180 24 60 140 144 45 166 135 11 45 127 62 153 196 170 240 176 255 73 23 60 234 202 103 236 122] [49195 49199 49196 49200 52393 52392 49161 49171 49162 49172 156 157 47 53 49170 10 4865 4866 4867] [0] client.tlsfingerprint.io true [29 23 24 25] [0] false [] [2052 1027 2055 2053 2054 1025 1281 1537 1283 1539 513 515] [] true [] [http/1.1 h2] true [772 771] [] [{29 [167 251 198 8 140 33 83 72 28 131 198 89 114 164 41 178 113 106 237 16 108 136 175 233 244 4 165 168 183 59 233 64]}] false [] [] []}

Fingerprint: ceedf21a052638a0
IsConnReused false

Code

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://client.tlsfingerprint.io"

type tlsFingerprintResponse struct {
	NormID string `json:"norm_id"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
}

func main() {
	client := req.C().EnableTraceAll() //.DisableKeepAlives()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, _ := client.R().Get(urlString)
	PrintFingerprint(resp.Body)
	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)
	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, _ = client.R().Get(urlString)
	PrintFingerprint(resp.Body)
	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)
}
```

imroc · 2023-10-04T09:29:23Z

It seems that https://client.tlsfingerprint.io is down, cannot reproduce the issue. Tried other websites, the second IsConnReused is true, maybe https://client.tlsfingerprint.io returns Connection: Close header, any alternative site to https://client.tlsfingerprint.io exists?

adaml881 · 2023-10-04T12:02:07Z

Damn! I'm sure it will be back up over the next few days, they've had downtime before

The only site i've found that is similar is https://tls.browserleaks.com/tls. It doesn't show the same problem, but it does highlight a different issue.

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://tls.browserleaks.com/tls"

type tlsFingerprintResponse struct {
	NormID string `json:"ja3_hash"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
}

func main() {
	client := req.C().EnableTraceAll() //.DisableKeepAlives()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, err := client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 1", err)
	} else {
		PrintFingerprint(resp.Body)
	}
	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, err = client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 2", err)
	} else {
		PrintFingerprint(resp.Body)
	}

	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, err = client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 3", err)
	} else {
		PrintFingerprint(resp.Body)
	}

	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)
}

imroc · 2023-10-06T06:24:24Z

err on request 2 Get "https://tls.browserleaks.com/tls": unexpected EOF
IsConnReused true

It seems the server does not allow reuse connection (cuz server cannot detect tls info if reuse connection)

imroc · 2023-10-06T06:30:07Z

And ja3_hash is not equal with norm_id, If you use SetTLSFingerprintChrome, it is expected that the ja3_hash of each request is different (Chrome will shuffle TLS extensions)

adaml881 · 2023-10-06T08:23:06Z

Sorry you're right, I should have used ja3n_hash (normalised version)

In this example the hash still changes on the third request (after the second one failing)
The first ja3n_hash is correct, and matches my fingerprint in browser no matter how many times I refresh.
I think this is evidence of the same issue I was seeing on the other URL

package main

import (
	"encoding/json"
	"fmt"
	"io"

	"github.com/imroc/req/v3"
)

const urlString string = "https://tls.browserleaks.com/tls"

type tlsFingerprintResponse struct {
	NormID   string `json:"ja3n_hash"`
	NormText string `json:"ja3n_text"`
}

func PrintFingerprint(respBody io.Reader) {
	body, _ := io.ReadAll(respBody)

	var res tlsFingerprintResponse
	json.Unmarshal(body, &res)

	fmt.Println(res.NormID)
	fmt.Println(res.NormText)
	fmt.Println("")
}

func main() {
	client := req.C().EnableTraceAll() //.DisableKeepAlives()

	client = client.SetUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36").
		SetTLSFingerprintChrome()

	resp, err := client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 1", err)
	} else {
		PrintFingerprint(resp.Body)
	}
	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, err = client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 2", err)
	} else {
		PrintFingerprint(resp.Body)
	}

	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)

	resp, err = client.R().Get(urlString)
	if err != nil {
		fmt.Println("err on request 3", err)
	} else {
		PrintFingerprint(resp.Body)
	}

	fmt.Println("IsConnReused", resp.TraceInfo().IsConnReused)
}

…290)

imroc · 2023-10-06T10:39:10Z

@adaml881 Nice catch! I reproduced this issue and fixed in the latest release, upgrade and try it!

adaml881 · 2023-10-06T10:46:04Z

@imroc great job. I've tested both original issues from the start of this thread. Can confirm they are fixed. Thank you so much

imroc added a commit that referenced this issue Oct 6, 2023

fix SetTLSFingerprintXXX does not take effect in subsequent requests(#…

bc2bf86

…290)

adaml881 closed this as completed Oct 6, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TLSFingerprint is lost on all requests after the first #290

TLSFingerprint is lost on all requests after the first #290

adaml881 commented Sep 26, 2023

packman80 commented Sep 28, 2023

imroc commented Sep 30, 2023

imroc commented Sep 30, 2023

imroc commented Sep 30, 2023

adaml881 commented Sep 30, 2023

adaml881 commented Sep 30, 2023

imroc commented Oct 4, 2023

adaml881 commented Oct 4, 2023

imroc commented Oct 6, 2023

imroc commented Oct 6, 2023

adaml881 commented Oct 6, 2023

imroc commented Oct 6, 2023

adaml881 commented Oct 6, 2023

TLSFingerprint is lost on all requests after the first #290

TLSFingerprint is lost on all requests after the first #290

Comments

adaml881 commented Sep 26, 2023

packman80 commented Sep 28, 2023

imroc commented Sep 30, 2023

imroc commented Sep 30, 2023

imroc commented Sep 30, 2023

adaml881 commented Sep 30, 2023

adaml881 commented Sep 30, 2023

imroc commented Oct 4, 2023

adaml881 commented Oct 4, 2023

imroc commented Oct 6, 2023

imroc commented Oct 6, 2023

adaml881 commented Oct 6, 2023

imroc commented Oct 6, 2023

adaml881 commented Oct 6, 2023