Add tor_hidden_service.md

[jhunkeler]

This PR intends to implement issue #187 . The only thing I'm iffy on right now is how to hook this up to the main documentation. Configuring Tor and running an onion site probably isn't everybody's cup of tea.

A second pair of eyes on this wouldn't hurt either. For some reason I wasn't able to proxy the .onion address to http://lemmy. I suspect it was related to SSL, because it isn't supported on the Tor network (last I remember anyway, I could be wrong).

• Which page/section should link to tor_hidden_service.md?
• How should it be mentioned in the documentation? i.e., "If you want to proxy Lemmy through Tor click here."
• Can someone validate what I've done here? I didn't see anything out of the ordinary security-wise in my initial round of testing.

Feel free to commit directly to my branch if you want to.

[Nutomic]

Is your instance not reachable over clearnet? Then federation with other instances can only work if they use tor proxy for outgoing requests (not currently supported). So effectively federation is only possible over clearnet for now.

You can see how Mitra handles Tor. Its necessary to set a onion_proxy_url config value to federate over Tor.

[jhunkeler]

My instance is reachable over clearnet. I assumed federation over Tor was impossible, so yes all federation tasks are still handled by the clearnet HTTPS server.

Full federation over Tor would be nice for a siloed instance but in my case, if I'm only hosting the same site "twice" the federation traffic would become redundant.

[Nutomic]

Okay then mention that please to avoid any confusion. Additionally you can mention in the beginning that it requires an existing Lemmy installation according to one of the other installation guides.

Also you need to link the file from SUMMARY.md so it shows up in the navigation.

[jhunkeler]

Okay, I mentioned the requirement at the beginning and added a link to SUMMARY.md

[Nutomic]

Current instructions are for Ubuntu, so I would prefer to use apt here for consistency.

[jhunkeler]

Done.

[Nutomic]

You dont need to explain sudo.

[k4r4b3y]

Can you also add an example server config for caddyserver?

[monerobull]

I've used the docker-compose from here: https://github.com/LemmyNet/lemmy-ansible/blob/main/templates/docker-compose.yml
How would I go about setting this up? I'm already stuck at the docker compose file since i don't have a proxy service.

[maltfield]

@jhunkeler thanks for doing this!

I suspect it was related to SSL, because it isn't supported on the Tor network (last I remember anyway, I could be wrong).

You can run a .onion with SSL, but it doesn't add much security because Onion Services already have a better PKI encryption solution than SSL's X.509. Here's a list of popular Onion Services. Note some have https and some have http

• https://github.com/alecmuffett/real-world-onion-sites

Currently you have to pay for an https cert, but there is a pending ticket for support of Onion Services in Let's Encrypt

• Support for FQDNs under .onion letsencrypt/boulder#4620

[hackerncoder]

HiddenServicePort 10080 127.0.0.1:80 makes no sense with the rest of the docs, you probably want to swap 10080 and 80.

Maybe? (I don't know much about this) the limit_req_zone can be too low. Nginx won't be able to distinguish between Tor users, since each request just comes from 127.0.0.1.

I was going to ask why you're making both docker and nginx use 10080, but it's meant to be mutually exclusive. If one uses nginx as a reverse proxy, one wants the latter, and if not one should edit the docker compose. A small note that both of these are not meant to be done at the same time, might be good.

Strict-Transport-Security makes no sense. As maltfield said.

[maltfield]

Nginx won't be able to distinguish between Tor users

I use limit_req_zone with my Onion Services and haven't noticed any issues. I would expect Tor to pass the exit node's IP to the web server.

So it wouldn't be able to distinguish between users that share the same exit node IP in their tor circuit, but it should be able to distinguish between most users (currently there's about 5 million users sharing over 2,000 exit nodes)

[jhunkeler]

HiddenServicePort 10080 127.0.0.1:80 makes no sense with the rest of the docs, you probably want to swap 10080 and 80.

This was definitely a typo, sorry. In my own configuration I have HiddenServicePort 80 127.0.0.1:10080.

Maybe? (I don't know much about this) the limit_req_zone can be too low. Nginx won't be able to distinguish between Tor users, since each request just comes from 127.0.0.1.

I haven't observed any wonkiness with the limit_req_zone settings either.

I was going to ask why you're making both docker and nginx use 10080, but it's meant to be mutually exclusive. If one uses nginx as a reverse proxy, one wants the latter, and if not one should edit the docker compose. A small note that both of these are not meant to be done at the same time, might be good.

If you try to forward the raw HTTPS traffic on Tor smoke pours out. I thought I mentioned this in the docs but if not I can add a blurb.

Strict-Transport-Security makes no sense. As maltfield said.

I agree.

[jhunkeler]

Rereading what you wrote @hackerncoder... I think I understand what you mean about the limit_req_zone setting.

There is no "exit node" when you access the onion site directly over tor. The traffic arrives at nginx's door through tor, thus 127.0.0.1 -> 172.16.0.0/16 -> nginx_server_here, not an external IP. So in the end it looks like this settings just limits traffic arriving on the loopback, not a unique/traceable user.

Any suggestions for how to handle it? Is there an alternative method? I don't think it'd be a good idea to stop rate limiting altogether or it'll leave you open to getting DoS'd.

[maltfield]

Can you dump a complete request to your web server that originated via a client using Tor Browser connecting to your Onion Service?

I'd specifically be interested in headers like X-Forwared-For and X-Real-IP.

There is no "exit node" when you access the onion site directly over tor.

By default there's two exit nodes. Both the client and the server form their own 3-hop Onion Circuit.

Please share the complete request here to help others.

[hackerncoder]

I'd specifically be interested in headers like X-Forwared-For and X-Real-IP.

I'm pretty sure there is none. Tor just forwards TCP packets. Edit: right, you mean those in the nginx config, they would be filled with 127.0.0.1 (when tor -> nginx on "root", without any docker or other things that might change the address)

By default there's two exit nodes.

No.

The traffic arrives at nginx's door through tor, thus 127.0.0.1 -> 172.16.0.0/16 -> nginx_server_here, not an external IP

Oh... nginx is running inside docker?

---

edit:

If you try to forward the raw HTTPS traffic on Tor smoke pours out

I don't understand what that is supposed to mean.

So in the end it looks like this settings just limits traffic arriving on the loopback, not a unique/traceable user.

Any suggestions for how to handle it? Is there an alternative method? I don't think it'd be a good idea to stop rate limiting altogether or it'll leave you open to getting DoS'd.

Yea, that's why I said it might be too low. Depending on the amount of users the server gets.

[jhunkeler]

Oh... nginx is running inside docker?

Yes nginx is running inside of docker.

I don't understand what that is supposed to mean.

Lemmy's stock nginx configuration wants to redirect 80 to 443. If you create a service like HiddenService 80 0.0.0.0:80 Tor Browser throws a certificate error and kills the connection. That's how I ended up duplicating the stock config with tor-specific settings listening on 10080.

i.e. the same config minus HTTPS.

Any suggestions for how to handle it? Is there an alternative method? I don't think it'd be a good idea to stop rate limiting altogether or it'll leave you open to getting DoS'd.

Yea, that's why I said it might be too low. Depending on the amount of users the server gets.

My server barely gets any meaningful traffic. I guess that explains why I haven't noticed anything out of the ordinary. 😁

Do you have any recommendations for what to set the limits to? I can RTFM but its difficult to know what's going to work for a high traffic onion site. I could be wrong... I feel like a lot of people are hosting Lemmy on potatoes so I don't want to set the defaults too high, or too loose.

[hackerncoder]

Lemmy's stock nginx configuration wants to redirect 80 to 443 [...]

Right, right. Yup. I was thinking that nginx ran outside of docker, that's what my original comment was talking about.

Do you have any recommendations for what to set the limits to?

Nope.

[jhunkeler]

@maltfield

tor -> onion

Traffic never leaves tor, so it arrives on the loopback interface. Traffic originates from "within" the server.

Client side

$ torsocks wget -q -S -O /dev/null  http://ONIONADDRESS/
  HTTP/1.1 200 OK
  Server: nginx
  Date: Mon, 17 Jul 2023 23:24:53 GMT
  Content-Type: text/html; charset=utf-8
  Content-Length: 102942
  Connection: keep-alive
  Vary: Accept-Encoding
  X-Powered-By: Express
  Cache-Control: public, max-age=5
  Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *; media-src * data:
  ETag: W/"1921e-T1/h9gnZPgPyD+FM0Ml2Z+dfPHo"
  Referrer-Policy: same-origin
  X-Content-Type-Options: nosniff
  X-Frame-Options: DENY
  X-XSS-Protection: 1; mode=block

Server side

lemmy-proxy-1  | 172.*.0.1 - - [17/Jul/2023:23:24:53 +0000] "GET / HTTP/1.1" 200 102942 "-" "Wget/1.21.4"

tor -> internet

Tor traffic passes through an exit node like normal internet traffic. The exit node IP is logged by nginx.

Client side

$ torsocks wget -q -S -O /dev/null http://INTERNETHOSTNAME/
  HTTP/1.1 301 Moved Permanently
  Server: nginx/1.25.1
  Date: Mon, 17 Jul 2023 23:35:42 GMT
  Content-Type: text/html
  Content-Length: 169
  Connection: keep-alive
  Location: https://INTERNETHOSTNAME/
  HTTP/1.1 200 OK
  Server: nginx
  Date: Mon, 17 Jul 2023 23:35:42 GMT
  Content-Type: text/html; charset=utf-8
  Content-Length: 102942
  Connection: keep-alive
  Vary: Accept-Encoding
  X-Powered-By: Express
  Cache-Control: public, max-age=5
  Content-Security-Policy: default-src 'self'; manifest-src *; connect-src *; img-src * data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; form-action 'self'; base-uri 'self'; frame-src *; media-src * data:
  ETag: W/"1921e-HSNpfmpb8hJPzxXW4vHfn1NHnhQ"
  Onion-Location: http://ONIONADDRESS/

Server side

lemmy-proxy-1  | 162.247.74.* - - [17/Jul/2023:23:35:42 +0000] "GET / HTTP/1.1" 301 169 "-" "Wget/1.21.4"
lemmy-proxy-1  | 162.247.74.* - - [17/Jul/2023:23:35:43 +0000] "GET / HTTP/1.1" 200 102942 "-" "Wget/1.21.4"

[maltfield]

I've been monitoring logs from my Onion Services and confirmed that, sure enough, it's rate-limiting 127.0.0.1. Maybe that explains why every few days my .onion monitoring alerts tell me my site is down, but I can't ever reproduce it. Huh.

2023/07/17 23:06:58 [error] 1388479#1388479: *1175178 limiting connections by zone "addr", client: 127.0.0.1, server: www.buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion, request: "GET /wp-content/uploads/sites/8/2021/10/product_as-bka-kit_2021-10_002.png HTTP/1.1", host: "www.buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion", referrer: "http://www.buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion/blog/"    

I haven't confirmed with a dump myself, but honestly I'm surprised that Tor wouldn't set X-Forwarded-For to make rate limiting easier. Instead, they have an article describing DoS defenses for Onion Services. Towards the bottom it says web servers can rate-limit using HiddenServiceExportCircuitID

• https://community.torproject.org/onion-services/advanced/dos/

It looks like HiddenServiceExportCircuitID was created 4 years ago to allow Onion Services to uniquely identify their clients (eg for the purposes of rate limiting)

• https://gitlab.torproject.org/legacy/trac/-/issues/4700

I haven't found any guides that describe how to use it with, say, nginx's limit_req_zone

[hackerncoder]

@maltfield See "Accepting the PROXY protocol" on the nginx docs.

---

Edit:

but honestly I'm surprised that Tor wouldn't set X-Forwarded-For to make rate limiting easier

First and foremost: Tor is a TCP proxy. And even if it were to, that would be impossible for HTTPS.

[jhunkeler]

From tor(1):

       HiddenServiceExportCircuitID protocol
           The onion service will use the given protocol to expose the global circuit identifier
           of each inbound client circuit. The only protocol supported right now 'haproxy'. This
           option is only for v3 services. (Default: none)

           The haproxy option works in the following way: when the feature is enabled, the Tor
           process will write a header line when a client is connecting to the onion service.
           The header will look like this:

           "PROXY TCP6 fc00:dead:beef:4dad::ffff:ffff ::1 65535 42\r\n"

           We encode the "global circuit identifier" as the last 32-bits of the first IPv6
           address. All other values in the header can safely be ignored. You can compute the
           global circuit identifier using the following formula given the IPv6 address
           "fc00:dead:beef:4dad::AABB:CCDD":

           global_circuit_id = (0xAA << 24) + (0xBB << 16) + (0xCC << 8) + 0xDD;

           In the case above, where the last 32-bits are 0xffffffff, the global circuit
           identifier would be 4294967295. You can use this value together with Tor’s control
           port to terminate particular circuits using their global circuit identifiers. For
           more information about this see control-spec.txt.

           The HAProxy version 1 protocol is described in detail at
           https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

The ^nginx doc link^ for future reference:
https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/

[maltfield]

Cool, I was able to set this up. It required 4 changes:

1. torrc

First I had to update my torrc file by adding this to the end of every HiddenServiceDir block

HiddenServiceExportCircuitID haproxy

2. listen lines

I had to append every listen line in nginx with proxy_protocol. So the lines that used-to say this in the server blocks of my nginx configs

   listen 127.0.0.1:8050;
   listen [::1]:8050;

Now say this

   listen 127.0.0.1:8050 proxy_protocol;
   listen [::1]:8050 proxy_protocol;

3. logging changes (optional?)

To log the "IP Address" of the Tor client (instead of 127.0.0.1), I had to setup a new "tor" logging format in nginx.conf

   log_format  tor  '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

And then the same server blocks whoose listen lines I updated above, I changed their access logs line from this

   access_log /var/log/nginx/www.buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion/access.log main;

to this

   access_log /var/log/nginx/www.buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion/access.log tor;

4. proxy changes

Finally, in the location block where I configure nginx to reach back to the backend, I had to add the following

      proxy_pass http://127.0.0.1:8080;
      ...
      # for tor and upstream PROXY servers (haproxy protocol)
      #  * https://docs.nginx.com/nginx/admin-guide/load-balancer/using-proxy-protocol/
      proxy_set_header X-Real-IP       $proxy_protocol_addr;
      proxy_set_header X-Forwarded-For $proxy_protocol_addr;

Results

As a result, my backend now sees these X-Real-IP and X-Forwarded-For headers on requests made to my Onion Services . These aren't real IPv6 addresses, but they should be unique to a tor circuit and allow for rate-limiting. Indeed, tracking like this is better than tracking the exit node's IP, as it's actually unique to every client.

For example:

X-Real-IP: fc00:dead:beef:4dad::0:3a
X-Forwarded-For: fc00:dead:beef:4dad::0:3a

I'm also getting these "IPs" logged to my nginx access logs like so:

fc00:dead:beef:4dad::0:5d - - [18/Jul/2023:01:42:47 +0000] "GET /cart HTTP/1.1" 200 30630 "http://buskillvampfih2iucxhit3qp36i2zzql3u6pmkeafvlxs3tlmot5yad.onion/canary-005" "Mozilla/5.0 (Windows NT 6.1; rv:68.0) Gecko/20100101 Firefox/68.0" "-"   

I'll have to wait and see how this affects limit_req_zone :)