Nginx configuration for the image proxy is not working

[josecelano]

In the droplet compose example the image proxy for torrent description is not working:

https://index.torrust-demo.com/api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png

[josecelano]

It looks like a problem with the Ngxin proxy configuration. I can connect directly to the API service from the server:

$ curl -i http://127.0.0.1:3001/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png
HTTP/1.1 200 OK
content-type: image/png
content-length: 2687
access-control-allow-origin: *
vary: origin
vary: access-control-request-method
vary: access-control-request-headers
access-control-expose-headers: *
date: Thu, 14 Dec 2023 14:39:21 GMT

Warning: Binary output can mess up your terminal. Use "--output -" to tell 
Warning: curl to output it to your terminal anyway, or consider "--output 
Warning: <FILE>" to save to a file.

[josecelano]

I have enabled the Nginx debug mode with:

server
{
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        server_name index.torrust-demo.com;

	error_log /var/log/nginx/error.log debug; <- THIS NEW LINE

	server_tokens off;
        ...

        location ^~/api/
        {
                rewrite ^/api/(.*)$ /$1 break;
		try_files $uri @index;
        }

        ... 

        location @index
        {
                proxy_pass http://index:3001;
                add_header X-Frame-Options "SAMEORIGIN" always;
                add_header X-XSS-Protection "1; mode=block" always;
                add_header X-Content-Type-Options "nosniff" always;
                add_header Referrer-Policy "no-referrer-when-downgrade" always;
                add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
                #add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
                # enable strict transport security only if you understand the implications
        }

        ...

        root /var/www/html;
        index index.html index.htm index.nginx-debian.html;
}

And it seems Nginx is removing one slash in the redirection URL:

2023/12/14 16:53:39 [notice] 20#20: *1 "^/api/(.*)$" matches "/api/v1/proxy/image/https:/raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png", client: 2.137.102.79, server: index.torrust-demo.com, request: "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0", host: "index.torrust-demo.com", referrer: "https://index.torrust-demo.com/torrent/443c7602b4fde83d1154d6d9da48808418b181b6/ubuntu-2304-desktop-amd64"
2023/12/14 16:53:39 [notice] 20#20: *1 rewritten data: "/v1/proxy/image/https:/raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png", args: "", client: 2.137.102.79, server: index.torrust-demo.com, request: "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0", host: "index.torrust-demo.com", referrer: "https://index.torrust-demo.com/torrent/443c7602b4fde83d1154d6d9da48808418b181b6/ubuntu-2304-desktop-amd64"
2.137.102.79 - - [14/Dec/2023:16:53:39 +0000] "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0" 404 0 "https://index.torrust-demo.com/torrent/443c7602b4fde83d1154d6d9da48808418b181b6/ubuntu-2304-desktop-amd64" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "-"

As you can see is using /v1/proxy/image/https:/raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png with https:/r... instead of https://r...

[josecelano]

By enabling the option merge_slashes off; inside the server block I get this log limes:

2.137.102.79 - - [14/Dec/2023:17:18:06 +0000] "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0" 404 0 "https://index.torrust-demo.com/torrent/edit/443c7602b4fde83d1154d6d9da48808418b181b6" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" "-"
2023/12/14 17:18:06 [notice] 19#19: *9 "^/api/(.*)$" matches "/api/v1/proxy/image/https://raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png", client: 2.137.102.79, server: index.torrust-demo.com, request: "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0", host: "index.torrust-demo.com", referrer: "https://index.torrust-demo.com/torrent/edit/443c7602b4fde83d1154d6d9da48808418b181b6"
2023/12/14 17:18:06 [notice] 19#19: *9 rewritten data: "/v1/proxy/image/https://raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png", args: "", client: 2.137.102.79, server: index.torrust-demo.com, request: "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/2.0", host: "index.torrust-demo.com", referrer: "https://index.torrust-demo.com/torrent/edit/443c7602b4fde83d1154d6d9da48808418b181b6"

It seems that now it's using this URL:

/v1/proxy/image/https://raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png

It adds two slashes but It's not working because this URL:

http://index.torrust-demo.com:3001/v1/proxy/image/https://raw.githubusercontent.com/torrust/torrust-index/develop/docs/media/torrust_logo.png

does not work. hHe URL of the image has to be URL encoded in the PATH.

[josecelano]

It seems I have to change the Nginx config:

• https://stackoverflow.com/q/28684300/3012842
• https://trac.nginx.org/nginx/ticket/727
• https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

Because:

Characters which are not allowed to be passed unescaped are escaped after URI transformation. The ":" and "/" characters are allowed within URI (though may have special meaning in some cases).

It seems I have to use the variable $request_uri which contains the original unescaped values.

[josecelano]

ChatGTP suggested to do this:

location /api/ {
    # Capture the part after '/api/' and store it in a variable
    if ($request_uri ~* "^/api/(.+)") {
        set $captured_path $1;
    }

    # Strip off the '/api/' part for internal processing
    rewrite ^/api/(.*)$ /$1 break;

    # Pass the captured path to the named location
    try_files $uri @index;
}

location @index {
    # Use the captured path for proxying
    proxy_pass http://index:3001/$captured_path$is_args$args;
    # ... rest of your configuration
}

[josecelano]

Hey @da2ce7 @WarmBeer this seems to be very problematic. Maybe we should find an alternative way to encode URLs inside another URL. What do you thing?

[josecelano]

Links:

• https://s7ng6n.medium.com/how-to-embed-a-url-in-another-url-19c5ca4a89fe

[josecelano]

Relates to: torrust/torrust-index-gui#409

[josecelano]

I've tried a lot of different configurations for Nginx but none of them have worked.

• https://trac.nginx.org/nginx/ticket/786?cversion=0&cnum_hist=2
• https://stackoverflow.com/questions/28684300/nginx-pass-proxy-subdirectory-without-url-decoding/37584637#37584637
• https://stackoverflow.com/questions/20496963/avoid-nginx-decoding-query-parameters-on-proxy-pass-equivalent-to-allowencodeds

It seems that if you want to parse and modify the URI, Nginx decodes it. Some options like:

location /api/ {
   set $modified_uri $uri;

   if ($request_uri ~ ^/api/(.*)$) {
	      set $modified_uri $1;
	  }

   proxy_pass http://index:3001/$modified_uri;
}

or

location /api {
rewrite  ^  $request_uri;            # get original URI
rewrite  ^/api(/.*) $1 break;        # drop /api, put /app
return 400;                          # if the second rewrite won't match
proxy_pass    http://index:3001$request_uri;
}

seems to work but I get this error:

proxy      | 2024/02/05 12:58:48 [error] 20#20: *1 no resolver defined to resolve index, client: 172.26.0.1, server: index.torrust-demo.com, request: "GET /api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png HTTP/1.1", host: "index.torrust-demo.com"
proxy      | 2024/02/05 12:58:48 [debug] 20#20: *1 finalize http upstream request: 502

[josecelano]

I'm using this configuration to enable debug log level in Nginx:

  proxy:
    image: nginx
    container_name: proxy
    restart: unless-stopped
    networks:
      - frontend_network
      - backend_network
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./storage/proxy/webroot:/var/www/html
      - ./storage/proxy/etc/nginx-conf:/etc/nginx/conf.d
      - ./storage/certbot/etc:/etc/letsencrypt
      - ./storage/certbot/lib:/var/lib/letsencrypt
      - ./storage/dhparam:/etc/ssl/certs
    command: [nginx-debug, '-g', 'daemon off;']
    logging:
      options:
        max-size: "10m"
        max-file: "10"
    depends_on:
      - index-gui
      - index
      - tracker

[josecelano]

I finally found a working configuration:

server
{
	listen 80;
	listen [::]:80;

	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;

	server_name index.torrust-demo.com;

	#error_log /var/log/nginx/error.log debug;

	location /api/v1/proxy
	{
		rewrite ^ $request_uri;
		rewrite ^/api(/.*) $1 break;
		proxy_pass http://index:3001;
	}

	location ^~ /api/
	{
		rewrite ^/api/(.*)$ /$1 break;
		proxy_pass http://index:3001;
	}

	location /
	{
		proxy_pass http://index-gui:3000;
	}

	location ~ /.well-known/acme-challenge
	{
		allow all;
		root /var/www/html;
	}
}

It works, but it has a potential problem. The proxy_pass in the following location does not include the URL query.

location /api/v1/proxy
{
  rewrite ^ $request_uri;
  rewrite ^/api(/.*) $1 break;
  proxy_pass http://index:3001;
}

CUrrently, it's not a problem because URL are like this:

http://index.torrust-demo.com/api/v1/proxy/image/https%3A%2F%2Fraw.githubusercontent.com%2Ftorrust%2Ftorrust-index%2Fdevelop%2Fdocs%2Fmedia%2Ftorrust_logo.png

We do not have any "query" in the image proxy URLs.

[josecelano]

It seems this also works:

server
{
	listen 80;
	listen [::]:80;

	root /var/www/html;
	index index.html index.htm index.nginx-debian.html;

	server_name index.torrust-demo.com;

	#error_log /var/log/nginx/error.log debug;

	location ^~ /api/
	{
		rewrite ^ $request_uri;
		rewrite ^/api(/.*) $1 break;
		proxy_pass http://index:3001;
	}

	location /
	{
		proxy_pass http://index-gui:3000;
	}

	location ~ /.well-known/acme-challenge
	{
		allow all;
		root /var/www/html;
	}
}

mergin the two locations, but it does not work for HTTPs using a named location.

[josecelano]

I'm going to keep it as it's for the time being. It would be nice to try to simplify the configuration.