Runtime placeholders

[vnxme]

Following our discussion in slack, these changes relax Caddyfile parsing and ensure runtime placeholders support across most handler and matcher options. Besides, I've updated README to document how runtime placeholders are treated in the code.

Examples of syntax which will work with these changes:

• Listen address of a layer4 server filled with an environment variable evaluated at provision

  {env.VAR} {
  	route {
  		proxy localhost:5000
  	}
  }
  

• Dial address of proxy upstream filled with an environment variable evaluated at provision

  :5000 {
  	route {
  		proxy {env.VAR}
  		# syntax below used to work before these changes thanks to replacement in dialPeers
  		# proxy {env.VAR}:5001
  	}
  }
  

• Address or range of remote_ip and local_ip matchers filled with environment variables evaluated at provision

  :5001 {
  	@ips {
  		local_ip {env.VAR_1}
  		remote_ip {env.VAR_2}
  	}
  	route @ips {
  		proxy localhost:5002
  	}
  }
  

• Credentials of socks5 handler filled with environment variables evaluated at provision

  :5002 {
  	@s5 socks5
  	route @s5 {
  		socks5 {
  			credentials {env.USER_1} {env.PASS_1} {env.USER_2} {env.PASS_2}
  		}
  	}
  }
  

• Cookie hash of rdp matcher filled with an environment variable evaluated at match

  :5003 {
  	@rdp rdp {
  		cookie_hash {env.VAR}
  	}
  	route @rdp {
  		proxy windows.machine.local:3389
  	}
  }
  

• Alpn of tls matcher filled with an environment variable evaluated at match

  :5004 {
  	# it's not really useful with alpn, but it would great if we could do the same with sni
  	@test tls alpn {env.VAR}
  	route @test {
  		proxy localhost:443
  	}
  }
  

[mholt]

Thanks for contributing this. I haven't scrutinized each line but the approach looks good. If we're ready to merge it I will go ahead and do so 👍

[mholt]

I wonder if this function should move to caddy instead of caddyhttp (since it can be used for more than just HTTP).

[mholt]

I missed the PR back at the caddy repo but I still wonder about this. What's the right package for this, do we think?

[mholt]

Looks like a rebase is needed after the lint fixes

[vnxme]

Seems like the build-test-check doesn't respect go.mod: it's trying to build caddy-l4 with caddy@v2.8.5 (unreleased yet, meaning the latest commit I believe) instead of caddy@v2.8.4. Or is it my fault somewhere?

Shall we fix it here?

[mholt]

Huh, that's strange. Weren't the tests succeeding at first?

[vnxme]

Huh, that's strange. Weren't the tests succeeding at first?

The tests had been successful before you merged caddyserver/caddy@59cbb2c moving PrivateRangesCIDR() from caddyhttp to internal. And the builds are made with xcaddy build master, that's the problem here.

[mholt]

Ohh. Yes, you're right. Sorry I see now that you linked to the cause.

I think that's intentional because we want to be sure that this module is compatible with up-and-coming releases of Caddy.

[vnxme]

Then I believe we should wait until Caddy v2.8.5 is released, then I will update go.mod and this PR as well.

[mohammed90]

moving PrivateRangesCIDR() from caddyhttp to internal.

Oops, it's my fault for not realizing it's used here and breaking it by suggesting moving the func 😶

Then I believe we should wait until Caddy v2.8.5 is released, then I will update go.mod and this PR as well.

I doubt this will solve anything. Should we duplicate the func or move it somewhere, as Matt suggested?

[vnxme]

I doubt this will solve anything. Should we duplicate the func or move it somewhere, as Matt suggested?

As of now caddy-l4 builds in an IDE with go.mod requiring caddy v.2.8.4, but won't build here with xcaddy build master.

If we release caddy v2.8.5 and adjust go.mod, caddy-l4 will build both in an IDE and here on github.

On the other hand, if we adjust this PR now, there will be no failures on github, but caddy-l4 won't build in an IDE.

Alternatively, we could adjust the build-test-check to respect go.mod, but @mholt mentioned that master usage is intentional.

Finally, I don't think it's a good idea to duplicate the code.

[mohammed90]

If we release caddy v2.8.5 and adjust go.mod

Adjust it how? There's no way to edit go.mod that'd fix it. If you change go.mod here to Caddy v2.8.5, it'll be the same issue as with master.

I made a mistake suggesting moving the function to an internal package, which screwed up this PR. I can move it back to caddyhttp package. It'll fix everything and make everyone's life easy. Sorry for causing the trouble.

[vnxme]

@mohammed90 I mean I can adjust this PR in 2 ways: 1) update go.mod to require Caddy v2.8.5; 2) update the code to call internal.PrivateRangesCIDR() instead of caddyhttp.PrivateRangesCIDR(). This way the build-test-check will pass, and the code will build in an IDE as well.

I do think it was a good idea of yours to move the function. I didn’t know the build-test-check didn’t respect go.mod, and it was intentional.

[mohammed90]

update the code to call internal.PrivateRangesCIDR()

Code in internal package cannot be imported & called by external packges (see: https://pkg.go.dev/cmd/go#hdr-Internal_Directories).

[vnxme]

Oops, now I understand your point. Is there any proper importable place in Caddy mainline where we could move the function other than back to caddyhttp? If no, then I agree, we can use caddyhttp as an easy fix.

[mohammed90]

Is there any proper importable place in Caddy mainline where we could move the function other than back to caddyhttp?

Given the burden of changing versions caused if put anywhere other than caddyhttp, I think it's best to move it back to caddyhttp.

[mholt]

Tests are passing now! 🎉 But I am still wondering on the package home for that CIDR prefix function. It's not specific to HTTP, so I wonder if it should go in the caddy core package

[vnxme]

If we move that CIDR prefix function to the caddy core package, we have to a) release Caddy v2.8.5; b) update the code in this PR as well as require v2.8.5 instead of v2.8.4 in go.mod. Otherwise caddy-l4 won't build either on github or in an IDE.

[mholt]

True... well... I guess it's not a big deal, unless the caddy package does end up needing/using that someday. Maybe that's a change we wait until 2.9 for.