replacer: Implement file.* global replacements

[francislavoie]

Closes #5374

[mholt]

Thanks Francis -- let's document this as an experimental feature. I'd also like to see some confirmation from someone that this is in fact useful for them and what they need. And yes, I agree we can lean on the kernel to cache files in hot paths and that will probably get us 99% of the way there in terms of peak performance. 👍

Thanks again!

[francislavoie]

The Docker secrets argument is compelling enough for me. It's a legitimate trend.

[mholt]

Ok -- if we don't hear from anyone in a couple days, I'd say go ahead and merge it. Thank you!

[polarathene]

The Docker secrets argument is compelling enough for me. It's a legitimate trend.

But often a misunderstood one since it's not at parity to the swarm feature, it's just a volume mount under the hood without Docker Swarm.

However I think it might work properly with Podman and Kubernetes equivalents. Even with plain Docker, so long as the user is aware none of the Docker Swarm features apply, they can manually adjust ownership/permissions for the file before mounting it if needed.

[iliana]

Wanted to reach out and mention that I applied this PR to a v2.6.4 build and it works well for me!

One confusing thing is that this style of matcher didn't work for me:

@valid {
  method POST
  path {file./path/to/secret}
}

but this works:

@valid {
  method POST
  vars {path} {file./path/to/secret}
}

but I'm unsure whether that's expected.

[francislavoie]

I don't understand what you're trying to do with that matcher. Please elaborate with more specifics. What does the request look like, and what's in the file?

[iliana]

What I am doing is configuring a POST webhook with a secret parameter in the URL, e.g. /webhook/WpFohBGulVjWHE2v3MP1tHB9ztcPAiD9PBNWF4HtvNQ. Because this config is built by a public repo, and the third-party system that is going to hit this webhook doesn't provide any way of authenticating requests, I am adding the long secret hash to help prevent guessing the URL.

The request I am matching is (using the above example) POST https://example.com/webhook/WpFohBGulVjWHE2v3MP1tHB9ztcPAiD9PBNWF4HtvNQ, and the section for this domain in my working Caddyfile looks like:

example.com {
        bind

        log {
                output file /var/log/caddy/access-example.com.log
        }

        encode zstd gzip
        tls {
                on_demand
        }

        route {
                @webhook {
                        method POST
                        vars {path} {file./run/agenix/webhook-path}
                }

                reverse_proxy @webhook unix//run/fcgiwrap.sock {
                        transport fastcgi {
                                env SCRIPT_FILENAME /nix/store/y2aawd2bjw9dzp1g4p5pjawn8786n7kn-pkgf/bin/pkgf
                                env WEBHOOK_SECRET {file./run/agenix/webhook-secret}
                        }
                }

                respond /yo "yo"

                error 404
        }
}

/run/agenix/webhook-path contains exactly /webhook/WpFohBGulVjWHE2v3MP1tHB9ztcPAiD9PBNWF4HtvNQ (no whitespace). With this configuration, POST requests to that path get proxied to fcgiwrap and Caddy responds with 200.

I tried this matcher first, however:

...
                @webhook {
                        method POST
                        path {file./run/agenix/webhook-path}
                }
...

and given the same request, Caddy responds with 404. This is unexpected behavior to me, but I am unsure if the path matcher would otherwise correctly handle this for some other replacer as I've never tried it.

[znkr]

I'd also like to see some confirmation from someone that this is in fact useful for them and what they need

I found this PR while trying to understand how to best use docker, podman, or kubernetes secrets in a Caddyfile. That turned out to be quite the pain. Having a way to read secrets from a file directly would definitely help.

Podman and Kubernetes can add a secret into an environment variable, but that's not recommended as environment variables tend to leak.

[mholt]

Francis and I talked about this in Slack and while we agree there are use cases for this, we want to be sure we're careful since this currently is not jailed to the site root, and we're not always confident that placeholders come from trusted sources (though they should). For example, templates can invoke placeholders, and we're not sure that templates are all always trusted (i.e. if a site allows users to customize templates).

I might defer this a little later until we can figure out a safe mechanism for this.

[francislavoie]

Took me a while to think about a good way to make it safe. I think a "good enough" approach is to provide a way for specific modules to opt-out of the file placeholders.

The simplest way I could think of doing it is by prefixing the providers with one that rejects file placeholders before it reaches the existing one. I wanted to filter out the fileReplacements function, but apparently Go doesn't allow comparing functions (which I'm totally used to doing in languages like JS and PHP, so I found that surprising).

This still needs some more tests before merging (edit: added tests), but I think it might be an improvement in terms of safety. I'm not sure if there's other modules that should also opt-out of the placeholder, we'll need to review that.

[mohammed90]

apparently Go doesn't allow comparing functions

I don't know what's the design you had in mind, but maybe you can work around needing to compare functions by interface-asserting into single-method interfaces and branch that way?

[mholt]

One minor change since I'm not 100% committed to this feature/approach, but we can give it a try. :) Thanks

[mholt]

Mark this as Experimental. I'm still nervous 😅

[francislavoie]

Sure. But I think it's less-so this function needing to be experimental, cause this is actually the opt-out.

[mholt]

I see what you mean. I mostly am just worried about compatibility promises in the very off-chance this is used externally 😆

[lorenzleutgeb]

Looking forward to use this feature! Thanks! I tried to reconstruct a Caddyfile example from this to understand how I would use such a replacement in my own configuration, but it's a little too dense for a user to see right away. I hope you'll find the time to add an example to the docs somewhere, that would be great!

[francislavoie]

It's just this: {file./path/to/some/file}. Put that in your config wherever you need to read a config value from a file.