Feature Request: Sanitize/reformat/remap content retrieved via reverse_proxy

[kevinelliott]

It would be helpful to be able to sanitize, reformat, and remap content in response bodies retrieved via reverse_proxy before handing back to the requestor. For example, it could allow you to convert an XML response body to JSON, or to remap attributes to a more common format before it is returned.

[francislavoie]

You can already do this with the handle_response handler chain in the reverse_proxy handler: https://caddyserver.com/docs/json/apps/http/servers/routes/handle/reverse_proxy/handle_response/ (Caddyfile support is being worked on in #3712)

This type of rewriting of the response is more a job for a Caddy plugin to handle. https://caddyserver.com/docs/extending-caddy

[mholt]

I can't remember if those handlers have access to the response body...

[francislavoie]

@mholt the description says:

The response body from the backend will not be written to the client; it is up to the handler to finish handling the response.

So, I think so?

Edit: Hmm, nope. It does res.Body.Close() then runs the handlers.

caddy/modules/caddyhttp/reverseproxy/reverseproxy.go

Line 590 in 7c28ecb

res.Body.Close()

So @mholt 🤷‍♂️ what do you want to do with this?

[kevinelliott]

Help me out here since I'm new to caddy and a little naive, do I need to write a rust-based module to extend caddy in order to implement the handler or is there something in the config that lets me write some code that allows me to translate the XML to JSON already?

[francislavoie]

Caddy's a Go project, not Rust.

There's nothing currently in Caddy to make this possible, and frankly it's out of scope of the core of the project, and should be implemented as a plugin if you need it.

But as @mholt pointed out, I don't think it's possible to grab the reverse_proxy response body in handle_response as I thought it was. But it should be possible to write a module similar to encode which handles the response on the way back up the handler middleware chain.

[kevinelliott]

Oops, sorry, handling a lot of languages these days and confused Rust with Go, sorry.

OK, so if I go lookup the encode module source (do you have a direct link handy?) I just need to implement my effort there, and return to the chain. How do I harness my own plugin? Do I link it separately, or must I clone the repo and make my additions inline and rebuild?

[francislavoie]

Sure, it's in here: https://github.com/caddyserver/caddy/tree/master/modules/caddyhttp/encode

This document explains everything you should need to know to write plugins: https://caddyserver.com/docs/extending-caddy

A plugin would be a separate repo, and it's easiest to compile it in with the xcaddy tool which automates the process.

[mholt]

When I implemented that, I only thought about it for a few minutes and was like, "Well I don't know what to do with the body... I guess we'll just close it," and came to the same conclusion you did, about a middleware handler wrapping the response so that it could intercept it itself. I would like a better way to handle responses from proxy upstreams, but didn't figure anything out in the few minutes I spent on it.