request help: Does response-rewrite plugin support partial or regular substitution

[jagerzhang]

Issue description

比如Nginx自带的响应内容替换库可以替换局部内容：http://nginx.org/en/docs/http/ngx_http_sub_module.html

sub_filter '<a href="http://127.0.0.1:8080/'  '<a href="https://$host/';

比如还有个支持正则替换的库（印象中Openresty已默认支持）：ngx_http_substitutions_filter_module，可以通过正则表达式来替换内容：

subs_filter_types text/html text/css text/xml;
subs_filter st(\d*).example.com $1.example.com ir;
subs_filter a.example.com s.example.com;
subs_filter http://$host https://$host;

但是我看APISIX的 response-rewrite 插件文档，好像只能支持完整替换，相当于直接将所有响应都换成插件设置的返回，并不支持部分内容替换:

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/test/index.html",
    "plugins": {
        "response-rewrite": {
            "body": "{\"code\":\"ok\",\"message\":\"new json body\"}",
            "headers": {
                "X-Server-id": 3,
                "X-Server-status": "on",
                "X-Server-balancer_addr": "$balancer_ip:$balancer_port"
            },
            "vars":[
                [ "status","==","200" ]
            ]
        }
    },
    "upstream": {
        "type": "roundrobin",
        "nodes": {
            "127.0.0.1:80": 1
        }
    }
}'

想问下这个插件是否支持部分替换或正则替换？如果不支持，APISIX有没有其他解决方案呢？

Environment

• apisix version (cmd: apisix version): 2.10.0

[tokers]

@jagerzhang Yes, it cannot be supported by this plugin, but if you desire, just use the nginx or openresty way on APISIX, they're also feasible.

[jagerzhang]

@tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~

[tokers]

@tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~

Please check out Customize Nginx Configuration.

[tzssangglass]

Does Customize Nginx Configuration work globally? I think it would be better to make the response-rewrite plugin support regular substitution.

[tokers]

Does Customize Nginx Configuration work globally?

Yes, the range cross the route boundaries.

I think it would be better to make the response-rewrite plugin support regular substitution.

That's might be a good idea, let's listen to more sounds.

[jagerzhang]

@tzssangglass @tokers 非常期待能支持一下这个Openresty默认支持的特性。

[jagerzhang]

@tokers 在APISIX配置中使用Nginx或openresty的配置，求一个指引文档，多谢~

Please check out Customize Nginx Configuration.

粗略看了下，这样用需要改动配置文件，相当于每次改动都需要reload或重启，动作比较重。还是期待能在插件里面动态支持~

[kwanhur]

Above the conversation, I got these information:

Target: support specify content-type to filter, substitute response body by matching specific regex(fixed string as well) with replacement.

regex mode supports to replace all or once, and case-sensitive or not

Solution:
Properties table

name	description	type	default
filter_type	which response content-type matched	array	"text/plain"
filter_pattern	pattern to match response body, could be regex	string
filter_replace	substitution content	string
filter_options	options for regex, detail see here	string
filter_kind	regex or completely substitute mode, range [1,2] 1-completely 2-regex	integer	1
filter_scope	regex substitute once or global, range [1,2], 1-once 2-global	integer	2

Main steps:

• step 1: check conf.filter_kind, equals to 1 then fallback to completely substitute, otherwise goto regex substitute(both in header_filter and body_filter).
• step 2: at header_filter phase, check match conf.filter_type or not, if not then return, otherwise then mark ctx.response_rewrite_matched true.
• step 3: at body_filter phase, conf.filter_pattern empty or if not ctx.response_rewrite_matched then return, otherwise check conf.filter_scope: scope 1 use ngx.re.sub, scope 2 use ngx.re.gsub.
• step 4. substitution:

local body = ngx.arg[1]
ngx.arg[1] = sub(conf.filter_pattern, conf.filter_replace, conf.filter_options)

• step 5: mark eof

ngx.arg[2] = true

Any advices welcome :-).

[spacewander]

Well, personally I prefer to use an array of filters:

"filters": [{
  "expr": ... # if given, run the filter when it's evaluated to true. Can be used to support "content-type" and more.
  "kind": "regex", # a string enum would be better
  "scope": ...,
  "pattern": ...,
  ...
}]

equals to 1 then fallback to completely substitute, otherwise goto regex substitute(both in header_filter and body_filter).

"completely substitute" can be named as sub_filter substitute? As Nginx's sub_fitler supports variable, which isn't a plain substitute.

local body = ngx.arg[1]
ngx.arg[1] = sub(conf.filter_pattern, conf.filter_replace, conf.filter_options)

I am afraid we can't do regex in the stream. This requires the regex engine to have stream mode, for example, see google/re2#127.
In fact, Nginx's sub_filter will buffer the data before substitution.

[kwanhur]

Well, personally I prefer to use an array of filters:

On this way, filter would take effect orderly by itself, right?

I am afraid we can't do regex in the stream.

From directives, Nginx's sub_filter only support in the http not stream.
So, I think support in the http by first, and support in the stream by second in the future.

[spacewander]

On this way, filter would take effect orderly by itself, right?

Yes.

I am afraid we can't do regex in the stream.

Actually, I mean the streaming processing (not the L4 proxy).

[kwanhur]

Solution updated.

Properties table adds one filters

name	description	type	default
filters	filter expressions, include key attributes how to filter	array	[] empty

filter key attributes

name	description	type	default
expr	match expression, adapters to lua-resty-expr , only on header	map	{}
kind	regex or subfilter mode	string	subfilter
options	regex options, detail see here	string	""
scope	regex substitute once or global	string	"once"
replace	regex substitution content	string	""
pattern	match pattern on response body, regex or fixed string both ok	string	""

Main steps:

step 1: at header_filter body_filter phase, if filters empty, then back to original body-replaced logic(backward compatibly); else then goto step 2.

step 2: at header_filter phase, foreach filters and eval expr ok or not.

• ctx.filter_matches to store every filter expr result[true|false]
• ctx.filter_matched to store total filters result(use to check response body need to filter or not)

step 3: at body_filter phase,

1. if ctx.filter_matched true and ngx.arg[2] false, then buffer the response body into ctx.filter_body.

2. if ctx.filter_matched true and ngx.arg[2] true, then foreach filters to filter, pseudo-code:

   if ctx.filter_matched and ngx.arg[2] then
     for i, filter in ipairs(conf.filters) do
       if ctx.filter_matched[i] then
         if filter.scope == "once" then
           ctx.body = ngx.re.sub(ctx.body,filter.pattern, filter.replace, filter.options)
         else
           ctx.body = ngx.re.gsub(ctx.body,filter.pattern, filter.replace, filter.options)
         end
       end
     end
     
     ngx.arg[1] = ctx.body
   end

   ​

Q1: it seems filter.kind is redundant, since step 1.

[kwanhur]

@spacewander @tokers Please take a look on the second solution, I'm not sure if it's on the right way.

[tokers]

@kwanhur Could you also give some data examples according to the second solution?

[spacewander]

step 3: at body_filter phase,

We can use

apisix/apisix/core/response.lua

Line 177 in 9e19885

function _M.hold_body_chunk(ctx, hold_the_copy)

to buffer the body

[kwanhur]

After have a look at existed codes, update solution v3 and example:

Solution

Properties table adds one filters

name	description	type	default
filters	filter expressions, include key attributes how to filter	array	[] empty

filter key attributes

name	description	type	default
expr	match expression, adapters to lua-resty-expr , only on header reuse vars	map	{}
kind	regex or subfilter mode	string	subfilter
scope	regex substitute once or global	string	"once"
patternregex	match pattern on response body, regex or fixed string both ok	string	""
replace	regex substitution content	string	""
options	regex options, detail see here	string	""

Main steps:

step 1: at header_filter phase, reuse vars to match the response headers.

step 2: at body_filter phase if ctx.response_rewrite_matched false then return, otherwise

• if conf.filters empty, backward to original body-replaced logic(backward compatibly).
• if conf.filters not empty, buffer response body with core.response.hold_body_chunk(ctx), then foreach filters to filter, pseudo-code:

  local body = core.response.hold_body_chunk(ctx)
    for i, filter in ipairs(conf.filters) do
        if filter.scope == "once" then
          body = ngx.re.sub(body, filter.pattern, filter.replace, filter.options)
        else
          body = ngx.re.gsub(body, filter.pattern, filter.replace, filter.options)
        end
    end
  
  ngx.arg[1] = body

Example

• with response code 200, fixed string hello replace with world

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/test/index.html",
    "plugins": {
        "response-rewrite": {
            "filters": [
                {"scope": "global", "regex": "hello", "replace": "world", "options": "jio"}
            ]
            "vars":[
                [ "status","==",200 ]
            ]
        }
    }
}'

• with response content-type text/plain, regex \d replace with *

curl http://127.0.0.1:9080/apisix/admin/routes/1  -H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "methods": ["GET"],
    "uri": "/test/index.html",
    "plugins": {
        "response-rewrite": {
            "filters": [
                {"scope": "global", "regex": "\d", "replace": "*", "options": "jio"}
            ]
            "vars":[
                [ "content-type","==","text/plain"]
            ]
        }
    }
}'

[spacewander]

LGTM, except for a small problem:
We need to check the result of hold_body_chunk:

apisix/apisix/core/response.lua

Lines 168 to 173 in 6cd7b8d

	-- Usage:
	-- function _M.body_filter(conf, ctx)
	-- local final_body = core.response.hold_body_chunk(ctx)
	-- if not final_body then
	-- return
	-- end

[ada012]

@spacewander @kwanhur

Hi, I would also like to use nginx sub_filter in apisix, but it still doesn't work when I use it in apache/apisix:2.99.0-debian

Plugin config：

{
  "disable": false,
  "filters": [
    {
      "regex": "head",
      "replace": "1212",
      "scope": "global"
    }
  ],
  "headers": {
    "set": {
      "x-powered-by": "",
      "x-server-balancer_addr": "$balancer_ip:$balancer_port"
    }
  },
  "vars": [
    [
      "status",
      "==",
      200
    ],
    [
      "content-type",
      "==",
      "text/html; charset=utf-8"
    ]
  ]
}

[tzssangglass]

Hi, I would also like to use nginx sub_filter in apisix, but it still doesn't work when I use it in apache/apisix:2.99.0-debian

Plugin config：

Please open a new issue and provide the full reproduction steps