implement "goto" and "ban" actions

[dalf]

What

It would interesting to have:

• a main section with the rules as described now
• other sections, each one with a label
• implement a "goto" action: the query would be process by the rules in the targeted section.
• A "ban" action . The parameters would a selector list, and a time.

To avoid infinite loop, "goto" can only go below in rules.json, not above.

Use case

For now, it is possible to deny access when there are too much queries per seconds.
But if there is a continuous flood, it is not possible to something more.
With a "goto" it would be possible to:

• deny access to the web page if there a 4 requests per seconds (as it is possible now).
• "ban" the access if there are 10 "deny" access in 10 minutes.

Or:

• deny access to the web page if there a 4 requests per seconds (as it is possible now).
• deny access to the web page for the JSON/RSS/CSV format
• if any of the two rules as activated more than 10 time in 10 minutes, then ban.

Up to the person writing rules.json to make something coherent about the selectors.

[return42]

May be I misunderstood you, but I fear that the rules.json becomes a mess.

About "deny & ban" what I think: To get the bots under control you should always give them something to eat.

If you ban a bot, the bots admin will detect this fast and adjust the bot. That's why I think, it is better to handle queries from bots unpredictable and to make it harder for bot admins to get over the wall.

That is why I vote for a redirection feature. E.g.: In case to protect searx, for bots I would like to response (slow) a dummy/empty searx result page .. without calling any search-engine.

I guess this will be harder for bot admins to detect and solve.

I am complete new to this thematics, may be I am talking bullshit. If so, please correct me / thanks!

[asciimoo]

That's not a bad idea, we can send them a dummy result page =)

[return42]

@dalf if it is not already known to you or others who come around, I want to clear that ..

deny access to the web page for the JSON/RSS/CSV format

can be configured by filter: "Param:format=(csv|json|rss)". E.g:

      {
	"name":"rss/json limit",
	"interval":60,
	"limit":15,
	"stop":true,
	"filters":[
          "Param:format=(csv|json|rss)"
	],
	"actions":[
          {
            "name":"block",
            "params":{
              "message":"rate limit exceeded"
            }
          }
	]
      },

[dalf]

Yes I'm aware about the csv/json/xml filter rule.

Currently filtron can block users who:

• sends too much requests in a short time
• sends requests with the "wrongs" parameters

In the first case, the user can retry later, after the colddown, being block again, etc...

The purpose of this PR is to add a limit to the number of time a user can be blocked.

So after some (okay, send 429) cycles, I suggest to use a different set of filtron rules (using the goto action, most probably there are others way to implement this feature ; if this feature makes sense).

So if a bot sends requests until it gets blocks, wait, retry, it ends up to being block for a longer time or ban.

Ban action can be replaced by the backend, url, or file actions (just some ideas) :

• backend action: set the backend server to use. So a fake searx instance can server the request, or a static web server (note: currently filtron is designed to use only one backend provided on the command line).
• url action: change the url sent to the backend server.
• file action: serve the request using a static file from the file system (or even the filtron.json)