AGH uses a broken filter list

[Sakura-Luna]

Prerequisites

• I have checked the Wiki and Discussions and found no answer

• I have searched other issues and found no duplicates

• I want to report a bug and not ask a question

Operating system type

Linux, Other (please mention the version in the description)

CPU architecture

AMD64

Installation

Docker

Setup

On one machine

AdGuard Home version

v0.107.20

Description

What did you do?

I updated the latest version and found it difficult to update the (custom) filter list (poor network status), AGH updated the broken list and caused the entire filtering service to malfunction. In fact, in the old version, the update list will be stuck or crashed probabilistically.

Expected result

Successfully updated or only updated list of successfully downloaded

Actual result

Update unresponsive and broken lists

Screenshots (if applicable)

Additional information

Compared with expired lists, damaged lists will cause more serious problems. The integrity of the list should be ensured, such as adding an integrity check mechanism for the list, such as a comment mark at the end of the Goodbye Ads list.

[EugeneOne1]

@Sakura-Luna, hello. We've just pushed the edge build which is expected to deal with the issue, i.e. report errors about broken lists and ignore those. Could you please check if it works? Thanks.

[Sakura-Luna]

@EugeneOne1 I observed it for a few days and found no problems for the time being. But when I actively update the filter list, I encountered a program crash.

[Sakura-Luna]

@EugeneOne1 I removed the part at the end of the list and started AGH, which loaded it directly, causing the problem to recur.

[EugeneOne1]

@Sakura-Luna, what exactly do you mean by "actively update the filter list"? Do you mean pushing the updating button several times within a short period of time? Could you please collect a verbose log capturing the crash for us? You may send it to devteam@adguard.com, this would really help to troubleshoot further, thanks.

Also, which file have you edited manually (removed the part at the end)? Was it within the AdGuard Home's working directory under <WD>/data/filters or the actual file used to create custom list?

[Sakura-Luna]

@EugeneOne1 Not used many times, just press the update button once, AGH has a certain chance to crash after a few minutes, due to network problems, updating to a large list is prone to occur. I have checked the run.log, and there are only logs that are automatically restarted after a crash.

I deleted the end of a list file under the filter path, and loaded it directly after starting AGH. If a "||www" is left at the end of the line, a batch of websites cannot be accessed. This is the problem I describe. AGH requires a feature to support completeness or validity checking of lists.

[EugeneOne1]

@Sakura-Luna, actually the ||www is a valid rule matching any domain having a label which starts with www. I don't really think there is a way to distinguish such "broken" lists from valid ones. I mean AdGuard Home assumes the local rule lists are always valid since it is not intended to work within the setup suffering from recurring data corruption.

However, I believe we're able to validate the data transmission status. Particularly, AdGuard Home now requires each rule in the list to be trailed with \n, so that other rules (e.g. not fully transmitted) are ignored.

As for the crash, unfortunatelly, we cannot reproduce it, probably because it seems to require some specific network conditions. That's why we'd really like to look at your verbose log. Sorry, I didn't notice that you're using docker. The log_file there should point inside the working directory to expose it outside the container. On each restart AdGuard Home will just append the new lines at the end.

[Sakura-Luna]

@EugeneOne1 I know, the "||www" is just an example, what I actually encountered before was "||4", which caused me to spend a few hours before I found out that the list was wrong. I don’t know whether the current AGH can guarantee that every update is a valid list, so I proposed the need for integrity verification. Like Goodbye Ads, there are rules number at the beginning and comments at the end, so it is not difficult, of course, maybe not all of them can be like this, it can be a feature.

[EugeneOne1]

@Sakura-Luna, I think it makes sense to open a new feature request for that and focus on the crash here.

[Sakura-Luna]

@EugeneOne1 I rechecked the logs and think the crash is not an AGH issue, it might be that the reload list is taking too much CPU and being restarted by Docker, unfortunately I'm not likely to verify it. AGH's regular update is rarely a problem.

[EugeneOne1]

@Sakura-Luna, well, the original issue seems solved, so I'll close the issue if you don't mind. Could you please fill a separate one for the feature request, this is to keep topics as atomic as possible and to simplify the search.