Memory leak and device crash #4437
Comments
|
Bumping this up. This needs visibility. |
|
@timkgh, hello and apologies for a late response. Unfortunally, we can't reproduce the issue on our test machines. Could you please answer a couple of questions:
Also could you please collect some profiling info to enrich the issue details? Here is how to perform it:
Sorry for a bit complicated request. Thanks. |
|
I believe it's a cache issue. I've been running it with: for a few days now and it's stable, the memory use does not creep up. I don't think the cache limit of 4M from my previous config is being enforced, it appears that the cache grows unbounded.
As for your other question, I don't run lots of filters, just filters from the built-in lists: Query log size: |
|
@timkgh, thanks for clarifications. Could you please perform a few more tests?
Shortly, we suspect the optimistic caching feature causes these leaks, so the above test should confirm or deny the assumption. Also, we'd like to get a bit different profiling data due to your clarifications. So could you please do the same as I described here, but put |
|
Trying Meanwhile, I think you have sufficient info above to try to reproduce the issues in parallel and get the profiling you need. It is also possible that these are leaking too, I have not turned them off and I do have the corresponding features enabled: |
|
Just an observation, it seems that the memory leak starts after filter automatic updates. My configuration 3 days ago, I restarted AGH after updating filter lists and memory usage is low. Today, after the scheduled automatic filter update, memory usage grew significantly. Will continue to observe this behavior. |
|
@SiNONiMiTY you can try setting |
|
@timkgh I will be trying this one. It is good that it goes down in your case. Have been looking at this for the past few days and this is the only consistent behavior I am seeing. I don't even have caching that can be another cause of memory leak. Here are the preliminary results after setting update interval to 1 hour. BEFORE AFTER (UPDATE 1) AFTER (UPDATE 2) |
@EugeneOne1 , it appears there maybe some form of anomaly being reported by users with ARM based arch as well concerning similar memory leak issues. I just wanted to bring this to your attention that it may not only be mipse arch impacted by the occurrance of this anomaly. You can see where I and other Adguardhome on asuswrt-merlin router users have been discussing it here at this link. Here is where I have requested the aid of other asuswrt merlin users to see if we can get some more test. http://www.snbforums.com/threads/adguardhome-vs-diversion-some-basics.78332/post-756017 |
|
@EugeneOne1
I also took the time to email the pprof per request. |
|
Per request for goroutine, here is the pprof for that.
I will also submit it per email as well. |
|
Feel free to request any other info. I will provide what ever I am able to gain access to. I will also be available for test if you have any suggestions. |
|
goroutine 57646 [running]:
goroutine 1 [chan receive, 52 minutes]:
goroutine 7 [syscall, 52 minutes]:
goroutine 8 [chan receive, 52 minutes]:
goroutine 9 [chan receive, 52 minutes]:
goroutine 10 [chan receive, 2 minutes]:
goroutine 12 [syscall, 52 minutes]:
goroutine 13 [chan receive, 52 minutes]:
goroutine 14 [chan receive, 52 minutes]:
goroutine 15 [select, 52 minutes]:
goroutine 16 [chan receive, 52 minutes]:
goroutine 33 [IO wait]:
goroutine 36 [chan receive, 12 minutes]:
goroutine 37 [chan receive, 51 minutes]:
goroutine 66 [sync.Cond.Wait, 52 minutes]:
goroutine 67 [IO wait, 2 minutes]:
goroutine 68 [IO wait, 52 minutes]:
goroutine 69 [sleep, 2 minutes]:
goroutine 70 [IO wait]:
goroutine 71 [IO wait, 1 minutes]:
goroutine 72 [chan receive, 51 minutes]:
goroutine 73 [sleep, 51 minutes]:
goroutine 74 [sleep]:
goroutine 75 [chan receive, 51 minutes]:
goroutine 57661 [IO wait]:
goroutine 46 [select, 51 minutes]:
goroutine 41 [IO wait]:
goroutine 57598 [IO wait]:
goroutine 57647 [runnable]:
goroutine 57610 [IO wait]:
goroutine 16304 [IO wait]:
goroutine 57586 [IO wait]:
|
Definitely. |
|
Could it also have something to do with mmap'ing on these architectures? |
|
FWIW I've been running with |
|
TBH, What I think is needed is the adding the flag --no-mem-optimization to your init.d arguments. |
Can someone please explain what it really does? |
I am under impression that AdGuardHome attempts to optimize its memory usage in some way or shape. Maybe it is not properly doing such, or incompatible. But like you have asked, it would definitely be better to understand what this option is doing. |
|
@jumpsmm7, have you faced a similar problem? There are no extraordinary memory allocations in the provided profiling information actually. The most memory-hungry processes are the filter updates. How many filter rules have you got? |
|
@EugeneOne1 I only had actually 1 list I was using which was OISD full. |
The problem we are actually running into, over a couple of days of normal use, this memory consumption increases and never releases. Basically, if adguardhome updates the filter list, each time it does, it holds on to more and more memory. It does not appear to be letting go of the old memory before performing filter list updates. Maybe there is some sort of memory leak issue that needs to be looked into, particularly in regards to the filter update processing versus when adguardhome purges its current memory holdups. This issue seems to disappear if the cache is disabled with cache_optimistic: false. Definitely something is going on. |
|
@EugeneOne1 Can you please explain what |
In our case we would want to know if adding it may benefit. It appears the OP is not passing this as an argument when launching adguardhome. |
|
been running AGH with the maybe there is something wrong with the garbage collection You can check my previous post for the memory usage without the argument This is the current consumption now, and it is consistent at that range. |
|
@SiNONiMiTY, well that's interesting. @timkgh, @jumpsmm7, could you also please try to run AGH with |
|
@jumpsmm7, AFAIK, it performs some runtime hacks to improve memory usage, but it seems obsolete honestly. |
|
@jumpsmm7 @SiNONiMiTY @EugeneOne1 After 7 hours just doing nothing IT STILL SHOWS HIGH MEMORY on vps linux 1gb ram. And AGH sticks on "DNS server starting up"
|
|
hmmm I added blocklist with fewer URLs(still alot) and after reboot it remains around 35% ram and AGH starts up and it's stable. So I guess it is AGH blocklist url part cause memory issue🤷♂️
|
|
@trinib do me a favor, once you get the memory usage high, try using a different program that also demands alot of memory, then tell me if adguardhome starts releasing memory. From my understanding, this is a behavior of Go itself. GO does not free memory until it is taken back. |
|
@jumpsmm7 It does goes down running memory stress test but it rises right back up after stopping it. explorer_yR1r8LTdxb.mp4 |
|
I would like to add, from a clean install I used backup Why memory gets & stays so high when processing lots of blocklist filters or logs filling up from days/weeks of build up. 🤔🤔🤔 |
|
@trinib how much of the memory consumption is cache memory and how much of it is actually being used? |
|
|
|
Here is mine running on a router that uses busybox.
and the arch
AdGuardHome at the bottom of my htop
|
I do not understand . what this suppose to mean? Did you do something to get lower memory ? did you ran script with all the URLs ? Cause correct me if i am wrong i see 1gb of ram and 829 in use and AGH is 26% ? please tell me you ran script with those results . if so what tweak you did ? |
|
@trinib, thank you for the script! @EugeneOne1, please investigate using the script. |
|
@trinib Here is me using your script. (this is durring and after.) |
|
Adguardhome seems to struggle to know how to commit memory in the presence of this many filters having been added. |
hopefully it can help point out quicker by adding many urls to see where AGH has memory issue |
|
Well it seems I meet with the same problem. |
|
Could it be related to GC in go taking too much memory on small devices when not limited? Had AGH pile up to 1.5G on a 4G VM until I limited it to 384m (using docker) now it runs fine at 350m per average. Go itself is meant to introduce a soft limit via ENV |
|
@timkgh, is this still an issue? Because we could never really reproduce this. |
ainar-g
added
cannot reproduce
and removed
needs investigation
|
The only thing I can say when using an embedded or small device with a small amount of memory, you definitely need a swap and a good regard for how many filters/and or lists you use. If you expect to have alot of filters, then expect to have a swap incase there is the opportunity for an out of memory occurrence. If on a VM, you need to allocate an adequate amount of memory if you expect to be loading alot of filters. All of these types of blocklists are loading into memory. The more filters or block domains, the more caution needs to be taken in regards to memory needed. The memory doesn't fill indefinitely, however it may appear that it does when using memory stores that are inadequate in size for the amount of lists being used. This is not something that can be controlled; however, it relies on the user's good judgement and understanding of the system from which they operate. |
I turned off optimistic caching because of other issues (e.g. Chrome can cache stale DNS responses for 1 min which is annoying) and I haven't seen any memory problems. We can probably close this issue. |
Merge in DNS/adguard-home from 4437-rm-mem-opt to master Updates AdguardTeam#4437. Updates AdguardTeam#2044. Squashed commit of the following: commit d1e5520 Merge: 73a6b49 8bb9546 Author: Eugene Burkov <E.Burkov@AdGuard.COM> Date: Wed Apr 6 19:37:09 2022 +0300 Merge branch 'master' into 4437-rm-mem-opt commit 73a6b49 Author: Eugene Burkov <E.Burkov@AdGuard.COM> Date: Wed Apr 6 18:33:23 2022 +0300 all: imp log of changes commit a62efcd Author: Eugene Burkov <E.Burkov@AdGuard.COM> Date: Wed Apr 6 18:27:42 2022 +0300 home: depr memory opt
Merge in DNS/adguard-home from imp-help to master Updates AdguardTeam#4437. Squashed commit of the following: commit 941338b Author: Eugene Burkov <E.Burkov@AdGuard.COM> Date: Thu Apr 7 13:59:55 2022 +0300 home: imp help output
Issue Details
Expected Behavior
Run for long periods of time without crashes/memory leak.
Actual Behavior
Leaks memory until the router is out of memory and it crashes.
Screenshots
After about 26 days of uptime:
After the screenshot above, I restarted AGH and it shows a reasonable memory footprint:
You can see the memory use on the device drops from 456M to 177M after an AGH restart, but it continues to climb until it eventually exhausts the entire device memory.
Additional Information
Cache settings:
Clearing the stats and logs while AGH is running does not help reclaim memory, only an AGH restart does.
The text was updated successfully, but these errors were encountered: