[BUG] Subscribing to channel with many videos fails (youtube-dl out of free memory)

[notDavid]

Describe the bug
Hi, so for example when i subscribe to this playlist and set it to download the last 6 months, it will fail and my host freezes/crashes after running for about 45 minutes i think;

I found that what happens is, the youtube-dl process keeps consuming more and more memory and eventually at about 1GB the host runs out of free memory (and i think it terminates the process.)

I found this somewhat related issue, and was wondering if perhaps others have stumbled upon this too and something could be implemented as a workaround? :-)

Environment
Installed version: v4.3 - You are up to date.
Installation type: docker
Docker tag: latest
Commit hash: e726e99
Build date: 2022-06-27

Additional context
Log:

2022-06-29T13:07:08.748Z DEBUG: Inserted doc into subscriptions
2022-06-29T13:07:16.148Z VERBOSE: Subscription: getting videos for subscription Hardware Unboxed with args: --dump-json,-o,"users/david/subscriptions/channels/Hardware Unboxed/%(upload_date)s - %(uploader)s - %(title)s - [%(id)s].%(ext)s",-ciw,--write-info-json,--print-json,-f,bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4,--write-auto-subs,--download-archive,,--dateafter,now-6months,--write-thumbnail,--no-clean-infojson
2022-06-29T13:31:37.039Z VERBOSE: Subscription: finished check for Hardware Unboxed
2022-06-29T13:31:37.042Z ERROR: Command was killed with SIGKILL (Forced termination): node_modules/youtube-dl/bin/youtube-dl -i --dump-json -o "users/david/subscriptions/channels/Hardware Unboxed/%(upload_date)s - %(uploader)s - %(title)s - [%(id)s].%(ext)s" -ciw --write-info-json --print-json -f bestvideo[ext=mp4]+bestaudio[ext=m4a]/mp4 --write-auto-subs --download-archive  --dateafter now-6months --write-thumbnail --no-clean-infojson https://www.youtube.com/c/Hardwareunboxednow
2022-06-29T13:31:37.043Z ERROR: Subscription check failed!

[notDavid]

So when i try to subscribe and set Download videos uploaded in the last to 1 week it also fails. So whatever i do, i can't subscribe to this playlist it seems...

Is this an issue in my setup/config or a bug? Perhaps someone can try if they can reproduce... :-)

[adripo]

I have the same issue, even setting a small period. It seems that it parses all videos first regardless of the period.
I have seen it go up saturating RAM and then crash.

[samssausages]

I also have this problem, locks up when subbing to large channels. Tried it on two separate Unraid servers with same results.

[XtremeOwnageDotCom]

I managed to grab a stack trace for this issue. Essentially a fresh installation. Added a single channel subscription... and...

<--- Last few GCs --->
[37:0x4fd7800]   450983 ms: Mark-sweep 4046.7 (4136.7) -> 4044.1 (4143.4) MB, 694.5 / 0.0 ms  (average mu = 0.259, current mu = 0.091) allocation failure scavenge might not succeed
[37:0x4fd7800]   452658 ms: Mark-sweep 4052.1 (4143.4) -> 4049.8 (4164.2) MB, 1641.1 / 0.0 ms  (average mu = 0.118, current mu = 0.020) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0xb09980 node::Abort() [node /app/app.js]
 2: 0xa1c235 node::FatalError(char const*, char const*) [node /app/app.js]
 3: 0xcf784e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node /app/app.js]
 4: 0xcf7bc7 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node /app/app.js]
 5: 0xeaf465  [node /app/app.js]
 6: 0xebf12d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node /app/app.js]
 7: 0xec1e2e v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node /app/app.js]
 8: 0xe830a2 v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [node /app/app.js]
 9: 0xe7b6b4 v8::internal::FactoryBase<v8::internal::Factory>::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [node /app/app.js]
10: 0xe7d3c0 v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [node /app/app.js]
11: 0xe851fd v8::internal::Factory::NewStringFromUtf8(v8::base::Vector<char const> const&, v8::internal::AllocationType) [node /app/app.js]
12: 0xd06415 v8::String::NewFromUtf8(v8::Isolate*, char const*, v8::NewStringType, int) [node /app/app.js]
13: 0xbf5139  [node /app/app.js]
14: 0xae3eb5  [node /app/app.js]
15: 0x15856cc  [node /app/app.js]

[jonathanweires]

I've got the same problem when I try to download playlists with several videos.

Environment:
Raspberry Pi 4 Model B with 4GB RAM
Ubuntu Server 22.04 LTS
YoutubeDL-Material v4.3.2 Docker Image with Mongo DB 4.4.18

The environment was set up yesterday (including the Pi). I managed to download single videos but every time I started to download playlists, the Pi got unresponsive after some time.

Today I re-pulled the containers and cleared the directories. But once I start to download playlists again, problems start again.

I just ran sudo docker stats and saw that the memory usage of YoutubeDL-Material increased significantly until my Pi got unresponsive:

In the Docker container log no errors can be seen in this regard.

[samssausages]

@jonathanweires Can you confirm if this is only with channels that have over 100 videos?

I had this issue with all the YoutubeDL versions, and seems like it's only with channels over 100 videos. (Last I tested was about 6 months ago, haven't checked recently)

So I changed my workflow with large channels... it's a bit of a workaround, but it works every time at least. I manually grab all the links for all the videos and paste them into a different downloader. Then, for new videos I set it up to automatically grab them.

1. I use JDownloader2 and a Chrome extension called "LinkClump"
2. I go to the Youtube Channel, use Page Down to scroll all the way to the bottom. Then use the LinkClump extension to grab all the URL's.
3. Then I paste the URL's into JDownloader2 and tell it to download

Bit of a pain, but at least I only have to do that when I'm adding a new, big, channel.

[SickProdigy]

Hello having a similar issue currently. Seem when I subscribe to a large uploader it will crash. Here are my crash logs from docker.

2023-12-14T22:09:57.546595520Z [entrypoint] setup permission, this may take a while
2023-12-14T22:11:10.660787831Z 
2023-12-14T22:11:10.660804083Z > backend@1.0.0 start
2023-12-14T22:11:10.660807508Z > pm2-runtime --raw pm2.config.js
2023-12-14T22:11:10.660810366Z 
2023-12-14T22:11:11.004064864Z 2023-12-14T22:11:10: PM2 log: Launching in no daemon mode
2023-12-14T22:11:11.040191853Z 2023-12-14T22:11:11: PM2 log: [Watch] Start watching YoutubeDL-Material
2023-12-14T22:11:11.040804275Z 2023-12-14T22:11:11: PM2 log: App [YoutubeDL-Material:0] starting in -fork mode-
2023-12-14T22:11:11.047528180Z 2023-12-14T22:11:11: PM2 log: App [YoutubeDL-Material:0] online
2023-12-14T22:11:12.106164901Z 2023-12-14T22:11:12.104Z INFO: Config items set using ENV variables.
2023-12-14T22:11:12.110751420Z (node:36) DeprecationWarning: uuidv4() is deprecated. Use v4() from the uuid module instead.
2023-12-14T22:11:12.110765132Z (Use `node --trace-deprecation ...` to show where the warning was created)
2023-12-14T22:11:12.251774040Z 2023-12-14T22:11:12.249Z INFO: Restarting server...
2023-12-14T22:11:12.262291130Z 2023-12-14T22:11:12: PM2 log: App [YoutubeDL-Material:0] exited with code [1] via signal [SIGINT]
2023-12-14T22:11:12.265307864Z 2023-12-14T22:11:12: PM2 log: App [YoutubeDL-Material:0] starting in -fork mode-
2023-12-14T22:11:12.269536042Z 2023-12-14T22:11:12: PM2 log: App [YoutubeDL-Material:0] online
2023-12-14T22:11:13.283333715Z 2023-12-14T22:11:13.281Z INFO: Config items set using ENV variables.
2023-12-14T22:11:13.288313062Z (node:47) DeprecationWarning: uuidv4() is deprecated. Use v4() from the uuid module instead.
2023-12-14T22:11:13.288327746Z (Use `node --trace-deprecation ...` to show where the warning was created)
2023-12-14T22:11:16.172464802Z 2023-12-14T22:11:16.171Z INFO: YoutubeDL-Material v4.3.2 started on PORT 17442
2023-12-14T22:12:20.025774026Z <--- Last few GCs --->
2023-12-14T22:12:20.025808086Z [47:0x545c7f0]    67049 ms: Scavenge 4025.4 (4119.4) -> 4024.9 (4129.9) MB, 9.6 / 0.0 ms  (average mu = 0.454, current mu = 0.398) allocation failure 
2023-12-14T22:12:20.025813250Z [47:0x545c7f0]    67091 ms: Scavenge 4031.5 (4129.9) -> 4030.3 (4130.6) MB, 10.5 / 0.0 ms  (average mu = 0.454, current mu = 0.398) allocation failure 
2023-12-14T22:12:20.025818536Z [47:0x545c7f0]    67749 ms: Scavenge 4032.6 (4130.6) -> 4031.5 (4151.1) MB, 645.9 / 0.0 ms  (average mu = 0.454, current mu = 0.398) allocation failure 

Is there an environmental variable that I can change to widen the RAM used as a bypass for now? I have 24gb+ available. If it used 20gb for like 10 min I wouldn't care. Seems like it's only allocating 4gb or so from log.
I figured with mongodb I wouldn't have anymore issues but this one has popped up.

Installed version: v4.3.2 -

You are up to date.

Installation type: docker
Docker tag: latest
Commit hash: 13a03a7
Build date: 2023-05-27

[yllekz]

I think I'm experiencing this as well. Trying to simply subscribe to UrinatingTree's YouTube channel + telling it to only download 1080p quality videos within the last 3 days = infinite loading spin after clicking Subscribe. Seems to happen with most other channels too.