Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Preventing Error 429: Too Many Requests #635

Open
nicolasdeory opened this issue May 25, 2020 · 76 comments · Fixed by #713
Open

Preventing Error 429: Too Many Requests #635

nicolasdeory opened this issue May 25, 2020 · 76 comments · Fixed by #713

Comments

@nicolasdeory
Copy link

I'm using ytdl-core from a browser, using browserify. Cross-domain requests are not allowed, so I'm using CORS Anywhere proxy to bypass CORS Header related errors.

After a few requests, I'm getting error 429s from YouTube, saying that I may be making too many requests (not really, one or two per minute at the most, downloading a webm stream). The method crashes afterward and won't do any additional requests.

Google seems to block the entire origin domain, so it won't work for anyone with the same origin header (I haven't verified yet).

There are multiple things wrong:

  • How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)
  • No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

Code example:

const ytdl = require('ytdl-core');
const concat = require('concat-stream');
 var concatStream = concat((arrayBuffer) =>
        {
            console.log('file downloaded');
           // Processing of video blob goes here...
        });

        var ytstream = ytdl(YOUTUBE_VIDEOURL,
            {
                requestOptions: {
                    transform: (parsed) =>
                    {  
                        var parsedHeaders = parsed.headers;
                        return {
                            host: 'cors-anywhere.herokuapp.com/', 
                            path: "http://youtube.com" + parsed.path,
                            maxRedirects: 10,
                            headers: parsedHeaders
                        }
                    },
                },
            });

        ytstream.on('error', function(e)
        {
           // Never gets called
        })
        .pipe(concatStream);
@fent
Copy link
Owner

fent commented May 26, 2020

  • How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)

I'm going to be adding throttling so that it doesn't make requests too fast to prevent these errors

  • No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

will be fixed soon

@fent fent added the bug label May 26, 2020
@akanellis
Copy link

I am a fairly new JS developer I come from old school C/C++ background and I am seriously really confused with this.

I am trying to get ytdl-core to pass cookies to miniget because this is a solution for the 429 error. I have tested using cookies with youtube-dl and it works perfectly but I want to make it work with ytdl-core because that other project is using deprecated libraries and I don't feel good with that.

const ytdl = require('ytdl-core');
const cookiefile = require('cookiefile');

const cookiemap = new cookiefile.CookieMap('cookies.txt');
const cookies = cookiemap.toRequestHeader().replace ('Cookie: ','');

stream = ytdl(URL, {
    quality: 'highestaudio',
    highWaterMark: 1024 * 1024 * 1024,
    requestOptions: {
        headers: {
            'Cookie': cookies
        }
    }
});

cookies.txt is a cookies export from chrome.
I have also modified miniget's index.js so that it prints out variable "parsed" on line 126 just so that I can see what I'm doing.
When I run the above code, I get the following output for parsed:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  query: 'v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  pathname: '/watch',
  path: '/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  href: 'https://www.youtube.com/watch?v=Rbm6GXllBiw&hl=en&bpctr=1590461824&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    Cookie: 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

Under headers, "cookie" is not inside quotes just like the rest of the headers and I guess it's natural that I get the following error:

(node:7309) UnhandledPromiseRejectionWarning: Error: Error parsing info: JSON.parse(...).reduce is not a function
    at exports.getBasicInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:43:11)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Object.exports.<computed> [as getBasicInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)
    at async exports.getFullInfo (/home/user/project/node_modules/ytdl-core/lib/info.js:188:14)
    at async Object.exports.<computed> [as getFullInfo] (/home/user/project/node_modules/ytdl-core/lib/info.js:297:18)

But when I use any other header name (for example 'custom-header', it gets inserted correctly like this:

Url {
  protocol: 'https:',
  slashes: true,
  auth: null,
  host: 'www.youtube.com',
  port: null,
  hostname: 'www.youtube.com',
  hash: null,
  search: '?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  query: 'v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  pathname: '/watch',
  path: '/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  href: 'https://www.youtube.com/watch?v=TnAgc1kgvLc&hl=en&bpctr=1590462647&pbj=1',
  maxRedirects: 2,
  maxRetries: 2,
  maxReconnects: 0,
  backoff: { inc: 100, max: 10000 },
  headers: {
    'custom-header': 'my-cookies-here',
    'x-youtube-client-name': '1',
    'x-youtube-client-version': '2.20191008.04.01'
  }
}

The above does not produce a JSON.parse error, it produces the usual Error: Status code: 429 which means I guess that the header was valid and went through the request.

So whichever name I use for header name is valid except for Cookie. Why is that? I've been trying to figure this out for the past 6 hours. I apologize in advance if my question is silly. My JS background is very limited.

@fent
Copy link
Owner

fent commented May 26, 2020

the way that console.log prints objects, it will put quotes around attribute names that contain a non-alphanumeric character, it does not affect the actual internal attribute name used.

as for why it's not working and the error you're getting, try lowercasing the header name to "cookie".

if that doesn't work, it's likely that having that cookie there prevents the endpoint ytdl-core uses internally to work. you could try only setting the cookie during the download. something like

let info = await ytdl.getInfo(id);
ytdl.downloadFromInfo(info, { requestOptions: { Cookie: cookie } });

@akanellis
Copy link

Thank you for the speedy answer. I'm afraid this doesn't work.. I still get the 429 error but now I get it from the getInfo() function. I need to set cookie for every request that I send to youtube and I can't find an example code anywhere online about setting cookies with ytdl-core that works :(

Maybe you could add an example for cookies in the example folder than you have?

If you are not throttled by youtube with error 429 you can recreate the scenario trying to download a private video. Only when using cookies you will be able to get it.

@harshmandan
Copy link

Same here. Getting a bursty stream of requests blocks all videos. Would love to see a correct cookie implementation

@FeX5858
Copy link

FeX5858 commented May 31, 2020

I have a similar problem with my discord.js bot and ytdl by itself. If i use ytdl "anyurl" --print-url it throws the Error: 429. When i use ytdl-core for my discord bot, the bot joins but has no audio. The player doesnt receive any ytdl data. (buffer length: 0). The error started around 7 to 8 months using the bot on 1 Server.

@NafiAF
Copy link

NafiAF commented Jun 11, 2020

I am also getting this error, it was working fine all this time (since I started using it, 3 months ago) but now I am suddenly getting a 429 error with my Discord Bot. It still happens after small requests even when the Bot did not request any data for around 12 hours.

@FeX5858
Copy link

FeX5858 commented Jun 11, 2020

@NafiAF I solved the problem for me. I waited around 4 to 5 days and then it was gone. The error triggeres when ur bot gets too many requests in a small amount of time. Then Google blocks the bot for some days to prevent DDOS. This is what works for me :)

@akanellis
Copy link

@NafiAF Personally I have solved this issue by not using ytdl-core anymore. I have moved to youtube-dl because that one has easy support for cookies. When you use cookies so that you are logged into youtube, the error 429 does not occur anymore.

@mahhov
Copy link

mahhov commented Aug 6, 2020

For what it's worth, @fent and @akanellis 's solution of using cookies worked, at least for now. I just hope my account doesn't get blocked or something.

I navigated to youtube.com in chrome, opened dev console, and copied the cookies from a network request.
Then do:

ytdl(<id>, {
            ...
            requestOptions: {
                headers: {
                    'Cookie': '<paste cookie as string>'
                }
            }
        })

@mahhov
Copy link

mahhov commented Aug 7, 2020

Apparently, even using oauth isn't immune to 429.
Using oauth in the request options (e.g. {filter:'audioonly', requestOptions: {headers: {Authorization: 'Bearer ...'}}}gives me ~15-25 videos, before getting another 429.
Refreshing the access token (but using the same refresh token and thereby not requiring user re-login) gives me another 15-25 videos.
This is just ~1 video per 3-5 minutes.
(I'm only 75% this comment is accurate :), I'll update it once I verify it again)

@mahhov
Copy link

mahhov commented Aug 7, 2020

Confirmed, my above comment is accurate. Refreshing the oath token does resolve the issue. It took 20 downloads to reach http 429. I guess something like this should work then

ytdl(...)
  .on('error', error => {
    if (error.message === 'Status code: 429') {
      // refreh token
      // try again
    }
  })

@eric-mathison
Copy link

I'm 429 blocked now too so I've been playing around with cookies. I've narrowed it down to these three:
'SID=xxx; HSID=xxx; SSID=xxxx;'

I've been able to make successful calls after setting these manually.
They are all set to expire in July 2022, so hopefully I can get some good use out of them until then.

@WaqasIbrahim
Copy link
Contributor

@eric-mathison Were you able to make requests from a blocked IP?

@eric-mathison
Copy link

@eric-mathison Were you able to make requests from a blocked IP?

I couldn't confirm that my IP was blocked, but I kept getting error 429 from my node application running local on my laptop. I waited over 14 hours and it was still receiving this error. Passing those 3 cookies is working. I just finished downloading 52 videos in around 7 minutes time. They are small 75mb and I downloaded them slow, 1 at a time. So far so good.

@WaqasIbrahim
Copy link
Contributor

WaqasIbrahim commented Aug 18, 2020

Screenshot_20200818_130349

Does not seem to work on a server that's already blocked.

Screenshot_20200818_131032

Edit: Same request works fine in Postman.

@fent
Copy link
Owner

fent commented Aug 18, 2020

Screenshot_20200818_130349

Does not seem to work on a server that's already blocked.

Screenshot_20200818_131032

Edit: Same request works fine in Postman.

it's showing 10 headers in postman, maybe that's helping

@WaqasIbrahim
Copy link
Contributor

WaqasIbrahim commented Aug 18, 2020

@fent Those are standard headers, I generated the curl command using postman.

Update:
I disabled IPv6 on the server.
Without headers and cookie:
Screenshot_20200818_192330

With headers and cookie
Screenshot_20200818_192106

Update 2:
Did not last long, went back to 429.

@eric-mathison
Copy link

Screenshot_20200818_130349

Does not seem to work on a server that's already blocked.

Screenshot_20200818_131032

Edit: Same request works fine in Postman.

@WaqasIbrahim that doesn't look like an IP block. You are getting blocked by reCaptcha. See if you can curl the get_video_info url directly. YouTube doesn't block the watch url. Only the get info.

@WaqasIbrahim
Copy link
Contributor

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

@eric-mathison
Copy link

@eric-mathison get_video_info seems to be working but watch page URL is used first. get_video_info is a fallback URL.

Eh, you're probably right. I was just able to get it working from the get_video_info URL since the watch pages never broke for me.

@Abhay07
Copy link

Abhay07 commented Sep 19, 2020

My IP was blocked and i can confirm that. setting proxy or anything didn't work. Had to spin up a new server.

@angelhodar
Copy link

there's already a memory cache i place, but it's currently for 1 sec, and currently non configurable

So that would be like a throttling mechanism right? What if people could configure that caching time? I have seen the youtube-dl repo and they only use caching for saving some time when decrypting video info but not the already requested video data.

@orfox32
Copy link

orfox32 commented Feb 24, 2021

does anyone know how to fix the error 429 coz i got it again after 1 week even though i have a cookie

@twarped
Copy link

twarped commented Feb 26, 2021

actually, i ended up using youtube-dl. after messing around, I accidentally came up with this solution:
youtubedl.getInfo(videoUrl, function(err, info) {
res.header("Content-Disposition", "attachment; filename=\""+info.title+".mp4\""); //filename optional. either use content-disposition or res.write("") to make it download directly to response
res.write("")//without this (if you don't have content-disposition), it just streams the video within the browser
require("request").get(info.url).pipe(res)
});
i tested it about 50 times without coming about with 429
I couldn't get 429

here's my code/example: (https://tattered-lead-archaeopteryx.glitch.me/)

@redbrain
Copy link
Contributor

Bottom line, for all you newcomers: 429 is YouTube trying to limit you. It is not a problem with the ytdl-core project.
Our friend Promise in the Discord guild pointed out the following solutions:

  1. Make sure you're at the latest version of ytdl-core
  2. Try using cookies (example in examples folder) This works quite well most of the time.
  3. Try using proxies (example in examples folder) This is difficult to set up but mostly foolproof
  4. Switch to IPv6 (if you aren't already) Block-rotating is coming soon, plus it gives you a new address upfront.
  5. If all else fails, just wait between a few hours and a few days, and it will go away.

fent, I don't know how GitHub Issues works, but perhaps place these pointers somewhere important so we don't get more people asking the same question.

@billpan04
Copy link

Can I get a detailed explanation on how to use cookies within my code? because I did not understand a thing with the little to no explanation provided in the examples folder

@redbrain
Copy link
Contributor

redbrain commented Mar 5, 2021

The file examples/cookies.js is an example of code that could be used to provide a cookie to a ytdl request, as well as instructions on how to obtain a cookie from your browser.

@billpan04
Copy link

I know... I asked for a detailed explanation because the explanation that the file had were so little that I couldnt understand anything

@redbrain
Copy link
Contributor

To use the YouTube cookie, you get it from your browser and put it as a string in ytdl's requestOptions.headers.cookie.

Lines 6-12 of the file contain very thorough instructions on how to find that cookie. From there you just paste the cookie into the request options as I've described, wherever you use a ytdl requesting function, such as ytdl() or getInfo().

I've basically just restated what that file said, because that's literally all possible information you would need to implement it.
If you still don't understand something, you'll have to explain what exactly you don't understand.

@billpan04
Copy link

Thank you I managed to figure out how to use cookies. For anyone with the same problem take a look at #661 it might help understand how to obtain and add them to your code.

@twarped
Copy link

twarped commented Apr 22, 2021

Do people still get 429? I just downloaded 100 videos as a test and I never got it... Maybe Youtube changed their systems

@mpirescarvalho
Copy link

Yes, I do... sometimes... I'm using heroku, when it starts to happen I just restart the dyno and it works fine again

@InfiniteMarcus
Copy link

I'm using Heroku too and 429 still appears, even with cookies. Restarting prevents it for some time, but unfortunately is not a solution.

@twarped
Copy link

twarped commented Apr 22, 2021

hmm... i have a fork that im using, it doesn't have full audio and quality options yet, but you could try using it, https://github.com/twarped/node-ytdl-core/tree/patch-3
https://tattered-lead-archaeopteryx.glitch.me is my code example... if you want to see that...

@YeyintThway13
Copy link

  • How can I prevent this? I'd like to have some sort of rate throttling so it doesn't incur into 429 errors. It renders ytdl completely unusable for a long time (no exact amount, last time it was more than an hour)

I'm going to be adding throttling so that it doesn't make requests too fast to prevent these errors

  • No error is thrown by ytdl whatsoever. This is not good because there is no way to handle the error. The callback never gets invoked, and error handlers are never called.

will be fixed soon

Still got these errors

@EPXOL
Copy link

EPXOL commented Jun 19, 2021

Hello I looked into this problem and find solution go to node_modules/lib/info.js in there find function named getBasicInfo you can see somethink like this in it:

  let info = await pipeline([id, options], validate, retryOptions, [
    getWatchHTMLPage,
    getWatchJSONPage,
    getVideoInfoPage,
  ]);

rewrite it to

  let info = await pipeline([id, options], validate, retryOptions, [
    getWatchHTMLPage,
    getVideoInfoPage,
  ]);

WatchHTMLPage is simular to WatchJSONPage it does not efect any of data everything will be downloaded from Watch Page

Why does this work ?
It works becouse JSONPage uses access token to make request and that is cousing too many requests error that meens that if we disable request to JSONPage access token will not be used and there will not be any more too many requests error.

This works with downloading of streams too becouse when you are downloading stream ytdl-core uses getInfo function to get video URLS.

@redbrain
Copy link
Contributor

Can we confirm that watchJSONPage is completely obsoleted by watchHTMLPage, or is it needed for some things?

@EPXOL
Copy link

EPXOL commented Jun 22, 2021

I can confirm that it's basicly the same becouse watchJSONPage is basicly just raw JSON from watchHTMLPage you can go to source of package and try it yourself. Basicly all of these 3
getWatchHTMLPage,
getWatchJSONPage,
getVideoInfoPage,
Are the same. It returns same data.

@EPXOL
Copy link

EPXOL commented Jun 22, 2021

Btw fixed new bug of ytdl-core when youtube updated API #939 (comment) Includes this fix within it as well.

@securing-ab
Copy link

Nope,it isn’t fixed,I’m still getting the error.

@smironescu
Copy link

Seems that on our server the errors 429 had started but locally, with the same cookie all is working fine :S.

@Instinzts
Copy link

Thank you I managed to figure out how to use cookies. For anyone with the same problem take a look at #661 it might help understand how to obtain and add them to your code.

Mind, Giving Me An Example How To Do It The Example Provided inst enough for me to understand

@github-actions
Copy link

github-actions bot commented Mar 5, 2022

🎉 This issue has been resolved in version 4.11.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

@YueMiyuki
Copy link

Please check this: https://www.npmjs.com/package/ytcr
:D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.