Circumventing twitter request quota

[ForestLamBZ]

I'm trying to pull media tweets and update them periodically in 600 profiles, two millions tweet in total

Lets say i already have all the tweets in the 600 profiles from yesterday.
To update the archive, the most straightforward way was to use the same command for the initial pull, let the app identify the existing tweet in local files and only downloading the new ones. However, in this way gallery-dl check every tweets, and as twitter imposed quota on requests, after checking 5 profiles i reached the quota. The remaining profiles will never be updated.

A while ago when i was trying to archive videos from youtube with yt-dlp, the same happened when i try to update my archive with the same command from the initial pull. yt-dlp do one requests for every single video in my playlist and it reach the quota after like 2000-2500 videos. The solution in the end was using yt-dlp --flat-playlist, getting the whole list of files and comparing that with the local side on the local machine, then only downloading the new videos not found in the local archive. With this i only need to do {new files}+1 requests, instead of {total files in playlist}.

i found -j, -G and -g to do the something similar. gallery-dl -j prints out a large json array containing every single tweets in each profiles, but again i reach the quota after 86 profiles ,with 74k tweets. similar happens to the remaining two. So my question is

1. Does there exists a way to do --flat-playlist a twitter profile like in yt-dlp, significantly reducing the request count for updating the files?
2. if not, how does the quota works? Im seeing some incoherent patterns over different commands, like the few hundred tweets from initial command, the 74k from -j. this question is definitely in the source code, but im having trouble understanding hte structure of the code so it would help if i got a little hint here

If i cant reduce the requests, i will only be able to update a few profiles per day, with each profile updating per week or two
Or, to create 10-20 accounts and distribute the quota to them(if there isnt another quota on my ip address)

Any information on the subject will help. Thanks

[mikf]

There is no --flat-playlist, but it wouldn't help here anyway.

You should probably use -A/--abort when updating a profile to stop early when going over previously downloaded media, or something like --filter "date < datetime(2024, 6, 31) or abort()" to stop when reaching Tweets before a certain date.

how does the quota works

Each API endpoint can only be called ~50 times in a 15 minute window. Any more and gallery-dl waits until those 15 minutes are over and Twitter allows to be queried again.

[Transwarpcom]

我正在尝试提取媒体推文，并在 600 个个人资料中定期更新它们，总共有 200 万条推文

假设我已经拥有了昨天 600 个个人资料中的所有推文。 要更新存档，最直接的方法是使用与初始拉取相同的命令，让应用程序识别本地文件中的现有推文并仅下载新推文。但是，以这种方式gallery-dl检查每条推文，并且由于 Twitter 对请求施加了配额，在检查了 5 个个人资料后，我达到了配额。其余个人资料将永远不会更新。

不久前，当我尝试使用 存档 YouTube 视频时yt-dlp，当我尝试使用与初始拉取相同的命令更新存档时，也发生了同样的情况。yt-dlp对播放列表中的每个视频执行一次请求，在 2000-2500 个视频后达到配额。最终的解决方案是使用yt-dlp --flat-playlist，获取整个文件列表并将其与本地计算机上的本地端进行比较，然后仅下载本地存档中未找到的新视频。这样，我只需要执行 {新文件}+1 次请求，而不是 {播放列表中的总文件数}。

我发现-j，-G并-g做了类似的事情。gallery-dl -j打印出一个包含每个个人资料中每条推文的大型 json 数组，但在 86 个个人资料之后，我再次达到配额，有 74k 条推文。其余两个也发生了类似的情况。所以我的问题是

1. 是否存在一种方法来创建--flat-playlist一个类似的 Twitter 个人资料yt-dlp，从而显著减少更新文件的请求数量？
2. 如果不是，配额是如何运作的？我看到不同命令中存在一些不连贯的模式，比如初始命令中的几百条推文，以及 中的 74k 条推文-j。这个问题肯定在源代码中，但我很难理解代码的结构，所以如果我在这里得到一点提示会有所帮助

如果我不能减少请求，我每天只能更新几个个人资料，每个个人资料每周或每两周更新一次， 或者创建 10-20 个帐户并将配额分配给他们（如果我的 IP 地址没有其他配额）

有关该主题的任何信息都会有帮助。谢谢

I am currently using "gallery-dl.exe https://x.com/just_for_example -T 5" which will stop downloading tweets for user "just_for_example" if it detects 5 tweets have been downloaded in a row. You can change this number to match your requirements, but in the case you describe, I'm sure changing 5 to 1 is acceptable.