[twitter] Guest access has been revoked.

[Aerocatia]

Attempting to download a post without valid login cookies will now return a 404 error.
Apparently there are much stronger rate limits based on account age.
I'm not sure if there is a way around this.

[jadedgnome]

reddit can probably be used an referrer and the embed endpoint is working on discord too.

[Soeroah]

Is this at all related to the "waiting until X for cache limit to reset" error I've suddenly started receiving tonight, or is that just a twitter bug?

[daniels1989]

You need an account to even be able to access twitter since yesterday, and just a few hours ago they implemented rate limited for all accounts as well. Subscribers are now limited to viewing 6000 tweets per day, unverified accounts to 600, and new accounts to 300. All to "combat" scraping.

[biggestsonicfan]

Terrified. I've been trying my hardest to archive as much of Twitter as I can, but now after gallery-dl scrapes my limit, I won't be able to use the site unless I pay for verification... Even then 6k posts is not enough for my daily archiving...

[serveral1]

the rates are so low that this is going to tank their activity, which means less people being served ads. elon said this is a temporary measure, and most likely a test before they decide on a hard limit that doesn't fuck up the site's engagement.

[AlttiRi]

Twitter now requires you to login in order to look at anything on the site

https://www.reddit.com/r/mildlyinfuriating/comments/14mxx8c/twitter_now_requires_you_to_login_in_order_to/
https://twitter.com/TitterDaily/status/1674765247255654400
(2023.06.30)

To address extreme levels of data scraping & system manipulation, we’ve applied the following temporary limits:

• Verified accounts are limited to reading 6000 posts/day
• Unverified accounts to 600 posts/day
• New unverified accounts to 300/day

https://twitter.com/elonmusk/status/1675187969420828672
(2023.07.01)

Cursed.

[grug-son-of-thog]

Well, this change is gonna suck. I archived a lot of art accounts. Guess that's over.

[Twi-Hard]

This is probably why he said "temporary"
https://twitter.com/filmthepolicela/status/1675243929451765760

Edit:
https://www.engadget.com/twitter-has-supposedly-started-paying-its-google-cloud-bill-again-213824844.html

[Doofy420]

I'm able to bypass this somewhat with multiple accounts since the rate limit is per user. Not sure if it's feasible but maybe adding the option for g-dl to automatically cycle through cookies/accounts when limits are met would help. I honestly wouldn't put it beyond Elon to permanently implement this more draconian limit just to get people to get blue.

[Infinitay]

What if we use mobile a mobile endpoint or a mobile twitter app as the user agent? I haven't tried it so I can't confirm it, but I want to say that it's possible because when they first implemented this I was able to view tweets at some moments using the app where the website was unusable for me

[biscuitsan]

Does the temporary("temporary") 300/600-tweet daily rate limit apply to the types of requests gallery-dl does?
Because... that's, uh... not a lot. The average artist account in my head might use, what, 4000?
Obviously not happy, but I have some ideas in mind for getting around it if this doesn't go away. I won't give up on archiving culture that easily.

[aleksusklim]

Access to media files on twimg.com by direct links is working for guests without login (at least for now).
Also, embedding of tweets by ID is working too: https://platform.twitter.com/embed/Tweet.html?id=1674865731136020505

Found it at zedeus/nitter#919 (comment)
Looks like they found something down the thread...

Personally, I think it's better to give Elon several days to change his decision, or give Twitter enough time to restrict other methods as well (so we don't spend time on implementing workarounds that will stop work literally tomorrow).

[ImTheDeveloper]

Access to media files on twimg.com by direct links is working for guests without login (at least for now). Also, embedding of tweets by ID is working too: https://platform.twitter.com/embed/Tweet.html?id=1674865731136020505

Found it at zedeus/nitter#919 (comment) Looks like they found something down the thread...

Personally, I think it's better to give Elon several days to change his decision, or give Twitter enough time to restrict other methods as well (so we don't spend time on implementing workarounds that will stop work literally tomorrow).

zedeus/nitter#919 (comment) we've got a bit further with this method. Accessing for example https://syndication.twitter.com/srv/timeline-profile/screen-name/elonmusk

[nisehime]

Looks like the daily rate limits are gone already, or at least the number have been increased. I've been able to update my list of about 500 account links today with no problem.

[a84r7a3rga76fg]

I get rate limited for each GET request

[twitter][info] Waiting until 01:00:29 for rate limit reset.
[urllib3.connectionpool][debug] Resetting dropped connection: api.twitter.com
[urllib3.connectionpool][debug] https://api.twitter.com:443 "GET
[twitter][info] Waiting until 01:15:31 for rate limit reset.
[urllib3.connectionpool][debug] Resetting dropped connection: api.twitter.com
[urllib3.connectionpool][debug] https://api.twitter.com:443 "GET

		"timeline.strategy": "media",
		"image-filter": "author is user",
		"replies": "self",
		"quoted": true,
		"twitpic": true,
		"retweets": "original",
		"videos": true,
		"cards": false,
		"expand": false,
		"pinned": true,
		"skip": true,
		"retries": -1,
		"metadata": true

[anonymous721]

Looks like the daily rate limits are gone already, or at least the number have been increased. I've been able to update my list of about 500 account links today with no problem.

They were still active for me as of ten hours ago. I only got as far as roughly the 600th artist in my list.

[nisehime]

I only got as far as roughly the 600th artist in my list.

It stopped for me somewhere on 130th artist previously. Today I was able to go through the whole list + use Twitter normally without limits (not for long though). I'll try updating list today a few hours later again.

[anonymous721]

It looks like there's more going on than the simple tweets/day limit previously announced. I have two pre-existing accounts, and both were able to view tweets on the actual website, even though a gallery-dl script I had just started to run using one of them was hanging. Then as I typed this, it resumed downloading. And after it hung a second time, I could still view tweets in-website with that account.

[Soeroah]

I've noticed a pretty drastic drop in the amount of art my usual artists are putting on twitter. Understandably so. I wish there was a way to more easily transition all my links to their twitters to their pixiv accounts or what have you.

[Twi-Hard]

I wonder if scraping only search results would trigger the limit. It still gets all the metadata without needing to go to each tweets page individually I think. Search results are where I get most of the tweets through gallery-dl because the profile and media feeds are very limited.

[buratsy]

is anyone else getting this error? it's like this even with a /media link. my last download was 2 days ago and it was working fine at the time.
[twitter][error] 404 Not Found ()

i'm not rate-limited right now and can access twitter just fine. i just updated my g-dl but it still won't work.

i forgot but this also appeared at some point in my attempts which is weird because i have a login in the config. are cookies also needed now?
[twitter][info] Requesting guest token

[Sphirye]

I can still remember myself and many of us several months ago actually believing that the Elon twitter acquisition was going to be favorable.

[AAndyProgram]

@Sphirye, Musk is a fascist and he shouldn't own such a company!

[Soeroah]

I'm surprised anyone thought him buying Twitter would be favourable, honestly, but I supposed not everyone is in the same circles. Everyone I was seeing knew it was a bad idea from the word go.

Regardless, I've noticed a bit of new behaviour today - at least, this is the first time I'm noticing it: I'm hitting the actions per minute (?) rate limit more frequently. Not the daily one, cuz my gallery continues downloading after 15 minutes unlike when I hit the daily limit, but it seems easier to hit the temporary rate limit than it was the last couple of days.

Anyone else noticed this?

Edit: Yeah, this is definitely new behaviour. The last few days I'd be able to pull from 260 accounts in about 20 minutes; today I'm still only halfway through after close to two hours. Seems to cap out at around 45-50 pull requests within a several-minute timeframe.

[a84r7a3rga76fg]

I am no longer getting rate limited for each request, now it lets me download tens of tweets before rate limiting me for half an hour.

[Sphirye]

Are you using the last version? Im still receiving this 404 Not Found thing

[nisehime]

I also haven't been getting any rate limits for last days. Btw you can access tweets without logging in again now, but still not whole profiles

[mikf]

Are you using the last version? Im still receiving this 404 Not Found thing

Be logged in and use -o search-endpoint=graphql (#4264 (comment))

[a84r7a3rga76fg]

I misspoke, I'm back to being rate limited in perpetuity again.

[twitter][info] Waiting until 10:45:57 for rate limit reset.
[twitter][info] Waiting until 10:60:58 for rate limit reset.
[twitter][info] Waiting until 11:15:59 for rate limit reset.
[twitter][info] Waiting until 12:31:00 for rate limit reset.
...
[twitter][info] Waiting until 12:31:43 for rate limit reset.

[grug-son-of-thog]

The problem went away for a bit, but now it's back. (On v1.25.8-dev).

[buratsy]

i can download from twitter normally again, but i'm still getting this message
[twitter][info] Requesting guest token

i think this is gonna be an issue when i try to download from follower-only profiles. will this be fixed in the next update? i have a login in my config, but no cookies. do i need to put in twitter cookies now?

[Infinitay]

After the recent config addition to specify which endpoint to use: https://github.com/mikf/gallery-dl/blob/master/docs/configuration.rst#extractortwittertweet-endpoint

Is there a way to make it so that gallery-dl uses the guest endpoint (restid) whenever we hit the rate limit for the logged in account? or will we have to run two separate processes? How about having it the otherway around where gallery-dl automatically uses the guest endpoint for public accounts and switches to the logged in endpoint when fetching from protected accounts?

EDIT: I created an issue with more details on this #4353

[margot-l]

I still can't download from twitter and am stilling getting the 404 Not Found error

gallery-dl "https://twitter.com/kyungsoo612_/status/1685676195718041601" --verbose
[gallery-dl][debug] Version 1.25.8 - Git HEAD: 3db1acf3e
[gallery-dl][debug] Python 3.11.4 - macOS-13.5-arm64-arm-64bit
[gallery-dl][debug] requests 2.31.0 - urllib3 2.0.4
[gallery-dl][debug] Configuration Files ['${HOME}/.config/gallery-dl/config.json']
[gallery-dl][debug] Starting DownloadJob for 'https://twitter.com/kyungsoo612_/status/1685676195718041601'
[twitter][debug] Using TwitterTweetExtractor for 'https://twitter.com/kyungsoo612_/status/1685676195718041601'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/api/graphql/JlLZj42Ltr2qwjasw-l5lQ/TweetDetail?variables=%7B%22focalTweetId%22%3A%221685676195718041601%22%2C%22referrer%22%3A%22profile%22%2C%22with_rux_injections%22%3Afalse%2C%22includePromotedContent%22%3Atrue%2C%22withCommunity%22%3Atrue%2C%22withQuickPromoteEligibilityTweetFields%22%3Atrue%2C%22withBirdwatchNotes%22%3Atrue%2C%22withVoice%22%3Atrue%2C%22withV2Timeline%22%3Atrue%7D&features=%7B%22rweb_lists_timeline_redesign_enabled%22%3Atrue%2C%22responsive_web_graphql_exclude_directive_enabled%22%3Atrue%2C%22verified_phone_label_enabled%22%3Afalse%2C%22creator_subscriptions_tweet_preview_api_enabled%22%3Atrue%2C%22responsive_web_graphql_timeline_navigation_enabled%22%3Atrue%2C%22responsive_web_graphql_skip_user_profile_image_extensions_enabled%22%3Afalse%2C%22tweetypie_unmention_optimization_enabled%22%3Atrue%2C%22responsive_web_edit_tweet_api_enabled%22%3Atrue%2C%22graphql_is_translatable_rweb_tweet_is_translatable_enabled%22%3Atrue%2C%22view_counts_everywhere_api_enabled%22%3Atrue%2C%22longform_notetweets_consumption_enabled%22%3Atrue%2C%22tweet_awards_web_tipping_enabled%22%3Afalse%2C%22freedom_of_speech_not_reach_fetch_enabled%22%3Atrue%2C%22standardized_nudges_misinfo%22%3Atrue%2C%22tweet_with_visibility_results_prefer_gql_limited_actions_policy_enabled%22%3Afalse%2C%22interactive_text_enabled%22%3Atrue%2C%22responsive_web_text_conversations_enabled%22%3Afalse%2C%22longform_notetweets_rich_text_read_enabled%22%3Atrue%2C%22longform_notetweets_inline_media_enabled%22%3Afalse%2C%22responsive_web_enhance_cards_enabled%22%3Afalse%7D HTTP/1.1" 404 0
[twitter][error] 404 Not Found ()