Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error downloading user videos from Twitter while providing profile link #452

Closed
KaMyKaSii opened this issue Oct 20, 2019 · 8 comments
Closed

Error downloading user videos from Twitter while providing profile link #452

KaMyKaSii opened this issue Oct 20, 2019 · 8 comments

Comments

@KaMyKaSii
Copy link

KaMyKaSii commented Oct 20, 2019 •
edited by mikf
Loading

The download is successful with individual video links (but saved in an "i" folder rather than one with the correct username). Apparently it's a problem of youtube-dl itself, I'm not sure if it will be possible for you to find a solution

$ gallery-dl -v twitter.com/bb_puta
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.4 - Linux-4.4.141-perf+-aarch64-with-libc
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
[gallery-dl][debug] Starting DownloadJob for 'twitter.com/bb_puta'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterTimelineExtractor for 'twitter.com/bb_puta'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/bb_puta/ti
meline/tweets?include_available_features=1&include_entities=1&reset_error_state=false&la
ng=en HTTP/1.1" 200 19339
/sdcard/gallery-dl/twitter/bb_puta/1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1184559664303878150: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1162026165206487040: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
[downloader.ytdl][debug] [twitter] 1161866872616366081: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1116823538307039232_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1161866431119679488_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_1.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_2.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_3.jpg
/sdcard/gallery-dl/twitter/bb_puta/1160620638358384641_4.jpg
/sdcard/gallery-dl/twitter/bb_puta/1156267861675008006_1.jpg
[downloader.ytdl][debug] [twitter] 1156069994196852736: Downloading webpage
[downloader.ytdl][error] ERROR: Unable to extract OpenGraph title; please report this is
sue on https://yt-dl.org/bug . Make sure you are using the latest version; see  https://
yt-dl.org/update  on how to update. Be sure to call youtube-dl with the --verbose flag a
nd include its complete output.
[download][error] Failed to download 1156267861675008006_1.jpg
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/profiles/show/bb_puta/ti
meline/tweets?include_available_features=1&include_entities=1&max_position=1155506042488
442880&reset_error_state=false&lang=en HTTP/1.1" 200 19141
[downloader.ytdl][debug] [twitter] 1154461534921396230: Downloading webpage
^C
KeyboardInterrupt
$
@mikf
Copy link
Owner

mikf commented Oct 20, 2019 •
edited
Loading

See #352. Downloading Twitter videos with youtube-dl while being logged in doesn't work at the moment, but you can disable forwading cookies to youtube-dl and it will work again, although I'm not sure if you can download private videos like that.

edit: you should put console logs in triple backticks to make them more readable:

```
<your log here>
```

@KaMyKaSii
Copy link
Author

See #352. Downloading Twitter videos with youtube-dl while being logged in doesn't work at the moment, but you can disable forwading cookies to youtube-dl and it will work again, although I'm not sure if you can download private videos like that.

edit: you should put console logs in triple backticks to make them more readable:

<your log here>

Thanks, your tip has fixed downloading public videos but not private videos, as you suspected. I did a bit of research and found that private videos don't need authentication to download, it just needs to get the url. The log below is an attempt to download a private video, apparently gallery-dl is unable to extract the url. And if it is not gallery-dl's job to extract direct video links to youtube-dl, could you consider an option to extract them and use ffmpeg instead?

$ gallery-dl -v https://mobile.twitter.com/ecchikawaiigirl/status/1184539361641140226
[gallery-dl][debug] Version 1.11.0-dev
[gallery-dl][debug] Python 3.7.4 - Linux-4.4.141-perf+-aarch64-with-libc
[gallery-dl][debug] requests 2.22.0 - urllib3 1.25.3
[gallery-dl][debug] Starting DownloadJob for 'https://mobile.twitter.com/ecchikawaiigirl
/status/1184539361641140226'
[gallery-dl][debug] Updating urllib3 ciphers
[twitter][debug] Using TwitterTweetExtractor for 'https://mobile.twitter.com/ecchikawaii
girl/status/1184539361641140226'
[urllib3.connectionpool][debug] Starting new HTTPS connection (1): twitter.com:443
[urllib3.connectionpool][debug] https://twitter.com:443 "GET /i/web/status/1184539361641
140226 HTTP/1.1" 302 103
[urllib3.connectionpool][debug] https://twitter.com:443 "GET / HTTP/1.1" 200 50449
[twitter][error] An unexpected error occurred: ValueError - substring not found. Please 
run gallery-dl again with the --verbose flag, copy its output and report this issue on h
ttps://github.com/mikf/gallery-dl/issues .
[twitter][debug]
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/job.py", 
line 47, in run
    for msg in self.extractor:
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/extractor
/twitter.py", line 42, in items
    for tweet in self.tweets():
  File "/data/data/com.termux/files/usr/lib/python3.7/site-packages/gallery_dl/extractor
/twitter.py", line 264, in tweets
    end = page.index('class="js-tweet-stats-container')
ValueError: substring not found
$

@mikf
Copy link
Owner

mikf commented Oct 20, 2019

Extracting and downloading video streams can be rather complicated, hence youtube-dl gets used for it here. Maybe I'll look into it, but I'd rather wait for youtube-dl to fix their code.

... And I just realized that gallery-dl suffers from the same problem as youtube-dl when it comes to individual Tweets: Requesting a Tweet page while being logged in would get it in the new layout, so gallery-dl clears its cookies (and authentication data) before accessing that Tweet, since it currently only supports the old Twitter interface.

@alice945
Copy link
Contributor

Requesting a Tweet page while being logged in would get it in the new layout, so gallery-dl clears its cookies (and authentication data) before accessing that Tweet, since it currently only supports the old Twitter interface.

Changing your user agent to "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko" forces the old layout, it's what I've been using ever since twitter changed it. Maybe you could make it use this user agent anytime you use authentication?

mikf added a commit that referenced this issue Oct 23, 2019
@mikf
[twitter] fix image extraction when logged in (#452)
2eb3881 
... for individual tweets.

To get a Tweet page with the old Twitter layout, an Internet
Explorer User-Agent (e.g. Mozilla/5.0 (Windows NT 6.1; WOW64;
Trident/7.0; rv:11.0) like Gecko) as well as a Referer header
pointing to the page itself is required. The "app_shell_visited"
cookie appears to be optional at the moment, but that is what
a regular web browser would send.
@mikf
Copy link
Owner

mikf commented Oct 23, 2019

Thank you @alice945. Setting a user agent alone wasn't enough to convince Twitter to revert to its old layout, but it got me looking in the "right" direction. (See 2eb3881)

The "only" problem now is getting youtube-dl to do the same when extracting a video stream from a Twitter page, but as far as I can tell that's impossible. The only way appears to be updating youtube-dl's code and opening a pull request there, but I'd rather do what @KaMyKaSii suggested and look into getting the video information without youtube-dl.

mikf added a commit that referenced this issue Oct 26, 2019
@mikf
[twitter] extract video stream info without youtube-dl (#452)
49a6b1b 
This should allow video downloads when logged in without
'forward-cookies' disabled and from protected tweets.

youtube-dl still gets used to download HLS playlists, but the data
extraction part, which doesn't work with youtube-dl at the moment,
now gets handled by gallery-dl itself.
@mikf
Copy link
Owner

mikf commented Oct 26, 2019

OK, I've managed to implement video extraction (49a6b1b). It worked for me with and without being logged in, but I don't have access to any protected account so I couldn't test that.

@KaMyKaSii
Copy link
Author

OK, I've managed to implement video extraction (49a6b1b). It worked for me with and without being logged in, but I don't have access to any protected account so I couldn't test that.

It worked here, thank you very much!
Screenshot_20191026-100401961

@mikf mikf closed this as completed Oct 26, 2019
@github-userx
Copy link

github-userx commented Oct 26, 2019 via email

@mikf mikf mentioned this issue Oct 31, 2019
Closed
@mikf mikf mentioned this issue Dec 25, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mikf @KaMyKaSii @github-userx @alice945