Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Print traceback when raising UnavailableVideoError #10888

Closed

Conversation

johnhawkinson
Copy link
Contributor

  • I am the original author of this code and I am willing to release it under Unlicense
  • Improvement

An OSError or IOError generally indicates something a little more
wrong than a "simple" UnavailableVideoError, so print the actual
traceback that leads to the exception. Otherwise meaningful postmortem
debugging from a bug report is essentially infeasible.

Forked this pull request from #10876 (Fix UnavailableVideoError when writing to pipes) as requested by @yan12125.

An OSError or IOError generally indicates something a little more
wrong than a "simple" UnavailableVideoError, so print the actual
traceback that leads to the exception. Otherwise meaningful postmortem
debugging a bug report is essentially infeasible.
@yan12125
Copy link
Collaborator

yan12125 commented Oct 9, 2016

Exception handling in youtube-dl is quite messy...

The best way to fix this is exploiting Python 3's exception chaining. [1] python-future [2] provides raise_from function for similar semantic on Python 2, but #10581 should be resolved first before using it.

A possible short term fix is introducing the cause mechanism to UnavailableVideoError like ExtractorError.

Anyway the traceback should NOT be printed to stderr directly. Use self.to_stderr instead.

[1] https://www.python.org/dev/peps/pep-3134/
[2] https://github.com/PythonCharmers/python-future

johnhawkinson referenced this pull request in johnhawkinson/youtube-dl Oct 9, 2016
@johnhawkinson
Copy link
Contributor Author

Anyway the traceback should NOT be printed to stderr directly. Use self.to_stderr instead.

All of the traceback functions print to stderr, including traceback.print_exc(), called here.
Using self.to_stderr() would seem redundant and unnecessary.

@yan12125
Copy link
Collaborator

yan12125 commented Oct 9, 2016

In CLI they did print to stderr, while applications that use the Python API may intercept them using 'logger'.

@johnhawkinson
Copy link
Contributor Author

Huh? It's clear they go to stderr in youtube-dl.
See the repro case from #10809 where the traceback is printed (to stderr) while stdout is being redirected to a pipe.

Are you saying you still want me to do

_self.to_stderr(traceback.format_exc()))?

?

@yan12125
Copy link
Collaborator

yan12125 commented Oct 9, 2016

It's clear they go to stderr in youtube-dl

Only when logger is not used. See this example for how an logger can be specified.

Are you saying you still want me to do

Yes, just without the first underscore (_self => self)

Per @yan12125, traceback.print_exc() can send output to the wrong
place under some circumstances (e.g. use of logger), so send it to
stderr. Which requires a compat_str.
@johnhawkinson
Copy link
Contributor Author

OK, thanks, done!

Copy link
Collaborator

@yan12125 yan12125 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically this looks good. As it's not a clean solution, I'd like to wait for a few days to see whether there's objection or not.

@yan12125
Copy link
Collaborator

Hmm the first exception is not shown with this patch. Anything wrong?

$ youtube-dl -vo - http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/ > t2.mp4        
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['-vo', '-', 'http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/']
[debug] Encodings: locale UTF-8, fs utf-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.10.12
[debug] Git HEAD: 4102e64
[debug] Python version 3.5.2 - Linux-4.8.1-1-ARCH-x86_64-with-arch
[debug] exe versions: ffmpeg 3.1.4, ffprobe 3.1.4, rtmpdump 2.4
[debug] Proxy map: {}
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading XML
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading Downloadable SMIL data
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Checking video URL
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: video URL is invalid, skipping
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading HLS SMIL data
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
WARNING: Failed to download m3u8 information: HTTP Error 403: Forbidden
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading 3G SMIL data
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Checking video URL
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Checking video URL
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading RTMP SMIL data
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading StreamPack SMIL data
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading m3u8 information
[CBS] 4BB655AB-42C0-9322-9C18-A1600750A7D5: Downloading JSON metadata
[debug] Invoking downloader on 'https://cbsios-vh.akamaihd.net/i/temp_hd_gallery_video/CBS_Production_Outlet_VMS/video_robot/CBS_Production_Entertainment/2016/10/07/781612611729/CBS_COLBERT_0226_CLIP1_ROMANTIC_CIAN_934987_,2596,3596,1848,1248,848,548,298,158,000.mp4.csmil/index_1_av.m3u8?null=0&id=AgCMcObYoxwFN9%2fTAViES7MMOx5ikaEYBxOclZQE%2f+OXtUNtooL2DaflTtJiyzpar+ihD5tJwdxMkA%3d%3d'
[download] Destination: -
[debug] ffmpeg command line: ffmpeg -y -headers 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0) Gecko/20150101 Firefox/47.0 (Chrome)
Cookie: _alid_=icGONENR8YCN5yt9OM7a/A==; hdntl=exp=1476601183~acl=%2fi%2ftemp_hd_gallery_video%2fCBS_Production_Outlet_VMS%2fvideo_robot%2fCBS_Production_Entertainment%2f2016%2f10%2f07%2f781612611729%2fCBS_COLBERT_0226_CLIP1_ROMANTIC_CIAN_934987_*~data=hdntl~hmac=0b7f0c204acf8fa482ea4ee874dd086f788b04d1a0f2f4c1c571ea0ea412c761
Accept-Encoding: gzip, deflate
Accept-Language: en-us,en;q=0.5
' -i 'https://cbsios-vh.akamaihd.net/i/temp_hd_gallery_video/CBS_Production_Outlet_VMS/video_robot/CBS_Production_Entertainment/2016/10/07/781612611729/CBS_COLBERT_0226_CLIP1_ROMANTIC_CIAN_934987_,2596,3596,1848,1248,848,548,298,158,000.mp4.csmil/index_1_av.m3u8?null=0&id=AgCMcObYoxwFN9%2fTAViES7MMOx5ikaEYBxOclZQE%2f+OXtUNtooL2DaflTtJiyzpar+ihD5tJwdxMkA%3d%3d' -c copy -f mpegts -
ffmpeg version 3.1.4 Copyright (c) 2000-2016 the FFmpeg developers
  built with gcc 6.2.1 (GCC) 20160830
  configuration: --prefix=/usr --disable-debug --disable-static --disable-stripping --enable-avisynth --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-netcdf --enable-shared --enable-version3 --enable-x11grab
  libavutil      55. 28.100 / 55. 28.100
  libavcodec     57. 48.101 / 57. 48.101
  libavformat    57. 41.100 / 57. 41.100
  libavdevice    57.  0.101 / 57.  0.101
  libavfilter     6. 47.100 /  6. 47.100
  libavresample   3.  0.  0 /  3.  0.  0
  libswscale      4.  1.100 /  4.  1.100
  libswresample   2.  1.100 /  2.  1.100
  libpostproc    54.  0.100 / 54.  0.100
Input #0, hls,applehttp, from 'https://cbsios-vh.akamaihd.net/i/temp_hd_gallery_video/CBS_Production_Outlet_VMS/video_robot/CBS_Production_Entertainment/2016/10/07/781612611729/CBS_COLBERT_0226_CLIP1_ROMANTIC_CIAN_934987_,2596,3596,1848,1248,848,548,298,158,000.mp4.csmil/index_1_av.m3u8?null=0&id=AgCMcObYoxwFN9%2fTAViES7MMOx5ikaEYBxOclZQE%2f+OXtUNtooL2DaflTtJiyzpar+ihD5tJwdxMkA%3d%3d':
  Duration: 00:01:35.12, start: 0.100511, bitrate: 0 kb/s
  Program 0 
    Metadata:
      variant_bitrate : 0
    Stream #0:0: Video: h264 (Main) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, fltp, 107 kb/s
[mpegts @ 0x558f9cfff920] Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.                                                                                                 
    Last message repeated 1 times                                                                               
Output #0, mpegts, to 'pipe:':
  Metadata:
    encoder         : Lavf57.41.100
    Stream #0:0: Video: h264 ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 29.97 fps, 29.97 tbr, 90k tbn, 90k tbc
    Stream #0:1: Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, stereo, 107 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #0:1 -> #0:1 (copy)
Press [q] to stop, [?] for help
frame= 2850 fps=445 q=-1.0 Lsize=   45286kB time=00:01:35.13 bitrate=3899.7kbits/s speed=14.9x    
video:40525kB audio:1184kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.576104%
Exception ignored in: <_io.FileIO name=5 mode='wb' closefd=True>
ResourceWarning: unclosed file <_io.BufferedWriter name=5>
ERROR: unable to download video data: [Errno 2] No such file or directory: '-'
Traceback (most recent call last):
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/common.py", line 353, in download
    return self.real_download(filename, info_dict)
  File "/home/yen/Projects/youtube-dl/youtube_dl/downloader/external.py", line 30, in real_download
    fsize = os.path.getsize(encodeFilename(tmpfilename))
  File "/usr/lib/python3.5/genericpath.py", line 50, in getsize
    return os.stat(filename).st_size
FileNotFoundError: [Errno 2] No such file or directory: '-'

@johnhawkinson
Copy link
Contributor Author

Hmm the first exception is not shown with this patch. Anything wrong?

Err, it works for me. Did you mean to run it out of the build tree instead of the PATH, i.e.

python -m youtube_dl -vo - http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/ > t2.mp4

instead of

youtube-dl -vo - http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/ > t2.mp4

?

I get:

pb3:youtube-dl jhawk$ python -m youtube_dl -vo - http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/ > t2.mp4
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-vo', u'-', u'http://www.cbs.com/shows/the-late-show-with-stephen-colbert/video/4BB655AB-42C0-9322-9C18-A1600750A7D5/diane-lane-s-life-is-one-long-romantic-movie/']
[debug] Encodings: locale UTF-8, fs utf-8, out None, pref UTF-8
[debug] youtube-dl version 2016.10.07
[debug] Git HEAD: 9d92ac8
[debug] Python version 2.7.10 - Darwin-14.5.0-x86_64-i386-64bit
...
frame= 2850 fps=253 q=-1.0 Lsize=   45286kB time=00:01:35.13 bitrate=3899.7kbits/s speed=8.45x    
video:40525kB audio:1184kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 8.576104%
Traceback (most recent call last):
  File "youtube_dl/YoutubeDL.py", line 1699, in process_info
    success = dl(filename, info_dict)
  File "youtube_dl/YoutubeDL.py", line 1641, in dl
    return fd.download(name, info)
  File "youtube_dl/downloader/common.py", line 354, in download
    return self.real_download(filename, info_dict)
  File "youtube_dl/downloader/external.py", line 30, in real_download
    fsize = os.path.getsize(encodeFilename(tmpfilename))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.py", line 57, in getsize
    return os.stat(filename).st_size
OSError: [Errno 2] No such file or directory: '-'

ERROR: unable to download video
Traceback (most recent call last):
  File "youtube_dl/YoutubeDL.py", line 1792, in download
    url, force_generic_extractor=self.params.get('force_generic_extractor', False))
  File "youtube_dl/YoutubeDL.py", line 705, in extract_info
    return self.process_ie_result(ie_result, download, extra_info)
  File "youtube_dl/YoutubeDL.py", line 751, in process_ie_result
    return self.process_video_result(ie_result, download=download)
  File "youtube_dl/YoutubeDL.py", line 1435, in process_video_result
    self.process_info(new_info)
  File "youtube_dl/YoutubeDL.py", line 1705, in process_info
    raise UnavailableVideoError(err)
UnavailableVideoError: [Errno 2] No such file or directory: '-'

Ah yes, your run was:

[debug] Git HEAD: 4102e64

pb3:youtube-dl jhawk$ git show 4102e64
commit 4102e64051e6949f2618f9a621641506c13a6163
Merge: 77c5b98 f605242
Author: Yen Chi Hsuan <yan12125@gmail.com>
Date:   Sat Oct 15 14:56:00 2016 +0800

    Merge branch 'pyx-huajiao'

pb3:youtube-dl jhawk$ 

@yan12125
Copy link
Collaborator

I apply this patch with a commit, so the commit ID does not change. Let me check it out more.

@johnhawkinson
Copy link
Contributor Author

Looks like a python3 issue. You confused me when you wrote:

Hmm the first exception is not shown with this patch.

But python3 shows the inner traceback but not the outer one, but I think that's the more useful one to show. So it looks like this patch is only relevant for python2. Still worth doing though.

@wader
Copy link
Contributor

wader commented Dec 30, 2017

Hello! what is needed to get this and #10876 merged?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defunct PR source branch is not accessible
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants