National Geographic url not recognized / [generic] / Unsupported URL

[BlohoJo]

Checklist

• I'm reporting a broken site support
• I've verified that I'm running youtube-dl version 2019.10.22
• I've checked that all provided URLs are alive and playable in a browser
• I've checked that all URLs and arguments with special characters are properly quoted or escaped
• I've searched the bugtracker for similar issues including closed ones

Verbose log

E:\USER\temp_ytdl>youtube-dl.exe --all-subs --cookies cookies.txt --verbose "htt
ps://www.nationalgeographic.com/tv/shows/dr-t-lone-star-vet/episode-guide/season
-01/episode-02-mother-of-dragons/vdka13590590"
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--all-subs', '--cookies', 'cookies.txt', '--verbose
', 'https://www.nationalgeographic.com/tv/shows/dr-t-lone-star-vet/episode-guide
/season-01/episode-02-mother-of-dragons/vdka13590590']
[debug] Encodings: locale cp1252, fs mbcs, out cp437, pref cp1252
[debug] youtube-dl version 2019.10.22
[debug] Python version 3.4.4 (CPython) - Windows-2008ServerR2-6.1.7601-SP1
[debug] exe versions: ffmpeg 4.1.3, ffprobe N-89595-g40d4b13228, rtmpdump 2.4-20
151223-gfa8646d-OpenSSL_1.0.2n-x86_64-static
[debug] Proxy map: {}
[generic] vdka13590590: Requesting header
WARNING: Falling back on generic information extractor.
[generic] vdka13590590: Downloading webpage
[generic] vdka13590590: Extracting information
ERROR: Unsupported URL: https://www.nationalgeographic.com/tv/shows/dr-t-lone-st
ar-vet/episode-guide/season-01/episode-02-mother-of-dragons/vdka13590590
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpifszpy
0p\build\youtube_dl\YoutubeDL.py", line 796, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpifszpy
0p\build\youtube_dl\extractor\common.py", line 530, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\ytdl-org\tmpifszpy
0p\build\youtube_dl\extractor\generic.py", line 3353, in _real_extract
youtube_dl.utils.UnsupportedError: Unsupported URL: https://www.nationalgeograph
ic.com/tv/shows/dr-t-lone-star-vet/episode-guide/season-01/episode-02-mother-of-
dragons/vdka13590590

Description

My understanding is National Geographic is supported. This was reported in January, but it was immediately closed as "Duplicate" and I cannot find what it's a duplicate of. It appears to be the same issue. I was therefor unable to find if there is any kind of workaround.

[tmthywynn8]

This looks identical to the scheme used on Disney's Go platform. I'm sure adding yet another brand to it is relatively trivial—I may give it a shot myself to see if it's doable, as I amateurishly fixed a related issue, but as I'm not actually putting a pull request in, who knows how long the fix will take to be put into an official release.

Update: Yep, just as trivial as I expected, though looks as if the Adobe pass credentials break for me, but it should get you started. What I did:

1. Add another brand to the dictionary or whatever it's called in go.py. Between lines 40-41, paste the following:

        },
       'nationalgeographic': {
           'brand': '024',
           'requestor_id': 'dtci',
   

2. Change VALID_URL, such that it reads:
   _VALID_URL = r'https?://(?:www\.)?(?:(?:(?P<sub_domain>%s)(?:\.go)?)?|(?P<sub_domain_2>disneynow))\.com/(?:(?:[^/]+/)*(?P<id>vdka\w+)|(?:[^/]+/)*(?P<display_id>[^/?#]+))'\
3. In the _real_extract section, change the regexp for video_id to read:
   r'"id":["\']*(VDKA\w+)', webpage, 'video id',

[tv21]

I'm seeing this issue also. And they seem to do things differently from others covered by go.py, so if there is some kind of workaround I would really appreciate hearing about it, even if it involves a few steps. Unfortunately tmthywynn8's response isn't too helpful because I looked in go.py and while I could easily make the addition in his point 1, I'm mystified as to what he added and where he added it in his points 2 and 3. Maybe the go.py app has been changed significantly since his post but I could not for the life of me figure out how to do either of those things, otherwise I would have tested it and reported back.

[silverbullettruck2001]

@tmthywynn8, I made your suggested changes, but I am still getting an error. I did make one change in your suggestion in regards to step number 2, I put 'nationalgeographic' instead of 'disneynow' so that it would get picked up correctly. Any help would be appreciated. Here is error I am getting now:

[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'--ap-mso', u'Charter_Direct', u'--ap-username', u'PRIVATE', u'--ap-password', u'PRIVATE', u'https://www.nationalgeographic.com/tv/shows/running-wild-with-bear-grylls/episode-guide/season-05/episode-09-bobby-bones-in-norway-fjords/vdka15497495', u'--verbose']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: 22cb949
[debug] Python version 2.7.16 (CPython) - Linux-5.0.0-38-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.2-static, ffprobe 4.2-static
[debug] Proxy map: {}
[Go] vdka15497495: Downloading JSON metadata
ERROR: An extractor error has occurred. (caused by KeyError(u'video',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
File "youtube_dl/extractor/common.py", line 530, in extract
ie_result = self._real_extract(url)
File "youtube_dl/extractor/go.py", line 170, in _real_extract
video_data = self._extract_videos(brand, video_id)[0]
File "youtube_dl/extractor/go.py", line 139, in _extract_videos
display_id)['video']
KeyError: u'video'
Traceback (most recent call last):
File "youtube_dl/YoutubeDL.py", line 796, in extract_info
ie_result = ie.extract(url)
File "youtube_dl/extractor/common.py", line 543, in extract
raise ExtractorError('An extractor error has occurred.', cause=e)
ExtractorError: An extractor error has occurred. (caused by KeyError(u'video',)); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; see https://yt-dl.org/update on how to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

[silverbullettruck2001]

@tmthywynn8 are you able to assist? Anyone else able to provide suggestions on how to resolve this?

[tmthywynn8]

Actually since my comment, thanks to this commit by @dstftw, the third step is no longer needed.

Using this episode as the example though, looks like one needs to:

Add two brands to the dictionary or whatever it's called in go.py. Between lines 40-41, paste the following:

        },
        'nationalgeographic': {
            'brand': '024',
            'requestor_id': 'dtci',
        },
        'natgeo': {
            'brand': '026',
            'requestor_id': 'dtci',

Note: I don't know which of the two are actually needed now.

Then, on the line reading (?P<sub_domain_2>, add |nationalgeographic to it.

There's an issue with retrieving the video ID in the code, but I haven't looked at the logic for that yet. Assume it's because they once again changed the layout to something like: "videoIdCode":"vdka15374108" which probably hasn't been accounted for.

[silverbullettruck2001]

@tmthywynn8 have you found anything about the layout? I would be interested in finding out more about the layout and how to troubleshoot this if you want to contact me you can find my email address on my profile. Thank you for your assistance in this. I look forward to hearing from you.

[silverbullettruck2001]

@tmthywynn8 do you have any suggestions?

[Zachary24]

any progress?

[silverbullettruck2001]

@tmthywynn8 would you be able to look at the layout again and see how this could be resolved?

[tmthywynn8]

Whoops, wrong issue. My apologies for the false hope. No, haven't had the time. While adding in national Geographic to the dictionary and accounting for the www addition in the regular expression, there are still issues with obtaining metadata information, probably because (1) it has its own legacy information for doing so, or because (2) there are different logic paths being taken if it's ABC or Freeform vs. everything else, and it's going for the latter which probably won't work—haven't looked into the way NG presents its information. I'm not a developer—just somewhat decent at regexp, so would need to conceptualize the logic in go.py first before adapting it for NG (if necessary).

[jamakal]

I managed to get this working with the following patch:

Add the following brand to the _SITE_INFO dictionary in go.py:

'nationalgeographic': {
    'brand': '026', 
    'requestor_id': 'dtci',
}

And replace the following line:

(?P<sub_domain_2>abc|freeform|disneynow)

With:

(?:www\.)?(?P<sub_domain_2>abc|freeform|disneynow|nationalgeographic)

[tv21]

JamKage any possibility this might work for FX? Please see #23754 which might be a very similar issue.

[senior777]

@JamKage, any progress on the National Geographic #26785? It's still not able to be downloaded by YouTube-dl; as it was flagged to have "conflicts that must be resolved" before it is the "PR is merged" as well. Your assistance on it will be very much appreciated, just as the Fxnetworks that is now running fine. Thanks.

[nobodyspecial]

Can someone take a look at this? National Geographic is still not working after all this time.

[xxtensazenxx]

I wish I could help with this. NatGeo support would be a great feature