.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -18,7 +18,7 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.21.1. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.19. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -26,7 +26,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **2020.11.21.1**
+- [ ] I've verified that I'm running youtube-dl version **2020.11.19**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -41,7 +41,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2020.11.21.1
+ [debug] youtube-dl version 2020.11.19
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -19,7 +19,7 @@ labels: 'site-support-request'
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.21.1. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.19. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **2020.11.21.1**
+- [ ] I've verified that I'm running youtube-dl version **2020.11.19**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -18,13 +18,13 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.21.1. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.19. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **2020.11.21.1**
+- [ ] I've verified that I'm running youtube-dl version **2020.11.19**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -18,7 +18,7 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.21.1. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.19. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **2020.11.21.1**
+- [ ] I've verified that I'm running youtube-dl version **2020.11.19**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -43,7 +43,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2020.11.21.1
+ [debug] youtube-dl version 2020.11.19
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -19,13 +19,13 @@ labels: 'request'
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.21.1. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.11.19. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **2020.11.21.1**
+- [ ] I've verified that I'm running youtube-dl version **2020.11.19**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

ChangeLog

@@ -1,34 +1,3 @@
-version 2020.11.21.1
-
-Core
-* [downloader/http] Fix crash during urlopen caused by missing reason
-  of URLError
-* [YoutubeDL] Fix --ignore-errors for playlists with generator-based entries
-  of url_transparent (#27064)
-
-Extractors
-+ [svtplay] Add support for svt.se/barnkanalen (#24817)
-+ [svt] Extract timestamp (#27130)
-* [svtplay] Improve thumbnail extraction (#27130)
-* [youtube] Fix error reason extraction (#27081)
-* [youtube] Fix like and dislike count extraction (#25977)
-+ [youtube:tab] Add support for current video and fix lives extraction (#27126)
-* [infoq] Fix format extraction (#25984)
-* [francetv] Update to fix thumbnail URL issue (#27120)
-* [youtube] Improve yt initial data extraction (#27093)
-+ [discoverynetworks] Add support new TLC/DMAX URLs (#27100)
-* [rai] Fix protocol relative relinker URLs (#22766)
-* [rai] Fix unavailable video format detection
-* [rai] Improve extraction
-* [rai] Fix extraction (#27077)
-* [viki] Improve format extraction
-* [viki] Fix stream extraction from MPD (#27092)
-* [googledrive] Fix format extraction (#26979)
-+ [amara] Add support for amara.org (#20618)
-* [vimeo:album] Fix extraction (#27079)
-* [mtv] Fix mgid extraction (#26841)
-
-
 version 2020.11.19
 
 Core

docs/supportedsites.md

@@ -41,7 +41,6 @@
  - **AlJazeera**
  - **Allocine**
  - **AlphaPorno**
- - **Amara**
  - **AMCNetworks**
  - **AmericasTestKitchen**
  - **anderetijden**: npo.nl, ntr.nl, omroepwnl.nl, zapp.nl and npo3.nl
@@ -1131,6 +1130,7 @@
  - **YourUpload**
  - **youtube**: YouTube.com
  - **youtube:history**: Youtube watch history, ":ythistory" for short (requires authentication)
+ - **youtube:live**: YouTube.com live streams
  - **youtube:playlist**: YouTube.com playlists
  - **youtube:recommended**: YouTube.com recommended videos, ":ytrec" for short (requires authentication)
  - **youtube:search**: YouTube.com searches

youtube_dl/extractor/extractors.py

@@ -844,10 +844,6 @@ from .picarto import (
 )
 from .piksel import PikselIE
 from .pinkbike import PinkbikeIE
-from .pinterest import (
-    PinterestIE,
-    PinterestCollectionIE,
-)
 from .pladform import PladformIE
 from .platzi import (
     PlatziIE,

youtube_dl/extractor/pinterest.py

@@ -1,176 +0,0 @@
-# coding: utf-8
-from __future__ import unicode_literals
-
-from .common import InfoExtractor
-from ..compat import compat_str
-from ..utils import (
-    determine_ext,
-    float_or_none,
-    int_or_none,
-    try_get,
-    unified_timestamp,
-    url_or_none,
-)
-
-
-class PinterestBaseIE(InfoExtractor):
-    _VALID_URL_BASE = r'https?://(?:[^/]+\.)?pinterest\.(?:com|fr|de|ch|jp|cl|ca|it|co\.uk|nz|ru|com\.au|at|pt|co\.kr|es|com\.mx|dk|ph|th|com\.uy|co|nl|info|kr|ie|vn|com\.vn|ec|mx|in|pe|co\.at|hu|co\.in|co\.nz|id|com\.ec|com\.py|tw|be|uk|com\.bo|com\.pe)'
-
-    def _extract_resource(self, webpage, video_id):
-        return self._parse_json(
-            self._search_regex(
-                r'<script[^>]+\bid=["\']initial-state["\'][^>]*>({.+?})</script>',
-                webpage, 'application json'),
-            video_id)['resourceResponses']
-
-    def _extract_video(self, data, extract_formats=True):
-        video_id = data['id']
-
-        title = (data.get('title') or data.get('grid_title') or video_id).strip()
-
-        formats = []
-        duration = None
-        if extract_formats:
-            for format_id, format_dict in data['videos']['video_list'].items():
-                if not isinstance(format_dict, dict):
-                    continue
-                format_url = url_or_none(format_dict.get('url'))
-                if not format_url:
-                    continue
-                duration = float_or_none(format_dict.get('duration'), scale=1000)
-                ext = determine_ext(format_url)
-                if 'hls' in format_id.lower() or ext == 'm3u8':
-                    formats.extend(self._extract_m3u8_formats(
-                        format_url, video_id, 'mp4', entry_protocol='m3u8_native',
-                        m3u8_id=format_id, fatal=False))
-                else:
-                    formats.append({
-                        'url': format_url,
-                        'format_id': format_id,
-                        'width': int_or_none(format_dict.get('width')),
-                        'height': int_or_none(format_dict.get('height')),
-                        'duration': duration,
-                    })
-            self._sort_formats(
-                formats, field_preference=('height', 'width', 'tbr', 'format_id'))
-
-        description = data.get('description') or data.get('description_html') or data.get('seo_description')
-        timestamp = unified_timestamp(data.get('created_at'))
-
-        def _u(field):
-            return try_get(data, lambda x: x['closeup_attribution'][field], compat_str)
-
-        uploader = _u('full_name')
-        uploader_id = _u('id')
-
-        repost_count = int_or_none(data.get('repin_count'))
-        comment_count = int_or_none(data.get('comment_count'))
-        categories = try_get(data, lambda x: x['pin_join']['visual_annotation'], list)
-        tags = data.get('hashtags')
-
-        thumbnails = []
-        images = data.get('images')
-        if isinstance(images, dict):
-            for thumbnail_id, thumbnail in images.items():
-                if not isinstance(thumbnail, dict):
-                    continue
-                thumbnail_url = url_or_none(thumbnail.get('url'))
-                if not thumbnail_url:
-                    continue
-                thumbnails.append({
-                    'url': thumbnail_url,
-                    'width': int_or_none(thumbnail.get('width')),
-                    'height': int_or_none(thumbnail.get('height')),
-                })
-
-        return {
-            'id': video_id,
-            'title': title,
-            'description': description,
-            'duration': duration,
-            'timestamp': timestamp,
-            'thumbnails': thumbnails,
-            'uploader': uploader,
-            'uploader_id': uploader_id,
-            'repost_count': repost_count,
-            'comment_count': comment_count,
-            'categories': categories,
-            'tags': tags,
-            'formats': formats,
-            'extractor_key': PinterestIE.ie_key(),
-        }
-
-
-class PinterestIE(PinterestBaseIE):
-    _VALID_URL = r'%s/pin/(?P<id>\d+)' % PinterestBaseIE._VALID_URL_BASE
-    _TESTS = [{
-        'url': 'https://www.pinterest.com/pin/664281013778109217/',
-        'md5': '6550c2af85d6d9f3fe3b88954d1577fc',
-        'info_dict': {
-            'id': '664281013778109217',
-            'ext': 'mp4',
-            'title': 'Origami',
-            'description': 'md5:b9d90ddf7848e897882de9e73344f7dd',
-            'duration': 57.7,
-            'timestamp': 1593073622,
-            'upload_date': '20200625',
-            'uploader': 'Love origami -I am Dafei',
-            'uploader_id': '586523688879454212',
-            'repost_count': 50,
-            'comment_count': 0,
-            'categories': list,
-            'tags': list,
-        },
-    }, {
-        'url': 'https://co.pinterest.com/pin/824721750502199491/',
-        'only_matching': True,
-    }]
-
-    def _real_extract(self, url):
-        video_id = self._match_id(url)
-        webpage = self._download_webpage(url, video_id)
-        data = self._extract_resource(webpage, video_id)[0]['response']['data']
-        return self._extract_video(data)
-
-
-class PinterestCollectionIE(PinterestBaseIE):
-    _VALID_URL = r'%s/[^/]+/(?P<id>[^/?#&]+)' % PinterestBaseIE._VALID_URL_BASE
-    _TESTS = [{
-        'url': 'https://www.pinterest.ca/mashal0407/cool-diys/',
-        'info_dict': {
-            'id': '585890301462791043',
-            'title': 'cool diys',
-        },
-        'playlist_count': 8,
-    }]
-
-    @classmethod
-    def suitable(cls, url):
-        return False if PinterestIE.suitable(url) else super(
-            PinterestCollectionIE, cls).suitable(url)
-
-    def _real_extract(self, url):
-        collection_name = self._match_id(url)
-        webpage = self._download_webpage(url, collection_name)
-        resource = self._extract_resource(webpage, collection_name)[1]
-        entries = []
-        for item in resource['response']['data']:
-            if not isinstance(item, dict) or item.get('type') != 'pin':
-                continue
-            video_id = item.get('id')
-            if video_id:
-                # Some pins may not be available anonymously via pin URL
-                # video = self._extract_video(item, extract_formats=False)
-                # video.update({
-                #     '_type': 'url_transparent',
-                #     'url': 'https://www.pinterest.com/pin/%s/' % video_id,
-                # })
-                # entries.append(video)
-                entries.append(self._extract_video(item))
-        title = try_get(
-            resource, lambda x: x['options']['board_title'], compat_str)
-        collection_id = try_get(
-            resource, lambda x: x['options']['board_id'],
-            compat_str) or collection_name
-        return self.playlist_result(
-            entries, playlist_id=collection_id, playlist_title=title)

youtube_dl/extractor/svt.py

@@ -9,7 +9,6 @@ from ..utils import (
     determine_ext,
     dict_get,
     int_or_none,
-    unified_timestamp,
     str_or_none,
     strip_or_none,
     try_get,
@@ -45,8 +44,7 @@ class SVTBaseIE(InfoExtractor):
                     'format_id': player_type,
                     'url': vurl,
                 })
-        rights = try_get(video_info, lambda x: x['rights'], dict) or {}
-        if not formats and rights.get('geoBlockedSweden'):
+        if not formats and video_info.get('rights', {}).get('geoBlockedSweden'):
             self.raise_geo_restricted(
                 'This video is only available in Sweden',
                 countries=self._GEO_COUNTRIES)
@@ -72,7 +70,6 @@ class SVTBaseIE(InfoExtractor):
         episode = video_info.get('episodeTitle')
         episode_number = int_or_none(video_info.get('episodeNumber'))
 
-        timestamp = unified_timestamp(rights.get('validFrom'))
         duration = int_or_none(dict_get(video_info, ('materialLength', 'contentDuration')))
         age_limit = None
         adult = dict_get(
@@ -87,7 +84,6 @@ class SVTBaseIE(InfoExtractor):
             'formats': formats,
             'subtitles': subtitles,
             'duration': duration,
-            'timestamp': timestamp,
             'age_limit': age_limit,
             'series': series,
             'season_number': season_number,
@@ -140,39 +136,26 @@ class SVTPlayIE(SVTPlayBaseIE):
     IE_DESC = 'SVT Play and Öppet arkiv'
     _VALID_URL = r'''(?x)
                     (?:
-                        (?:
-                            svt:|
-                            https?://(?:www\.)?svt\.se/barnkanalen/barnplay/[^/]+/
-                        )
-                        (?P<svt_id>[^/?#&]+)|
+                        svt:(?P<svt_id>[^/?#&]+)|
                         https?://(?:www\.)?(?:svtplay|oppetarkiv)\.se/(?:video|klipp|kanaler)/(?P<id>[^/?#&]+)
                     )
                     '''
     _TESTS = [{
-        'url': 'https://www.svtplay.se/video/26194546/det-har-ar-himlen',
-        'md5': '2382036fd6f8c994856c323fe51c426e',
+        'url': 'http://www.svtplay.se/video/5996901/flygplan-till-haile-selassie/flygplan-till-haile-selassie-2',
+        'md5': '2b6704fe4a28801e1a098bbf3c5ac611',
         'info_dict': {
-            'id': 'jNwpV9P',
+            'id': '5996901',
             'ext': 'mp4',
-            'title': 'Det här är himlen',
-            'timestamp': 1586044800,
-            'upload_date': '20200405',
-            'duration': 3515,
-            'thumbnail': r're:^https?://(?:.*[\.-]jpg|www.svtstatic.se/image/.*)$',
+            'title': 'Flygplan till Haile Selassie',
+            'duration': 3527,
+            'thumbnail': r're:^https?://.*[\.-]jpg$',
             'age_limit': 0,
             'subtitles': {
                 'sv': [{
-                    'ext': 'vtt',
+                    'ext': 'wsrt',
                 }]
             },
         },
-        'params': {
-            'format': 'bestvideo',
-            # skip for now due to download test asserts that segment is > 10000 bytes and svt uses
-            # init segments that are smaller
-            # AssertionError: Expected test_SVTPlay_jNwpV9P.mp4 to be at least 9.77KiB, but it's only 864.00B
-            'skip_download': True,
-        },
     }, {
         # geo restricted to Sweden
         'url': 'http://www.oppetarkiv.se/video/5219710/trollflojten',
@@ -189,12 +172,6 @@ class SVTPlayIE(SVTPlayBaseIE):
     }, {
         'url': 'svt:14278044',
         'only_matching': True,
-    }, {
-        'url': 'https://www.svt.se/barnkanalen/barnplay/kar/eWv5MLX/',
-        'only_matching': True,
-    }, {
-        'url': 'svt:eWv5MLX',
-        'only_matching': True,
     }]
 
     def _adjust_title(self, info):
@@ -259,10 +236,7 @@ class SVTPlayIE(SVTPlayBaseIE):
                  r'["\']svtId["\']\s*:\s*["\']([\da-zA-Z-]+)'),
                 webpage, 'video id')
 
-        info_dict = self._extract_by_video_id(svt_id, webpage)
-        info_dict['thumbnail'] = thumbnail
-
-        return info_dict
+        return self._extract_by_video_id(svt_id, webpage)
 
 
 class SVTSeriesIE(SVTPlayBaseIE):
@@ -386,7 +360,7 @@ class SVTPageIE(InfoExtractor):
 
     @classmethod
     def suitable(cls, url):
-        return False if SVTIE.suitable(url) or SVTPlayIE.suitable(url) else super(SVTPageIE, cls).suitable(url)
+        return False if SVTIE.suitable(url) else super(SVTPageIE, cls).suitable(url)
 
     def _real_extract(self, url):
         path, display_id = re.match(self._VALID_URL, url).groups()

youtube_dl/extractor/youtube.py

@@ -2620,13 +2620,10 @@ class YoutubeTabIE(YoutubeBaseInfoExtractor):
     }, {
         'url': 'https://www.youtube.com/c/CommanderVideoHq/live',
         'only_matching': True,
-    },
-    # TODO
-    # {
-    #     'url': 'https://www.youtube.com/TheYoungTurks/live',
-    #     'only_matching': True,
-    # }
-    ]
+    }, {
+        'url': 'https://www.youtube.com/TheYoungTurks/live',
+        'only_matching': True,
+    }]
 
     def _extract_channel_id(self, webpage):
         channel_id = self._html_search_meta(

youtube_dl/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2020.11.21.1'
+__version__ = '2020.11.19'