Update extractor for current site behaviour

.github/ISSUE_TEMPLATE/1_broken_site.md

@@ -18,7 +18,7 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.06.06. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -26,7 +26,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
+- [ ] I've verified that I'm running youtube-dl version **2021.06.06**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar issues including closed ones
@@ -41,7 +41,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2021.12.17
+ [debug] youtube-dl version 2021.06.06
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/2_site_support_request.md

@@ -19,7 +19,7 @@ labels: 'site-support-request'
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.06.06. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that site you are requesting is not dedicated to copyright infringement, see https://yt-dl.org/copyright-infringement. youtube-dl does not support such sites. In order for site support request to be accepted all provided example URLs should not violate any copyrights.
 - Search the bugtracker for similar site support requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a new site support request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
+- [ ] I've verified that I'm running youtube-dl version **2021.06.06**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that none of provided URLs violate any copyrights
 - [ ] I've searched the bugtracker for similar site support requests including closed ones

.github/ISSUE_TEMPLATE/3_site_feature_request.md

@@ -18,13 +18,13 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.06.06. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar site feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 
 - [ ] I'm reporting a site feature request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
+- [ ] I've verified that I'm running youtube-dl version **2021.06.06**
 - [ ] I've searched the bugtracker for similar site feature requests including closed ones
 
 

.github/ISSUE_TEMPLATE/4_bug_report.md

@@ -18,7 +18,7 @@ title: ''
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.06.06. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
 - Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
 - Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
@@ -27,7 +27,7 @@ Carefully read and work through this check list in order to prevent the most com
 -->
 
 - [ ] I'm reporting a broken site support issue
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
+- [ ] I've verified that I'm running youtube-dl version **2021.06.06**
 - [ ] I've checked that all provided URLs are alive and playable in a browser
 - [ ] I've checked that all URLs and arguments with special characters are properly quoted or escaped
 - [ ] I've searched the bugtracker for similar bug reports including closed ones
@@ -43,7 +43,7 @@ Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <
  [debug] User config: []
  [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
  [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
- [debug] youtube-dl version 2021.12.17
+ [debug] youtube-dl version 2021.06.06
  [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
  [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
  [debug] Proxy map: {}

.github/ISSUE_TEMPLATE/5_feature_request.md

@@ -19,13 +19,13 @@ labels: 'request'
 
 <!--
 Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
-- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.12.17. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
+- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2021.06.06. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
 - Search the bugtracker for similar feature requests: http://yt-dl.org/search-issues. DO NOT post duplicates.
 - Finally, put x into all relevant boxes (like this [x])
 -->
 
 - [ ] I'm reporting a feature request
-- [ ] I've verified that I'm running youtube-dl version **2021.12.17**
+- [ ] I've verified that I'm running youtube-dl version **2021.06.06**
 - [ ] I've searched the bugtracker for similar feature requests including closed ones
 
 

ChangeLog

@@ -1,28 +1,3 @@
-version 2021.12.17
-
-Core
-* [postprocessor/ffmpeg] Show ffmpeg output on error (#22680, #29336)
-
-Extractors
-* [youtube] Update signature function patterns (#30363, #30366)
-* [peertube] Only call description endpoint if necessary (#29383)
-* [periscope] Pass referer to HLS requests (#29419)
-- [liveleak] Remove extractor (#17625, #24222, #29331)
-+ [pornhub] Add support for pornhubthbh7ap3u.onion
-* [pornhub] Detect geo restriction
-* [pornhub] Dismiss tbr extracted from download URLs (#28927)
-* [curiositystream:collection] Extend _VALID_URL (#26326, #29117)
-* [youtube] Make get_video_info processing more robust (#29333)
-* [youtube] Workaround for get_video_info request (#29333)
-* [bilibili] Strip uploader name (#29202)
-* [youtube] Update invidious instance list (#29281)
-* [umg:de] Update GraphQL API URL (#29304)
-* [nrk] Switch psapi URL to https (#29344)
-+ [egghead] Add support for app.egghead.io (#28404, #29303)
-* [appleconnect] Fix extraction (#29208)
-+ [orf:tvthek] Add support for MPD formats (#28672, #29236)
-
-
 version 2021.06.06
 
 Extractors

docs/supportedsites.md

@@ -472,6 +472,8 @@
  - **LinuxAcademy**
  - **LiTV**
  - **LiveJournal**
+ - **LiveLeak**
+ - **LiveLeakEmbed**
  - **livestream**
  - **livestream:original**
  - **LnkGo**

youtube_dl/extractor/extractors.py

@@ -1484,6 +1484,7 @@ from .voxmedia import (
     VoxMediaVolumeIE,
     VoxMediaIE,
 )
+from .voe import VOEIE
 from .vrt import VRTIE
 from .vrak import VrakIE
 from .vrv import (

youtube_dl/extractor/voe.py

@@ -0,0 +1,77 @@
+# coding: utf-8
+from __future__ import unicode_literals
+
+import base64
+
+from .common import InfoExtractor
+from ..utils import (
+    int_or_none,
+    js_to_json,
+    url_or_none,
+)
+
+
+class VOEIE(InfoExtractor):
+    IE_NAME = 'voe'
+    IE_DESC = 'VOE.SX'
+    _VALID_URL = r'https?://voe\.sx(?P<path>/(e/)?(?P<id>[a-z0-9]+))'
+    _TESTS = [{
+        'url': 'https://voe.sx/e/ng7ja5n5n2y8',
+        'info_dict': {
+            'id': 'ng7ja5n5n2y8',
+            'title': 'md5:a86687fb962742f04652aee19ad34e06',
+            'thumbnail': r're:^https?://.*\.jpg$',
+            'ext': 'm3u8',
+        },
+    }]
+
+    def _real_extract(self, url):
+        video_id, video_path = self._match_valid_url(url).group('id', 'path')
+
+        webpage = self._download_webpage(
+            'https://voe.sx/e/%s' % video_id, video_id)
+
+        player_url = self._search_regex(
+            r'''("|')(?P<url>https://(?!voe\.sx/)[^/]+%s)\1\s*;''' % (path,),
+            webpage, 'redirect', group='url', default=None)
+        if player_url:
+            webpage = self._download_webpage(
+                player_url, video_id, note='Redirecting to player page')
+
+
+        sources = self._search_json(
+            r'\bsources\s*=', webpage, 'sources', video_id,  transform_source=js_to_json)      
+
+        title = self._search_regex(
+            r'<title>(?:Watch\s+)?(?P<title>.+?)(?:-\s+VOE\s+\|.+)?</title>',
+            webpage, 'title', group='title')
+
+        formats = []
+        for fmt in ('mp4, 'hls'):
+            if fmt not in sources:
+                continue
+            sources[fmt] = url_or_none(sources['fmt']) or url_or_none(base64.b64decode(sources['fmt']))
+            f_url = sources.get('hls')
+        if f_url:
+            formats.extend(self._extract_m3u8_formats(
+                f_url, video_id, entry_protocol='m3u8_native', fatal=False))
+        f_url = sources.get('mp4')
+        if f_url:
+            formats.append({
+                'url': f_url,
+                'ext': 'mp4',
+                'height': int_or_none(sources.get('video_height')),
+            })
+
+        self._sort_formats(formats)
+
+        thumbnail = url_or_none(self._search_regex(
+            r'''(?:VOEPlayer\s*\.\s*|data-)poster\s*=\s*("|')(?P<thumbnail>(?:(?!\1)\S)+)\1''',
+            webpage, 'thumbnail', group='thumbnail', default=None))
+
+        return {
+            'id': video_id,
+            'title': title,
+            'formats': formats,
+            'thumbnail': thumbnail,
+        }

youtube_dl/extractor/youtube.py

@@ -1323,10 +1323,10 @@ class YoutubeIE(YoutubeBaseInfoExtractor):
         funcname = self._search_regex(
             (r'\b[cs]\s*&&\s*[adf]\.set\([^,]+\s*,\s*encodeURIComponent\s*\(\s*(?P<sig>[a-zA-Z0-9$]+)\(',
              r'\b[a-zA-Z0-9]+\s*&&\s*[a-zA-Z0-9]+\.set\([^,]+\s*,\s*encodeURIComponent\s*\(\s*(?P<sig>[a-zA-Z0-9$]+)\(',
-             r'\bm=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(h\.s\)\)',
-             r'\bc&&\(c=(?P<sig>[a-zA-Z0-9$]{2,})\(decodeURIComponent\(c\)\)',
-             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',
-             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2,})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
+             r'\bm=(?P<sig>[a-zA-Z0-9$]{2})\(decodeURIComponent\(h\.s\)\)',
+             r'\bc&&\(c=(?P<sig>[a-zA-Z0-9$]{2})\(decodeURIComponent\(c\)\)',
+             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\);[a-zA-Z0-9$]{2}\.[a-zA-Z0-9$]{2}\(a,\d+\)',
+             r'(?:\b|[^a-zA-Z0-9$])(?P<sig>[a-zA-Z0-9$]{2})\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
              r'(?P<sig>[a-zA-Z0-9$]+)\s*=\s*function\(\s*a\s*\)\s*{\s*a\s*=\s*a\.split\(\s*""\s*\)',
              # Obsolete patterns
              r'(["\'])signature\1\s*,\s*(?P<sig>[a-zA-Z0-9$]+)\(',

youtube_dl/version.py

@@ -1,3 +1,3 @@
 from __future__ import unicode_literals
 
-__version__ = '2021.12.17'
+__version__ = '2021.06.06'