v1.4.12.40

sebdelsol · Mar 14, 2024 · c46f1e9 · c46f1e9
1 parent 2c33b89
commit c46f1e9
Show file tree

Hide file tree

Showing 28 changed files with 287 additions and 70 deletions.
diff --git a/README.md b/README.md
@@ -3,15 +3,16 @@
 * Insert an _All_ category when missing so you can easily **search your entire catalog**.  
 <kbd><img src="resources/readme/all.png"></kbd>
 * Update ***[Mpv](https://mpv.io/)*** and ***[Sfvip Player](https://github.com/K4L4Uz/SFVIP-Player/tree/master)*** so you can enjoy their latest features. 
+* Translated in all ***Sfvip Player*** languages.
 * Support an **external EPG**[^1].
 
 [^1]: External EPG doesn't work with **local** m3u accounts.
 # Downloads
-[<img src="https://custom-icon-badges.demolab.com/badge/Sfvip All v1.4.12.39-informational?logo=download-cloud&logoSource=feather&logoColor=white&style=flat-square" height="35"><img src="https://img.shields.io/badge/x64-informational?logo=Windows10&logoColor=lightblue&style=flat-square" height="35"><img src="https://custom-icon-badges.demolab.com/badge/clean-brightgreen?logo=shield-check&logoColor=white&style=flat-square" height="35">](https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.39/Install.Sfvip.All.1.4.12.39.x64.exe)
-<sup><sup>_by MS Defender • 1.1.24020.9 • 1.407.102.0_</sup></sup>
+[<img src="https://custom-icon-badges.demolab.com/badge/Sfvip All v1.4.12.40-informational?logo=download-cloud&logoSource=feather&logoColor=white&style=flat-square" height="35"><img src="https://img.shields.io/badge/x64-informational?logo=Windows10&logoColor=lightblue&style=flat-square" height="35"><img src="https://custom-icon-badges.demolab.com/badge/clean-brightgreen?logo=shield-check&logoColor=white&style=flat-square" height="35">](https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.40/Install.Sfvip.All.1.4.12.40.x64.exe)
+<sup><sup>_by MS Defender • 1.1.24020.9 • 1.407.423.0_</sup></sup>
 
-[<img src="https://custom-icon-badges.demolab.com/badge/Sfvip All v1.4.12.39-informational?logo=download-cloud&logoSource=feather&logoColor=white&style=flat-square" height="35"><img src="https://img.shields.io/badge/x86-informational?logo=Windows10&logoColor=lightblue&style=flat-square" height="35"><img src="https://custom-icon-badges.demolab.com/badge/clean-brightgreen?logo=shield-check&logoColor=white&style=flat-square" height="35">](https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.39/Install.Sfvip.All.1.4.12.39.x86.exe)
-<sup><sup>_by MS Defender • 1.1.24020.9 • 1.407.102.0_</sup></sup>
+[<img src="https://custom-icon-badges.demolab.com/badge/Sfvip All v1.4.12.40-informational?logo=download-cloud&logoSource=feather&logoColor=white&style=flat-square" height="35"><img src="https://img.shields.io/badge/x86-informational?logo=Windows10&logoColor=lightblue&style=flat-square" height="35"><img src="https://custom-icon-badges.demolab.com/badge/clean-brightgreen?logo=shield-check&logoColor=white&style=flat-square" height="35">](https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.40/Install.Sfvip.All.1.4.12.40.x86.exe)
+<sup><sup>_by MS Defender • 1.1.24020.9 • 1.407.423.0_</sup></sup>
 
 [<a href="https://tooomm.github.io/github-release-stats/?username=sebdelsol&repository=sfvip-all"><img src="https://img.shields.io/github/downloads/sebdelsol/sfvip-all/total?color=informational&logo=github&labelColor=informational&style=flat-square" height="25"></a><a href="https://github.com/sebdelsol/sfvip-all/issues?q=is%3Aopen"><img src="https://img.shields.io/github/issues/sebdelsol/sfvip-all?labelColor=success&style=flat-square" height="25"></a>]()
 
@@ -42,12 +43,12 @@ The logs go **in pairs** (one for each process: ***main*** & ***mitmproxy***) an
 
 # Build
 [![version](https://custom-icon-badges.demolab.com/badge/Build%201.4.12.40-informational?logo=github)](/build_config.py#L27)
-[![Sloc](https://custom-icon-badges.demolab.com/badge/Sloc%208.2k-informational?logo=file-code)](https://api.codetabs.com/v1/loc/?github=sebdelsol/sfvip-all)
+[![Sloc](https://custom-icon-badges.demolab.com/badge/Sloc%208.3k-informational?logo=file-code)](https://api.codetabs.com/v1/loc/?github=sebdelsol/sfvip-all)
 [![Ruff](https://custom-icon-badges.demolab.com/badge/Ruff-informational?logo=ruff-color)](https://docs.astral.sh/ruff/)
 [![Python](https://custom-icon-badges.demolab.com/badge/Python%203.11.8-linen?logo=python-color)](https://www.python.org/downloads/release/python-3118/)
 [![mitmproxy](https://custom-icon-badges.demolab.com/badge/Mitmproxy%2010.2.4-linen?logo=mitmproxy-black)](https://mitmproxy.org/)
 [![Nsis](https://custom-icon-badges.demolab.com/badge/Nsis%203.09-linen?logo=nsis-color)](https://nsis.sourceforge.io/Download)
-[![Nuitka](https://custom-icon-badges.demolab.com/badge/Nuitka%202.1.1-linen?logo=nuitka)](https://nuitka.net/)
+[![Nuitka](https://custom-icon-badges.demolab.com/badge/Nuitka%202.1.2-linen?logo=nuitka)](https://nuitka.net/)
 [![PyInstaller](https://custom-icon-badges.demolab.com/badge/PyInstaller%206.5.0-linen?logo=pyinstaller-windowed)](https://pyinstaller.org/en/stable/)
 
 * [***NSIS***](https://nsis.sourceforge.io/Download) will be automatically installed if missing.  

diff --git a/build/changelog.md b/build/changelog.md
@@ -1,7 +1,8 @@
 ## 1.4.12.40
-* Fix EPG channels with wrong programmes.
-* Fix EPG processing restarting when closing the UI.
+* Option to prefer the IPTV provider EPG over external EPG.
 * Faster EPG processing and loading that use less memory.
+* Fix EPG processing restarting when closing the UI.
+* Better fuzzy match for external EPG.
 
 ## 1.4.12.39
 * Bump _mitmproxy_ to 10.2.4.

diff --git a/build/update_x64.json b/build/update_x64.json
@@ -1,5 +1,5 @@
 {
-  "url": "https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.39/Install.Sfvip.All.1.4.12.39.x64.exe",
-  "md5": "4ce25c536d359eca0e7191485c7e6334",
-  "version": "1.4.12.39"
+  "url": "https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.40/Install.Sfvip.All.1.4.12.40.x64.exe",
+  "md5": "56f26c63ad43e960435271ee0adf5c6b",
+  "version": "1.4.12.40"
 }
diff --git a/build/update_x86.json b/build/update_x86.json
@@ -1,5 +1,5 @@
 {
-  "url": "https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.39/Install.Sfvip.All.1.4.12.39.x86.exe",
-  "md5": "966b1837969c1537549109f4075e0dec",
-  "version": "1.4.12.39"
+  "url": "https://github.com/sebdelsol/sfvip-all/releases/download/Sfvip.All.1.4.12.40/Install.Sfvip.All.1.4.12.40.x86.exe",
+  "md5": "898db7fcb22fc3d80b0c435e444f6d7d",
+  "version": "1.4.12.40"
 }
diff --git a/dev/build.py b/dev/build.py
@@ -12,6 +12,7 @@ def do_build(build: CfgBuild, environments: CfgEnvironments, templates: CfgTempl
     publisher = Publisher(build, environments, Github)
     if Builder(build, environments, LOC, publisher).build_all():
         publisher.show_versions()
+        print()
     Templater(build, environments, Github, publisher).create_all(templates)
 
 

diff --git a/resources/README_template.md b/resources/README_template.md
@@ -3,6 +3,7 @@
 * Insert an _All_ category when missing so you can easily **search your entire catalog**.  
 <kbd><img src="resources/readme/all.png"></kbd>
 * Update ***[Mpv](https://mpv.io/)*** and ***[Sfvip Player](https://github.com/K4L4Uz/SFVIP-Player/tree/master)*** so you can enjoy their latest features. 
+* Translated in all ***Sfvip Player*** languages.
 * Support an **external EPG**[^1].
 
 [^1]: External EPG doesn't work with **local** m3u accounts.

diff --git a/sfvip_all_config.py b/sfvip_all_config.py
@@ -24,6 +24,7 @@ class EPG:
         url: str | None = None
         confidence: int = 30
         requests_timeout: int = 5
+        prefer_internal: bool = True
 
     class AllCategory:
         inject_in_live: bool = False
diff --git a/src/mitm/addon/__init__.py b/src/mitm/addon/__init__.py
@@ -35,6 +35,10 @@ def get_short_epg(flow: http.HTTPFlow, epg: EPG, api: APItype) -> None:
     if response := flow.response:
 
         def set_response(stream_id: str, limit: str, programmes: str) -> None:
+            # already an epg ?
+            if epg.prefer_updater.prefer_internal and (json_response := response_json(flow.response)):
+                if isinstance(json_response, dict) and json_response.get(programmes):
+                    return
             server = flow.request.host_header
             if _id := get_query_key(flow, stream_id):
                 _limit = get_query_key(flow, limit)
@@ -153,12 +157,15 @@ def __init__(
         self.m3u_stream = M3UStream(self.epg)
         self.panels = AllPanels(all_config.all_name)
 
-    def epg_update(self, url: str):
+    def epg_update(self, url: str) -> None:
         self.epg.ask_update(url)
 
-    def epg_confidence_update(self, confidence: int):
+    def epg_confidence_update(self, confidence: int) -> None:
         self.epg.update_confidence(confidence)
 
+    def epg_prefer_update(self, prefer_internal: bool) -> None:
+        self.epg.update_prefer(prefer_internal)
+
     def running(self) -> None:
         self.epg.start()
 

diff --git a/src/mitm/epg/__init__.py b/src/mitm/epg/__init__.py
@@ -54,6 +54,23 @@ def confidence(self) -> Optional[int]:
             return self._confidence
 
 
+class PreferUpdater(JobRunner[bool]):
+    def __init__(self) -> None:
+        self._prefer_internal_lock = multiprocessing.Lock()
+        self._prefer_internal: Optional[bool] = None
+        super().__init__(self._updating, "Epg prefer internal updater")
+
+    def _updating(self, prefer_internal: bool) -> None:
+        with self._prefer_internal_lock:
+            self._prefer_internal = prefer_internal
+
+    @property
+    def prefer_internal(self) -> Optional[bool]:
+        with self._prefer_internal_lock:
+            return self._prefer_internal
+
+
+# pylint: disable=too-many-instance-attributes
 class EPG:
     _programme_type = {APItype.XC: EPGprogrammeXC, APItype.MAC: EPGprogrammeMAC, APItype.M3U: EPGprogrammeM3U}
     _m3u_server = "m3u.server"
@@ -63,6 +80,7 @@ def __init__(self, roaming: Path, callbacks: EpgCallbacks, timeout: int) -> None
         self.servers: dict[str, EPGserverChannels] = {}
         self.updater = EPGupdater(roaming, callbacks.update_status, timeout)
         self.confidence_updater = ConfidenceUpdater()
+        self.prefer_updater = PreferUpdater()
         self.show_channel = callbacks.show_channel
         self.channel_shown = False
         self.show_epg = callbacks.show_epg
@@ -74,15 +92,20 @@ def ask_update(self, url: str) -> None:
     def update_confidence(self, confidence: int) -> None:
         self.confidence_updater.add_job(confidence)
 
+    def update_prefer(self, prefer_internal: bool) -> None:
+        self.prefer_updater.add_job(prefer_internal)
+
     def wait_running(self, timeout: int) -> bool:
         return self.updater.wait_running(timeout)
 
     def start(self) -> None:
         self.confidence_updater.start()
+        self.prefer_updater.start()
         self.updater.start()
 
     def stop(self) -> None:
         self.updater.stop()
+        self.prefer_updater.stop()
         self.confidence_updater.stop()
 
     def set_server_channels(self, server: Optional[str], channels: Any, api: APItype) -> None:

diff --git a/src/mitm/epg/programme.py b/src/mitm/epg/programme.py
@@ -26,7 +26,6 @@ def from_programme(cls, programme: InternalProgramme, now: float) -> Optional[Se
                 return cls(start, end)
         return None
 
-    # TODO check time (CNN bug)
     @staticmethod
     def _get_timestamp(date: str) -> Optional[int]:
         try:

diff --git a/src/mitm/epg/update.py b/src/mitm/epg/update.py
@@ -5,9 +5,17 @@
 import re
 import tempfile
 from contextlib import contextmanager
-from enum import Enum, auto
+from enum import Enum, auto, member
 from pathlib import Path
-from typing import IO, Callable, Iterator, NamedTuple, Optional, Self
+from typing import (
+    IO,
+    Callable,
+    Container,
+    Iterator,
+    NamedTuple,
+    Optional,
+    Self,
+)
 from urllib.parse import urlparse
 
 import lxml.etree as ET
@@ -50,8 +58,11 @@ class EPGProcess(NamedTuple):
 
 def _normalize(name: str) -> str:
     name = re.sub(r"(\.)([\d]+)", r"\2", name)  # turn channel.2 into channel2
-    for sub, repl in (".+", "plus"), ("+", "plus"), ("*", "star"):
+    for sub, repl in ("+", "plus"), ("*", "star"):
         name = name.replace(sub, repl)
+    for char in ".|()[]-":
+        name = name.replace(char, " ")
+    name = re.sub(r"\s+", " ", name).strip()  # remove extra white spaces
     return name
 
 
@@ -66,6 +77,7 @@ def _valid_url(url: str) -> bool:
 def parse_programme(file_obj: IO[bytes] | gzip.GzipFile, epg_process: EPGProcess) -> Iterator[NamedProgrammes]:
     current_programmes: dict[InternalProgramme, bool] = {}
     current_channel_id: Optional[str] = None
+    normalized: dict[str, str] = {}
     progress_step = ProgressStep()
     elem: ET.ElementBase
     title: str = ""
@@ -88,12 +100,13 @@ def parse_programme(file_obj: IO[bytes] | gzip.GzipFile, epg_process: EPGProcess
                 desc = elem.text or ""
             case "programme":
                 if channel_id := elem.get("channel", None):
-                    channel_id = _normalize(channel_id)
-                    if channel_id != current_channel_id:
+                    if not (norm_channel_id := normalized.get(channel_id)):
+                        norm_channel_id = normalized[channel_id] = _normalize(channel_id)
+                    if norm_channel_id != current_channel_id:
                         if current_channel_id and current_programmes:
                             yield NamedProgrammes(tuple(current_programmes), current_channel_id)
                         current_programmes = {}
-                        current_channel_id = channel_id
+                        current_channel_id = norm_channel_id
                         if progress := progress_step.increment_progress(1):
                             epg_process.update_status(EPGProgress(EPGstatus.PROCESSING, progress))
                     start = elem.get("start", "")
@@ -112,6 +125,74 @@ class FoundProgammes(NamedTuple):
     confidence: int
 
 
+class FuzzResult(NamedTuple):
+    name: str
+    score: float
+
+    @classmethod
+    def from_result(cls, result: tuple) -> Self:
+        return cls(*result[:2])
+
+
+class Scorer(Enum):
+    RATIO = member(fuzz.ratio)
+    PARTIAL_RATIO = member(fuzz.partial_ratio)
+    TOKEN_SET_RATIO = member(fuzz.token_set_ratio)
+    TOKEN_SORT_RATIO = member(fuzz.token_sort_ratio)
+    PARTIAL_TOKEN_SET_RATIO = member(fuzz.partial_token_set_ratio)
+    PARTIAL_TOKEN_SORT_RATIO = member(fuzz.partial_token_sort_ratio)
+
+
+class FuzzBest:
+    _scorers = (
+        # used for cuttoff and overall score
+        (Scorer.TOKEN_SET_RATIO, 0.5),
+        # used overal score
+        (Scorer.TOKEN_SORT_RATIO, 1),
+        (Scorer.PARTIAL_TOKEN_SORT_RATIO, 0.5),
+        (Scorer.RATIO, 1),
+    )
+    _limit = 5
+
+    def __init__(self, choices: Container[str]) -> None:
+        self._choices = choices
+
+    @staticmethod
+    def _extract(query: str, choices: Container[str], scorer: Scorer, cutoff: int = 0) -> list[FuzzResult]:
+        results = process.extractBests(
+            query, choices, limit=FuzzBest._limit, scorer=scorer.value, score_cutoff=cutoff
+        )
+        return [FuzzResult.from_result(result) for result in results]
+
+    def _best(self, query: str, confidence: int) -> Optional[FuzzResult]:
+        # cutoff using the 1st scorer
+        scorer, weight = FuzzBest._scorers[0]
+        if results := self._extract(query, self._choices, scorer=scorer, cutoff=100 - confidence):
+            # print(f"{query=}")
+            # print("found", results)
+            if len(results) == 1:
+                return results[0]
+            # get the accumulated score with weight
+            scores = {result.name: result.score for result in results}
+            accumulated_scores = {name: score * weight for name, score in scores.items()}
+            for scorer, weight in FuzzBest._scorers[1:]:
+                for result in self._extract(query, scores.keys(), scorer=scorer):
+                    accumulated_scores[result.name] += result.score * weight
+            # print(accumulated_scores)
+            best = sorted(
+                (FuzzResult(name, score) for name, score in accumulated_scores.items()),
+                key=lambda result: result.score,
+                reverse=True,
+            )[0]
+            return FuzzResult(best.name, scores[best.name])
+        return None
+
+    def get(self, query: str, confidence: int) -> Optional[FuzzResult]:
+        if query in self._choices:
+            return FuzzResult(query, 100)
+        return self._best(query, confidence)
+
+
 class EPGupdate(NamedTuple):
     _chunk_size = 1024 * 128
     url: str
@@ -199,23 +280,16 @@ def from_url(cls, url: str, cache: ChannelsCache, epg_process: EPGProcess, timeo
 
     def get_programmes(self, epg_id: str, confidence: int) -> Optional[FoundProgammes]:
         if self.programmes:
-            normalized_epg_id = _normalize(epg_id)
-            if result := process.extractOne(
-                normalized_epg_id,
-                self.programmes.all_names,
-                scorer=fuzz.token_sort_ratio,
-                score_cutoff=100 - confidence,
-            ):
-                normalized_epg_id, score = result[:2]
-                if programmes := self.programmes.get_programmes(normalized_epg_id):
+            if found := FuzzBest(self.programmes.all_names).get(_normalize(epg_id), confidence):
+                if programmes := self.programmes.get_programmes(found.name):
                     logger.info(
-                        "Found Epg %s for %s with confidence %s%% (cut off @%s%%)",
-                        normalized_epg_id,
+                        "Found Epg '%s' for %s with confidence %s%% (cut off @%s%%)",
+                        found.name,
                         epg_id,
-                        score,
+                        found.score,
                         100 - confidence,
                     )
-                    return FoundProgammes(programmes, int(score))
+                    return FoundProgammes(programmes, int(found.score))
         return None