Flatpak won't open due to brotli

[Artim96]

While I can't tell if my last issue was fixed with this release, there's now another issue with 0.10.2 preventing the app from even opening:

flatpak run vn.hoabinh.quan.CoBang
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/app/lib/python3.10/site-packages/cobang/__main__.py", line 21, in <module>
    from .app import CoBangApplication
  File "/app/lib/python3.10/site-packages/cobang/app.py", line 33, in <module>
    from . import ui
  File "/app/lib/python3.10/site-packages/cobang/ui.py", line 16, in <module>
    from .resources import get_ui_filepath
  File "/app/lib/python3.10/site-packages/cobang/resources.py", line 8, in <module>
    import requests
  File "/app/lib/python3.10/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/app/lib/python3.10/site-packages/urllib3/__init__.py", line 13, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/app/lib/python3.10/site-packages/urllib3/connectionpool.py", line 40, in <module>
    from .response import HTTPResponse
  File "/app/lib/python3.10/site-packages/urllib3/response.py", line 162, in <module>
    class HTTPResponse(io.IOBase):
  File "/app/lib/python3.10/site-packages/urllib3/response.py", line 395, in HTTPResponse
    DECODER_ERROR_CLASSES += (brotli.error,)
AttributeError: module 'brotli' has no attribute 'error'

[hongquan]

Yes, since v0.10.1, CoBang has been broken in production (after releasing to FlatHub), while is still Ok if run in development. I'm still trying to fix. It is difficult though, because it can only reproducible after releasing to FlatHub.

[hongquan]

Could you try the v0.10.3? It is not listed on FlatHub yet, but can still try the test build as in flathub/vn.hoabinh.quan.CoBang#9 (comment)

[hongquan]

From #76 (comment), this still exists, though I cannot reproduce it.

[Artim96]

From #76 (comment), this still exists, though I cannot reproduce it.

I have recorded an strace log if that helps, after completely uninstalling CoBang and deleting its data and reinstalling it from flathub stable.
cobang_strace.txt

[hongquan]

@Artim96 Looking into your log, I find something weird. Your log said that CoBang is running with Python 3.10, but the base Flatpak image of the v0.10.3 comes with Python 3.11. Maybe your system still install the old version of CoBang?

[hongquan]

Ah, the strace.txt log is newer, it is Python 3.11.

[hongquan]

@Artim96 CoBang v0.10.4 has been released, with Python dependencies being upgraded. Could you test if the issue of missing brotli still happen?

It doesn't appear on FlatHub yet, you can install by the command in flathub/vn.hoabinh.quan.CoBang#10 (comment)

[Artim96]

Again, same error.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/app/lib/python3.11/site-packages/cobang/__main__.py", line 21, in <module>
    from .app import CoBangApplication
  File "/app/lib/python3.11/site-packages/cobang/app.py", line 33, in <module>
    from . import ui
  File "/app/lib/python3.11/site-packages/cobang/ui.py", line 16, in <module>
    from .resources import get_ui_filepath
  File "/app/lib/python3.11/site-packages/cobang/resources.py", line 8, in <module>
    import requests
  File "/app/lib/python3.11/site-packages/requests/__init__.py", line 43, in <module>
    import urllib3
  File "/app/lib/python3.11/site-packages/urllib3/__init__.py", line 17, in <module>
    from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
  File "/app/lib/python3.11/site-packages/urllib3/connectionpool.py", line 15, in <module>
    from ._request_methods import RequestMethods
  File "/app/lib/python3.11/site-packages/urllib3/_request_methods.py", line 10, in <module>
    from .response import BaseHTTPResponse
  File "/app/lib/python3.11/site-packages/urllib3/response.py", line 284, in <module>
    class BaseHTTPResponse(io.IOBase):
  File "/app/lib/python3.11/site-packages/urllib3/response.py", line 294, in BaseHTTPResponse
    DECODER_ERROR_CLASSES += (brotli.error,)
                              ^^^^^^^^^^^^
AttributeError: module 'brotli' has no attribute 'error'

cobang_strace.txt

[hongquan]

The Flatpak package of CoBang is supposed to not include Brotli, its sandbox environment should not have Brotli Python library. But somehow your sandbox environment has an old version of Brotli. I wonder where it is from, maybe the host system leaks to Flatpak sandbox?
Could you help me debug your sandbox to see where brotli comes from:

1. Access CoBang sandbox shell:

$ flatpak run --command=sh vn.hoabinh.quan.CoBang

2. Inside the sandbox, cd to /app/ then run python3.
3. In Python REPL, do this:

  >>> import brotli
  >>> brotli.__file__

and

   >>> import brotlicffi
   >>> brotlicffi.__file__

[Artim96]

python3
Python 3.11.7 (main, Nov 10 2011, 15:00:00) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import brotli
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotli'
>>> import brotlicffi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotlicffi'

[Artim96]

I also just went through a few flatpak directories with ripgrep-all to find all mentions of brotli. Here's what I found, maybe that might help.

 rga "brotli" .local/share/flatpak/app/vn.hoabinh.quan.CoBang/
.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/pkgconfig/libsoup-2.4.pc
Requires.private: sysprof-capture-4, libxml-2.0, sqlite3, libpsl >=  0.20, libbrotlidec, zlib

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/pkgconfig/libsoup-gnome-2.4.pc
Requires.private: sysprof-capture-4, glib-2.0 >=  2.58, gmodule-2.0 >=  2.58, gobject-2.0 >=  2.58, gio-2.0 >=  2.58, libxml-2.0, sqlite3, libpsl >=  0.20, libbrotlidec, zlib

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/Logbook-1.7.0.post0.dist-info/METADATA
Requires-Dist: brotli ; extra == 'compression'

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/Pillow-10.1.0.dist-info/LICENSE
BROTLI
Copyright (c) 2009, 2010, 2013-2016 by the Brotli Authors.

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/Pillow-10.1.0.dist-info/RECORD
Pillow.libs/libbrotlicommon-3ecfe81c.so.1,sha256=AkvHYFUCz_1Fs_fD83_gQ2lMw7S0y385rzuacnk-TC4,144425
Pillow.libs/libbrotlidec-922c819b.so.1,sha256=CGnqJ6LQOPrJ57Pf2TUjXYf7UdIu2DbE5s4ZY-CXXWQ,58225

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/urllib3/util/request.py
        import brotlicffi as _unused_module_brotli  # type: ignore[import] # noqa: F401
        import brotli as _unused_module_brotli  # type: ignore[import] # noqa: F401
        ``True`` translates to 'gzip,deflate'.  If either the ``brotli`` or
        ``brotlicffi`` package is installed 'gzip,deflate,br' is used instead.

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/urllib3-2.1.0.dist-info/METADATA
Provides-Extra: brotli
Requires-Dist: brotli>=1.0.9; platform_python_implementation == 'CPython' and extra == 'brotli'
Requires-Dist: brotlicffi>=0.8.0; platform_python_implementation != 'CPython' and extra == 'brotli'
- Support for gzip, deflate, brotli, and zstd encoding.

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/urllib3/response.py
        import brotlicffi as brotli  # type: ignore[import]
        import brotli  # type: ignore[import]
    brotli = None
if brotli is not None:
    class BrotliDecoder(ContentDecoder):
        # Supports both 'brotlipy' and 'Brotli' packages
        # are for 'brotlipy' and bottom branches for 'Brotli'
            self._obj = brotli.Decompressor()
    if brotli is not None and mode == "br":
        return BrotliDecoder()
    if brotli is not None:
    if brotli is not None:
        DECODER_ERROR_CLASSES += (brotli.error,)

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/logbook/handlers.py
class BrotliCompressionHandler(FileHandler):
            from brotli import Compressor
                "The brotli library is required for the BrotliCompressionHandler."

.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/logbook/__init__.py
    BrotliCompressionHandler,

And looking for files or directories containing brotli in their name, I found this:

find .local/share/flatpak/app/vn.hoabinh.quan.CoBang/ -iname "*brotli*"
.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/Pillow.libs/libbrotlidec-922c819b.so.1
.local/share/flatpak/app/vn.hoabinh.quan.CoBang/x86_64/test/57aabcce73d34c75b287a06a52673f0f7b691f35074575534a04ec5c33f9c56f/files/lib/python3.11/site-packages/Pillow.libs/libbrotlicommon-3ecfe81c.so.1

[hongquan]

Your Python shell shows that brotli Python module does not exist in the sandbox. That is expected. But somehow this code goes through:

    if brotli is not None:
        DECODER_ERROR_CLASSES += (brotli.error,)

So weird.

[hongquan]

Could you run Python REPL in the sandbox again and try to run parts of urllib3's code, to see how the brotli variable is.

First, run all the import lines, and until this line https://github.com/urllib3/urllib3/blob/69be2992f8a25a1f27e49f339e4d5b98dec07462/src/urllib3/response.py#L39

After this step, please check how brotli is.

Second, run these lines.

[Artim96]

python3
Python 3.11.7 (main, Nov 10 2011, 15:00:00) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import annotations
>>> import collections
>>> import io
>>> import json as _json
>>> import logging
>>> import re
>>> import sys
>>> import typing
>>> import warnings
>>> import zlib
>>> from contextlib import contextmanager
>>> from http.client import HTTPMessage as _HttplibHTTPMessage
>>> from http.client import HTTPResponse as _HttplibHTTPResponse
>>> from socket import timeout as SocketTimeout
>>> try:
...     import zstandard as zstd
...     _zstd_version = _zstd_version = tuple(
...         map(int, re.search(r"^([0-9]+)\.([0-9]+)", zstd.__version__).groups())
...     )
...     if _zstd_version < (0, 18):
...         zstd = None
... except (AttributeError, ImportError, ValueError):  # Defensive:
...     zstd = None
... 
>>> import brotli
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotli'
>>> import brotlicffi
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotlicffi'
>>> class BaseHTTPResponse(io.IOBase):
...     CONTENT_DECODERS = ["gzip", "x-gzip", "deflate"]
...     if brotli is not None:
...         CONTENT_DECODERS += ["br"]
...     if zstd is not None:
...         CONTENT_DECODERS += ["zstd"]
...     REDIRECT_STATUSES = [301, 302, 303, 307, 308]
...     DECODER_ERROR_CLASSES: tuple[type[Exception], ...] = (IOError, zlib.error)
...     if brotli is not None:
...         DECODER_ERROR_CLASSES += (brotli.error,)
...     if zstd is not None:
...         DECODER_ERROR_CLASSES += (zstd.ZstdError,)
... 
>>> import brotli
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotli'
>>> brotli.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute '__file__'. Did you mean: '__le__'?

[hongquan]

Thank you, looking into this

>>> import brotli
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'brotli'
>>> brotli.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute '__file__'. Did you mean: '__le__'?

I want to cry.

Right previously, Python said that ModuleNotFoundError: No module named 'brotli', but then in next line of code, it considers brotli to be NoneType object!

-- Update --

Ah, the result:

>>> brotli.__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute '__file__'. Did you mean: '__le__'?

is because brotli variable had been assigned None at https://github.com/urllib3/urllib3/blob/69be2992f8a25a1f27e49f339e4d5b98dec07462/src/urllib3/response.py#L23

[hongquan]

I notice that when you run the definition code of BaseHTTPResponse(io.IOBase) manually in Python shell, Python doesn't raise error at

DECODER_ERROR_CLASSES += (brotli.error,)

So in this shell test, the code behaves as expected.

[Artim96]

Sadly, I also can't test if there may be leakage from the OS into the Flatpak container. I have python3-brotli installed (on Debian) because it's required by python3-urllib3 which itself is required e.g. by gnome-music and gnome-browser-connector. And both are required by gnome itself. So Brotli as a Python package may be present on any system with Gnome installed.

[hongquan]

I use Ubuntu and also have python3-brotli installed, but it is not leaked to Flatpak sandbox. So weird. I will try changing the Flatpak build script to include Brotli, to see how things will be.

[hongquan]

Hi. I has released v0.10.5, where I include Python Brotli lib. Could you help test?

[Artim96]

When I'm able to update, sure. Right now I only get the error
error: Can't load uri https://dl.flathub.org/build-repo/74100/vn.hoabinh.quan.CoBang.flatpakref: Server returned status 404
so it seems the server is down right now.
`

[Artim96]

0.10.5 was now released. It works now with the only error message being
** (gst-plugin-scanner:7): CRITICAL **: 10:01:35.246: Couldn't g_module_open libpython. Reason: /usr/lib/x86_64-linux-gnu/libpython3.11.so: cannot open shared object file: No such file or directory

Though the UI seems a bit wonky. In maximized screen you have the results bar in the right, but when scanning e.g. a WiFi QR Code generated by Android (by the settings app, at least on Pixel devices, no idea if that's a Pixel or an Android feature), the results bar moves to the bottom edge to very litlle height with the content showing up in raw results underneath it. The scaling of the raw results bar is off though. It's barely high enough for the text, but the copy button is being cut off. In not maximized screen, there's also only the narrow results bar at the bottom with the raw results underneath, but here the text box is even narrower, also cutting off the top of the text.

But that would be something for a different issue.

[hongquan]

Thank you. The UI issue is known. It is the result of my attempt to make the UI responsive, working both on desktop screen and mobile (Linux phone) screen. To workaround it, I will work on a feature of remember window size.

The message of "missing libpython3.11.so" is undesired, though.