Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mirage: 0.6.4 -> 0.7.0 #115189

Closed
wants to merge 11 commits into from
Closed

mirage: 0.6.4 -> 0.7.0 #115189

wants to merge 11 commits into from

Conversation

vojta001
Copy link
Contributor

@vojta001 vojta001 commented Mar 5, 2021

Motivation for this change

Update Mirage to 0.7.0. To make this work, matrix-nio has to be updated as well, but I am struggling to do it (they began using Tox). Could you @tilpner, @emilazy and @symphorien, the maintainers of matrix-nio, have a look at it? Mirage can be built ATM, but does not work.

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@r-rmcgibbo
Copy link

r-rmcgibbo commented Mar 5, 2021

Result of nixpkgs-review pr 115189 at ec97688d run on x86_64-linux 1

10 packages built:
2 suggestions:
  • warning: unclear-gpl

    lgpl3 is a deprecated license, check if project uses lgpl3Plus or lgpl3Only and change meta.license accordingly.

    Near pkgs/applications/networking/instant-messengers/mirage/default.nix:54:5:

       |
    54 |     license = licenses.lgpl3;
       |     ^
    
  • warning: unused-argument

    Unused argument: qttools.
    Near pkgs/applications/networking/instant-messengers/mirage/default.nix:3:55:

      |
    3 | , qtbase, qtquickcontrols2, qtkeychain, qtmultimedia, qttools, qtgraphicaleffects
      |                                                       ^
    

Result of nixpkgs-review pr 115189 at ec97688d run on aarch64-linux 1

10 packages built:
2 suggestions:
  • warning: unclear-gpl

    lgpl3 is a deprecated license, check if project uses lgpl3Plus or lgpl3Only and change meta.license accordingly.

    Near pkgs/applications/networking/instant-messengers/mirage/default.nix:54:5:

       |
    54 |     license = licenses.lgpl3;
       |     ^
    
  • warning: unused-argument

    Unused argument: qttools.
    Near pkgs/applications/networking/instant-messengers/mirage/default.nix:3:55:

      |
    3 | , qtbase, qtquickcontrols2, qtkeychain, qtmultimedia, qttools, qtgraphicaleffects
      |                                                       ^
    

maintainers/maintainer-list.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/plyer/default.nix Outdated Show resolved Hide resolved
pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved
pkgs/top-level/python-packages.nix Outdated Show resolved Hide resolved
@symphorien
Copy link
Member

For matrix-nio, maybe you were stuck because you need to add format = "pyproject"
Here is what I have

diff --git a/pkgs/development/python-modules/matrix-nio/default.nix b/pkgs/development/python-modules/matrix-nio/default.nix
index 69774b58eb6..d14813bec21 100644
--- a/pkgs/development/python-modules/matrix-nio/default.nix
+++ b/pkgs/development/python-modules/matrix-nio/default.nix
@@ -5,6 +5,7 @@
 , attrs
 , future
 , aiohttp
+, aiohttp-socks
 , aiofiles
 , h11
 , h2
@@ -16,27 +17,32 @@
 , peewee
 , cachetools
 , atomicwrites
+, poetry-core
 }:
 
 buildPythonPackage rec {
   pname = "matrix-nio";
-  version = "0.15.2";
+  version = "0.17.0";
 
   src = fetchFromGitHub {
     owner = "poljar";
     repo = "matrix-nio";
     rev = version;
-    sha256 = "190xw3cvk4amr9pl8ip2i7k3xdjd0231kn2zl6chny5axx22p1dv";
+    sha256 = "sha256:0fdcbkrki5ni8aaq9xbrmpj1n7hf9z1vvi1b6jkzl55ahc0n50ki";
   };
 
   nativeBuildInputs = [
     git
+    poetry-core
   ];
 
+  format = "pyproject";
+
   propagatedBuildInputs = [
     attrs
     future
     aiohttp
+    aiohttp-socks
     aiofiles
     h11
     h2

but it's not finished because it needs newer dependencies all the way down. feel free to reuse this diff, as I won't make a PR very soon, it's not my priority right now.

@vojta001
Copy link
Contributor Author

vojta001 commented Mar 8, 2021

@symphorien thanks a lot! I didn't know format = "pyproject" was a thing.

I used your diff as a base and continued the hunt. Paradoxically the dependencies are too new, so I came up with this

diff --git a/pkgs/top-level/python-packages.nix b/pkgs/top-level/python-packages.nix
index 9e7ee0a7a1a..c5efd2c70f5 100644
--- a/pkgs/top-level/python-packages.nix
+++ b/pkgs/top-level/python-packages.nix
@@ -4023,7 +4023,37 @@ in {
 
   matrix-client = callPackage ../development/python-modules/matrix-client { };
 
-  matrix-nio = callPackage ../development/python-modules/matrix-nio { };
+  matrix-nio = callPackage ../development/python-modules/matrix-nio {
+    aiofiles = self.aiofiles.overridePythonAttrs (old: rec {
+      version = "0.4.0";
+      src = old.src.override {
+        inherit version;
+        sha256 = "sha256-Ah6gujFKhgJ8Fm7MS0wH8tQPwPSzqVDRhooPJXHCu+4=";
+      };
+    });
+    h11 = self.h11.overridePythonAttrs (old: rec {
+      version = "0.9.0";
+      src = old.src.override {
+        inherit version;
+        sha256 = "sha256-M9S8p74PoDn06E1QqwBTEEflPW7o/7yDUB6mAsFpyuE=";
+      };
+    });
+    h2 = (self.h2.overridePythonAttrs (old: rec {
+      version = "3.2.0";
+      src = old.src.override {
+        inherit version;
+        sha256 = "sha256-h19B69byxEeBJZAFsVf67RpQMd865ap7y0YopsB4LxQ=";
+      };
+    })).override {
+      hpack = self.hpack.overridePythonAttrs (old: rec {
+        version = "3.0.0";
+        src = old.src.override {
+          inherit version;
+          sha256 = "sha256-juycH0v640CKPzBQAmH35qZZEtwThSbqBU+a2YiS6dI=";
+        };
+      });
+    };
+  };
 
   mautrix = callPackage ../development/python-modules/mautrix { };
 

I don't know Nix/nixpkgs enough. matrix-nio's dependency h2, when downgraded, requires a downgraded hpack. I tried it with the override call, but it doesn't work. How do I override a dependency of a dependency?

@vojta001
Copy link
Contributor Author

vojta001 commented Mar 9, 2021

Result of nixpkgs-review pr 115189 run on x86_64-linux 1

1 package failed to build:
  • pantalaimon
12 packages built:
  • electrum
  • mirage-im
  • python38Packages.aiohttp-socks
  • python38Packages.hsluv
  • python38Packages.matrix-nio
  • python38Packages.plyer
  • python38Packages.python-socks
  • python38Packages.watchgod
  • python39Packages.hsluv
  • python39Packages.plyer
  • python39Packages.watchgod
  • weechatScripts.weechat-matrix

@vojta001
Copy link
Contributor Author

vojta001 commented Mar 9, 2021

Result of nixpkgs-review pr 115189 run on x86_64-linux 1

13 packages built:
  • electrum
  • mirage-im
  • pantalaimon
  • python38Packages.aiohttp-socks
  • python38Packages.hsluv
  • python38Packages.matrix-nio
  • python38Packages.plyer
  • python38Packages.python-socks
  • python38Packages.watchgod
  • python39Packages.hsluv
  • python39Packages.plyer
  • python39Packages.watchgod
  • weechatScripts.weechat-matrix

@ofborg ofborg bot requested a review from Valodim March 9, 2021 10:56
@vojta001
Copy link
Contributor Author

vojta001 commented Mar 9, 2021

@colemickens Now it builds and I resolved all stylistic suggestions

@vojta001 vojta001 marked this pull request as ready for review March 9, 2021 11:11
@vojta001 vojta001 requested review from FRidh and jonringer as code owners March 9, 2021 11:11
};

patches = [ ./update-matrix-nio.patch ];
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use substituteInPlace instead.

@symphorien
Copy link
Member

mirage works.

About updating the dependencies of matrix-nio: I asked upstream matrix-nio/matrix-nio#249 In the past they have been pretty responsive.

@SuperSandro2000
Copy link
Member

This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch).
If you have any questions or problems please reach out to SuperSandro2000 on IRC.

Result of nixpkgs-review pr 115189 run on x86_64-linux 1

13 packages built:
  • electrum
  • mirage-im
  • pantalaimon
  • python38Packages.aiohttp-socks
  • python38Packages.hsluv
  • python38Packages.matrix-nio
  • python38Packages.plyer
  • python38Packages.python-socks
  • python38Packages.watchgod
  • python39Packages.hsluv
  • python39Packages.plyer
  • python39Packages.watchgod
  • weechatScripts.weechat-matrix

];

patches = [ ./update-dependencies.patch ];

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could also do with being a substituteInPlace, and there's often little point in bumping it up to a specific version as it just means the next person will have to manually bump the bumps next time a dependency is bumped in nixpkgs. Does it work if you switch them all to = "*"?

Also it would be nice to have a go at enabling some of the tests (or at least a pythonImportsCheck) if we're going to be messing with the dependencies now.

@risicle
Copy link
Contributor

risicle commented Mar 14, 2021

nixpkgs-review is happy on macos 10.14 FWIW

@symphorien
Copy link
Member

This breaks pantalaimon. I provide a fix in ee15c78
wrt enabling test for matrix-nio as suggested by @risicle here is a fix: c48af03

It seems you haven't ticked the box allowing me to push these directly to your PR so you'll have to pick these commits by hand.

@vojta001 vojta001 mentioned this pull request Apr 29, 2021
10 tasks
@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 29, 2021
};

checkInputs = [ trio curio async-timeout ];

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pythonImportsCheck = [ "python_socks" ];

sha256 = "sha256-jeOLF7xDjfNu7erNVGzwUwOrhxhVrwZp/E6N9R8q35Q=";
};

doCheck = false; # tests fail in Nix sandbox
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does it fail?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I add keyring as a check input, it fails with the following. I don't understand the internals, but passing in xbacklight does not help.

@nix { "action": "setPhase", "phase": "setuptoolsCheckPhase" }
setuptoolsCheckPhase
Executing setuptoolsCheckPhase
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
writing plyer.egg-info/PKG-INFO
writing dependency_links to plyer.egg-info/dependency_links.txt
writing requirements to plyer.egg-info/requires.txt
writing top-level names to plyer.egg-info/top_level.txt
reading manifest file 'plyer.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*README.rst'
writing manifest file 'plyer.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.audio'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.barometer'
upower not found.Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.bluetooth'
sh: which: not found
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 94, in _ensure_obj
    obj = mod.instance()
  File "/build/plyer-2.0.0/plyer/platforms/linux/brightness.py", line 29, in instance
    return LinuxBrightness()
  File "/build/plyer-2.0.0/plyer/platforms/linux/brightness.py", line 18, in __init__
    raise Exception(msg)
Exception: It looks like 'xbacklight' is not installed. Try installing it with your distribution's package manager.
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.call'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.camera'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.compass'
xdg-open not found.Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.flash'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.gps'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.gravity'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.gyroscope'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.humidity'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.irblaster'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.light'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.proximity'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.sms'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.spatialorientation'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.stt'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.temperature'
lshw not found.Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
ModuleNotFoundError: No module named 'plyer.platforms.linux.vibrator'
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/platforms/linux/wifi.py", line 13, in <module>
    import wifi
ModuleNotFoundError: No module named 'wifi'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/utils.py", line 93, in _ensure_obj
    mod = __import__(module, fromlist='.')
  File "/build/plyer-2.0.0/plyer/platforms/linux/wifi.py", line 15, in <module>
    raise ModuleNotFoundError(
ModuleNotFoundError: python-wifi not installed. try: `pip install --user wifi`.
Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/platforms/win/uniqueid.py", line 6, in <module>
    import _winreg as regedit
ModuleNotFoundError: No module named '_winreg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/build/plyer-2.0.0/plyer/platforms/win/uniqueid.py", line 9, in <module>
    import winreg as regedit
ModuleNotFoundError: No module named 'winreg'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 56, in <module>
    setup(
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/command/test.py", line 232, in run
    self.run_tests()
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/command/test.py", line 250, in run_tests
    test = unittest.main(
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 405, in _find_tests
    tests, should_recurse = self._find_test_path(
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/command/test.py", line 50, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/command/test.py", line 50, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/z9rg0ff8qdmvg7v83fq0ixzp2vlpl3fl-python3.8-setuptools-50.3.1/lib/python3.8/site-packages/setuptools/command/test.py", line 50, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/yl69v76azrz4daiqksrhb8nnmdiqdjg9-python3-3.8.8/lib/python3.8/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/build/plyer-2.0.0/plyer/platforms/win/uniqueid.py", line 11, in <module>
    raise NotImplementedError()
NotImplementedError

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That does not look good. Did you check if the module still works?

"test_connect_wrapper"
# time dependent and flaky
"test_transfer_monitor_callbacks"
];

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
pythonImportsCheck = [ "nio" ];

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh one can have both normal checks and pythonImportsCheck? TIL

@siraben siraben closed this Jul 26, 2021
@vojta001 vojta001 deleted the mirage branch March 14, 2022 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants