Micromamba server

[wolfv]

So we can resolve more than 1 environment in a row

TODO:

• kick out virtual packages from Pool / kick out previous "installed" repo

[romain-intel]

Very interested in this :). Is the plan for this to also support virtual packages (it seems they are passed in but not used)?

[wolfv]

yeah, definitely.

[wolfv]

@romain-intel the virtual packages are now correctly being used :)

[romain-intel]

whoohoo. Thanks a ton. I am going to try this out now. I see the binaries are built (at least one linux).

[romain-intel]

I am probably missing something stupid but I launch the server with the latest binaries built on linux:

(metaflow-arti-split) --($:~/flows)---$ curl -vvv http://127.0.0.1:5000/hello
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET /hello HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.58.0
> Accept: */*
>

On the server side, I do see:

[08:34:44] 127.0.0.1:48900 - GET /hello 200 (took 0 ms)

but curl just hangs.

Trying an actual solve, I have:

curl -X POST -H 'Content-Type: application/json' -d'{"specs": ["python==3.7.12", "requests==>=2.21.0", "boto3==>=1.14.0", "pyarrow==>=0.17.1", "pandas==>=0.24.0", "cffi==<1.15.0"], "platform": "linux-64", "virtual_packages": ["__linux=5.4.0=0", "__glibc=2.27=0", "__unix=0=0", "__archspec=1=x86_64"], "channels": ["conda"]}' http://127.0.0.1:5000/solve

which does a bunch of stuff (but also caused curl to hang -- no surprise there if it hangs on a simple hello) but also gives this in the server (which seems a bit bad since it does not seem to solve it:

info     libmamba Parsing MatchSpec python==3.7.12
info     libmamba Parsing MatchSpec requests==>=2.21.0
info     libmamba Parsing MatchSpec boto3==>=1.14.0
info     libmamba Parsing MatchSpec pyarrow==>=0.17.1
info     libmamba Parsing MatchSpec pandas==>=0.24.0
info     libmamba Parsing MatchSpec cffi==<1.15.0
debug    libmamba Adding virtual package: __linux=5.4.0=0
debug    libmamba Adding virtual package: __glibc=2.27=0
debug    libmamba Adding virtual package: __unix=0=0
debug    libmamba Adding virtual package: __archspec=1=x86_64
info     libmamba Adding package record to repo __archspec
info     libmamba Adding package record to repo __glibc
info     libmamba Adding package record to repo __linux
info     libmamba Adding package record to repo __unix
info     libsolv  number of solvables: 19, memory used: 1 K
info     libsolv  number of ids: 273 + 38
info     libsolv  string memory used: 1 K array + 4 K data,  rel memory used: 0 K array
info     libsolv  string hash memory: 32 K, rel hash memory : 16 K
info     libsolv  provide ids: 7
info     libsolv  provide space needed: 28 + 256
info     libsolv  shrunk whatprovidesdata from 28 to 28
info     libsolv  shrunk whatprovidesauxdata from 28 to 18
info     libsolv  whatprovides memory used: 5 K id array, 1 K data
info     libsolv  whatprovidesaux memory used: 1 K id array, 0 K data
info     libsolv  createwhatprovides took 0 ms
info     libmamba Parsing MatchSpec python==3.7.12
info     libmamba Parsing MatchSpec python==3.7.12
info     libmamba Adding job: python 3.7.12
info     libmamba Parsing MatchSpec requests==>=2.21.0
info     libmamba Parsing MatchSpec requests==>=2.21.0
info     libmamba Adding job: requests >=2.21.0
info     libmamba Parsing MatchSpec boto3==>=1.14.0
info     libmamba Parsing MatchSpec boto3==>=1.14.0
info     libmamba Adding job: boto3 >=1.14.0
info     libmamba Parsing MatchSpec pyarrow==>=0.17.1
info     libmamba Parsing MatchSpec pyarrow==>=0.17.1
info     libmamba Adding job: pyarrow >=0.17.1
info     libmamba Parsing MatchSpec pandas==>=0.24.0
info     libmamba Parsing MatchSpec pandas==>=0.24.0
info     libmamba Adding job: pandas >=0.24.0
info     libmamba Parsing MatchSpec cffi==<1.15.0
info     libmamba Parsing MatchSpec cffi==<1.15.0
info     libmamba Adding job: cffi <1.15.0
info     libsolv  solver started
info     libsolv  dosplitprovides=0, noupdateprovide=0, noinfarchcheck=0
info     libsolv  allowuninstall=1, allowdowngrade=0, allownamechange=1, allowarchchange=0, allowvendorchange=0
info     libsolv  dupallowdowngrade=1, dupallownamechange=1, dupallowarchchange=1, dupallowvendorchange=1
info     libsolv  promoteepoch=0, forbidselfconflicts=0
info     libsolv  obsoleteusesprovides=0, implicitobsoleteusesprovides=0, obsoleteusescolors=0, implicitobsoleteusescolors=0
info     libsolv  dontinstallrecommended=0, addalreadyrecommended=0 onlynamespacerecommended=0
info     libsolv  obsoletes data: 1 entries
info     libsolv  added 0 pkg rules for installed solvables
info     libsolv  added 0 pkg rules for updaters of installed solvables
info     libsolv  added 0 pkg rules for packages involved in a job
info     libsolv  added 0 pkg rules because of weak dependencies
info     libsolv  5 of 18 installable solvables considered for solving
info     libsolv  pkg rule memory used: 0 K
info     libsolv  pkg rule creation took 1 ms
info     libsolv  job: install providing python 3.7.12
info     libsolv    - job Rule #9:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  job: install providing requests >=2.21.0
info     libsolv    - job Rule #10:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  job: install providing boto3 >=1.14.0
info     libsolv    - job Rule #11:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  job: install providing pyarrow >=0.17.1
info     libsolv    - job Rule #12:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  job: install providing pandas >=0.24.0
info     libsolv    - job Rule #13:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  job: install providing cffi <1.15.0
info     libsolv    - job Rule #14:
info     libsolv      !system:system.any [1] (w1)
info     libsolv      next rules: 0 0
info     libsolv  choice rule creation took 0 ms
info     libsolv  0 pkg rules, 2 * 4 update rules, 6 job rules, 0 infarch rules, 0 dup rules, 0 choice rules, 0 best rules, 0 yumobs rules
info     libsolv  0 black rules, 0 recommends rules, 0 repo priority rules
info     libsolv  overall rule memory used: 0 K
info     libsolv  solving...
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #9
info     libsolv  JOB Rule #9:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #10
info     libsolv  JOB Rule #10:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #11
info     libsolv  JOB Rule #11:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #12
info     libsolv  JOB Rule #12:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #13
info     libsolv  JOB Rule #13:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  ANALYZE UNSOLVABLE ASSERTION ----------------------
info     libsolv  conflict with system solvable, disabling rule #14
info     libsolv  JOB Rule #14:
info     libsolv      !system:system.any [1] (w1) Install.level1
info     libsolv      next rules: 0 0
info     libsolv  resolving job rules
info     libsolv  resolving installed packages
info     libsolv  deciding unresolved rules
info     libsolv  installing recommended packages
info     libsolv  deciding orphaned packages
info     libsolv  solver statistics: 0 learned rules, 0 unsolvable, 0 minimization steps
info     libsolv  done solving.

info     libsolv  solver took 1 ms
info     libsolv  final solver statistics: 6 problems, 0 learned rules, 0 unsolvable
info     libsolv  solver_solve took 2 ms
info     libmamba Problem count: 6
info     libmamba Freeing solver.
[08:33:14] 127.0.0.1:48684 - POST /solve 200 (took 2 ms)

I know, I know, I am probably a little too excited so feel free to ignore this but if there is a simple fix, would love to try it out :).

[wolfv]

Awesome, thanks for testing. I was actually only testing on macOS arm64 so far :/ maybe there is a difference in how things work in linux!

Also, how are you starting the binary? I am just using micromamba server. If you then navigate to localhost:1234 you should see the micromamba ascii art...

[wolfv]

btw. here is the testing script i was using to run a bunch of solves: https://gist.github.com/wolfv/874d4c83e69c43cddfebc15160542570

[wolfv]

I've tried more on Linux as well, and couldn't reproduce your issue (Fedora 37, x64)

[romain-intel]

Hey @wolfv , thanks for the speedy response. I was trying it out similarly (using micromamba server -p 5000 but I'll also try with the default 1234 port). I'll retry with the latest builds (from 668 which is the latest successful) with your script. This is on an ubuntu box although that should not make much difference. I'll try this later this morning.

Thanks!

[romain-intel]

whoohoo, build in 668 works fine and I am able to get the results back.