Re-implement python-wrapper with buildEnv.

[peti]

The new wrapper creates an environment that contains all files from
Python and the extra libraries that have been specified. All files are
found at run-time by means of the $PYTHONHOME variable; the wrapper no
longer uses $PYTHONPATH.

[domenkozar]

Interesting, what's the motivation behind this change?

[garbas]

not sure i like this idea. this will make things impure right?

[peti]

@iElectric, I override pythonFull with the ldap module added into the search path on my machine, but the ldap module doesn't show up anywhere in my profile. That's what I wanted to fix. Any executabes, libraries, headers, or man pages Python packages might ship won't be visible to the user with the python-wrapper we currently have.

@garbas, I'm not sure what you mean by "impure" in this context. Could you please explain your concern in a little more detail?

[domenkozar]

Not really sure how that works, but you would override pythonFull in your nix configuration to add extraLibs? Isn't that possible also at the moment?

What we are trying to avoid is pythonFull picking up modules automatically from the system.

[garbas]

by impure i mean that you will install pythonFull and at first everything is going to work. until something is added to pythonhome and then its stops working or worse it works differently then when you install it.

i agree with @iElectric here. we should avoid picking up modules automatically from the system.

[peti]

Python does never pick up any modules from the system because of this patch. There is no reason to worry about that.

[peti]

 > by impure i mean that you will install pythonFull and at first everything is going to work. 
 > until something is added to pythonhome and then its stops working or worse it works 
 > differently then when you install it.

This cannot happen. PYTHONHOME points to an immutable store path; nothing is ever going to get added there after the derivation has been built.

[edolstra]

You may want to use propagatedUserEnvPackages instead of buildEnv. That allows the wrapper to propagate packages into the user environment without needing an additional symlink tree.

[bjornfor]

Is this a supplement or replacement for #792 ?

[peti]

#792 and this patch are orthogonal. There is no relationship between those two.

[peti]

@edolstra, the nice thing about buildEnv is that I can set $PYTHONHOME to $out to make Python find all libraries that are part of the collection. If I use propagatedUserEnvPackages instead, that won't be possible any more, right?

[peti]

FYI: I plan to merge this patch in a day or two.

[garbas]

@peti i'll give a try tonight ... to test that it still works with my existing environments

[garbas]

@peti

this is usual dev.nix file i use to create develpment environment now

{ }:

let
  pkgs = import <nixpkgs> { };
in

with pkgs;

buildEnv {
  name = "buildout-plone-env";
  paths = [
    python27Full
     python27Packages.virtualenv
     python27Packages."zc.buildout-1.7.1"
     python27Packages."Plone-4.3.2"
  ];
}

and then i run nix-build --out-link nixenv dev.nix

when using your pull request i'm getting the following error

NIX_PATH=/home/rok/dev/nixos nix-build --out-link nixenv dev.nix
these derivations will be built:
  /nix/store/jkr675rmk61kwhhcm7lxn4igivhgwm56-buildout-deco-env.drv
  /nix/store/xkdgs034yd8ij8q95nj1ksgcl4k6kv2m-python-2.7.5-wrapper.drv
building path(s) `/nix/store/pna7nshcga4y7c13v4vz3xrv9bdlldn1-python-2.7.5-wrapper'
building /nix/store/pna7nshcga4y7c13v4vz3xrv9bdlldn1-python-2.7.5-wrapper
created 660 symlinks in user environment
2to3 --> /nix/store/pna7nshcga4y7c13v4vz3xrv9bdlldn1-python-2.7.5-wrapper/bin/2to3
rm: cannot remove '/nix/store/pna7nshcga4y7c13v4vz3xrv9bdlldn1-python-2.7.5-wrapper/bin/2to3': Permission denied
builder for `/nix/store/xkdgs034yd8ij8q95nj1ksgcl4k6kv2m-python-2.7.5-wrapper.drv' failed with exit code 1
cannot build derivation `/nix/store/jkr675rmk61kwhhcm7lxn4igivhgwm56-buildout-deco-env.drv': 1 dependencies couldn't be built
error: build of `/nix/store/jkr675rmk61kwhhcm7lxn4igivhgwm56-buildout-deco-env.drv' failed
make: *** [bootstrap] Error 1

i'm not quite sure how should i change buildout-plone-env nix expression to extend and use wrapper with buildEnv. i've tried few things but failed, so maybe you already know.

i would really appreciate if you don't merge this until we find some solution for above use-case. i guess also others python developers are using similar way of creating their development environment for their projects.

monday and tuesday i'll be travelling to brasil and i'm not sure how much i will be online, but i'll try to reply asap when i get online.

[peti]

@garbas, I cannot re-produce the error you reported. This is a build of your expression using the current master branch:

$ git checkout master
Switched to branch 'master'
$ nix-build -o /tmp/test ~/test.nix -I nixpkgs=/home/simons/.nix-defexpr
/nix/store/9biw8xbik5na04i3lq2pd44r5zq89qw8-buildout-plone-env

And this is the same build run in the current new-python-wrapper branch:

$ git checkout new-python-wrapper 
Switched to branch 'new-python-wrapper'
$ nix-build -o /tmp/test ~/test.nix -I nixpkgs=/home/simons/.nix-defexpr
/nix/store/x0hz6b0wh1rpd25lcfhydw38lw85agd3-buildout-plone-env

Both work fine.

What I find strange about your build log is that the expression buildout-plone-env doesn't even show up in the list of things that Nix attempts to build! Is it possible that there was some kind of mix-up when you ran those tests?

[peti]

@garbas, do you still have concerns about this update?

[garbas]

not sure why it was not working for me that day. works now.
just to be sure i'm also testing zope2 service which i added yesterday. will report on this but it will take some time to finish, but will report back asap

[peti]

@garbas, since this patch has been in the queue for 2+ months, I think it's okay to merge it now and deal with issues that might arise once they do arise. If you run into any problems, please let me know!

[shlevy]

Hi @peti, this seems to have broken the build of pythonFull and python26Full. Investigating now, but if you have a quick fix that'd be great :)

[peti]

@shlevy, pythonFull seems to work fine for me:

$ nix-env -iA pythonFull
installing `python-2.7.5-wrapper'
these derivations will be built:
  /nix/store/qpn4rfldxz1inkdm0m0q0vyl6cjy05mf-python-2.7.5-wrapper.drv
building path(s) `/nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper'
building /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper
created 683 symlinks in user environment
2to3 --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/2to3
idle --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/idle
pdb --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/pdb
pdb2.7 --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/pdb2.7
pydoc --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/pydoc
python --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python
python-config --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python-config
python2 --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python2
python2-config --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python2-config
python2.7 --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python2.7
python2.7-config --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/python2.7-config
smtpd.py --> /nix/store/bcsp31d6n4sg1a4iplf38wkj5gw04xcd-python-2.7.5-wrapper/bin/smtpd.py
building path(s) `/nix/store/plsdxfcldnjnzx4f22nh557zh9ai4gvy-user-environment'
created 4 symlinks in user environment

You are right, though, that there might be a problem with python26Full. I'm not sure yet what's causing it. I'll take a look ...

[shlevy]

So it turns out that buildEnv just symlinks $out/bin to ${python}/bin since python is the only dependency with a bin directory, causing the problem. I'll fix it by unlinking $out/bin iff it's a symlink, but I'm not sure that's the best option here.

[shlevy]

$ nix-instantiate -A pythonFull
warning: you did not specify `--add-root'; the result might be removed by the garbage collector
/nix/store/szx7rlarr6yl1dmgc7ld22aj93490yha-python-2.7.5-wrapper.drv

You seem to get a different path, are you using a different nixpkgs/do you have a config.nix?

[peti]

@shlevy , yes, you are right -- that is the problem. Python27Full comes with an additional f2py2.7 on my system, because I have numpy overridden into the full expression. Python26 doesn't have that, hence the failure. Your fix sounds right, though. If bin is a symlink, then we can just delete it and replace it with an empty directory. If bin is a real directory, then we replace each individual executable with a wrapper script.

If you have such a patch already, please commit it!

[cillianderoiste]

I'm having trouble with myEnvFun environments since this merge.

plone43 = pkgs.myEnvFun {
  name = "plone43";
  buildInputs = with pkgs; [
    python27Packages.Plone pythonFull python27Packages.zc_buildout_nix stdenv
  ];

};

at be67ab5, after loading the env I get:

plone43:[goibhniu@goibhniu:~/Projects/Plone/plone.chat]$ python -c "import sys; print len(sys.path)"
537

after f6e835a I get:

plone43:[goibhniu@goibhniu:~/Projects/Plone/plone.chat]$ python -c "import sys; print len(sys.path)"
59

So, none of the packages in buildInputs or their dependencies are available from the environment.

In the earlier revision I noticed that after I run load-env-plone43 I already have all the python modules for that env in $PYTHONPATH, but this is not the case after the merge.

@peti do I just need to adjust my myEnvFun expression, or do you know some other solution?

[cillianderoiste]

It confuses matters that I had set PYTHONPATH to /run/current-system/sw/lib/python2.7/site-packages, but now I have unset PYTHONPATH before loading the env, and I have 489 modules in my sys.path for the old version, and 8 in the new version. I'll have a look at myEnvFun to see if I can figure out what's happening.

[cillianderoiste]

I think I get it now:

pythonPlone = pkgs.pythonFull.override {
   extraLibs = with pkgs.python27Packages; [ Plone ];
};

plone43 = pkgs.myEnvFun {
  name = "plone43-new";
  buildInputs = with pkgs; [ pythonPlone python27Packages.zc_buildout_nix ];
};

Which is very nice IMO, and if I'm not mistaken, it's way way faster to start up python too! Thanks @peti!

[domenkozar]

Could you also add that to wiki/manual?

[cillianderoiste]

For sure, I'm just doing some testing now, I think I need to work on zc_buildout_nix a bit.

[cillianderoiste]

I guess it's a bit of a hack, but to share the same modules with other python programs included in myEnvFun, you can set PYTHONHOME in extraCmds:

plone43 = pkgs.myEnvFun {
  name = "plone43";
  buildInputs = with pkgs; [ 
    pythonPlone 
    pkgs.python27Packages.zc_buildout_nix
    pkgs.python27Packages.ipython
  ];
  extraCmds = ''
    export PYTHONHOME=${pythonPlone}
  '';
};

In this example, buildout-nix will pick up all modules it finds in the environment (and create symlinks to them in eggs/), and ipython will also have access to the same modules.

Otherwise, I don't know how to handle python programs such as IPython. @bjornfor you might be interested to look into this too.

[cillianderoiste]

@iElectric I've updated the relevant section of the wiki:

https://nixos.org/wiki/Plone_development#How_to_use_myEnvFun_and_buildout-nix_to_create_a_plone_development_environment

[peti]

The setting

extraCmds = ''
  export PYTHONHOME=${pythonPlone}
'';

looks odd to me. Python-wrapper sets the $PYTHONHOME variable, so every package that uses the correct python interpreter should have it automaticallly. If there are packages that the python-wrapper cannot find, then they should probably be added to the wrapper's extraLibs argument. Am I missing something?

[cillianderoiste]

Hmm, looking at the wrapped ipython I see that it uses the python binary directly rather than the wrapped version, is that the problem?

$ head -1 /nix/store/cgc0vcmmzqv6z9pw6kijkdyyw0v7mpxw-ipython-1.1.0/bin/.ipython-wrapped
#!/nix/store/1i5rd0mp4x57gzjii9z2pg3mjgzj95q6-python-2.7.5/bin/python2.7

Do we need to modify wrapPythonPrograms: https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/python-modules/generic/wrap.sh#L8 or should that already pick up the wrapped python?

[peti]

I've never used ipython and I don't know much about it, but I guess
that, yes, it would be better for ipython to refer to the wrapper script
instead of the unwrapped python binary. Otherwise, ipython won't know
about the extraLibs that have been configured in the wrapper's
environment.

[cillianderoiste]

I guess we should fix wrapPythonPrograms to do this for all python console scripts then, or perhaps there is some way to convince type -P python to pick up the wrapped python? I'm OK with exporting PYTHONHOME for now, unless you happen to know of a nice solution.