Running scripts that have not been installed

[brechtm]

I can succesfully test my console scripts when running from a Tox environment. However, I would also like to run these tests outside when not using Tox. The readme is not clear about whether this is possible or not.

My project layout is as follows:

src/
  package_name/
    __main__.py
    ...
setup.py

My setup.py:

setup(
    ...
    entry_points={
        'console_scripts': [
            'mycmd = package_name.__main__:main',
        ],
    ...
)

When running pytests in a development environment, it is common to set PYTHONPATH=src. Is it possible to point pytest-console-scripts to setup.py somehow in this case so that it can find the console scripts?

[kvas-it]

Hi Brecht,

Unfortunately at the moment the running of scripts that are not installed is not supported. It seems in principle possible to implement it as you describe (parse or execute setup.py, understand where to find the scripts, load and invoke them) but it's also quite a bit of work and would be hard to make it reliable (because setup.py can be doing all kinds of things and we'll have to support/understand them).

If your use case for running the tests outside of Tox is similar to mine (quickly run one test file during development) then perhaps my solution will work for you. I install the package I'm working on into one of the Tox virtualenvs in development mode (python setup.py develop) and then work with that virtualenv activated (if can also be a separate virtualenv, not related to Tox). This way I can run all the tests, or just one test file and any changes I've recently made to the source are picked up.

Hopefully this helps,
Cheers,
Vasily

[brechtm]

Thanks for you reply, Vasily. Please consider mentioning this limitation in the README, along with your suggestion on how to work around this issue.

In case anyone wants to have a go at implementing this, here's a start:

from distutils.core import run_setup

distribution = run_setup('path/to/setup.py', stop_after='init')
distribution.entry_points['console_scripts']

[kvas-it]

Hi Brecht,

Thanks for the suggestions. I have documented the testing during development use case and added a reference to this ticket, in case someone would like to take a shot at it (unfortunately I'm not able to prioritize this at the moment).

I will rename the ticket to look like a new feature instead of a question. Feel free to rename to the original name if you prefer that or let me know if you'd like to change anything else.

Cheers,
Vasily

[hidr0]

Hello, I am kind of new to python and I could not exactly follow the thread.
Our situation is similar we have a console-script and I would like to test it. We are running multithreaded tests using pytest.
@brechtm are you suggesting that I run that before every test I would like to use script_runner on?

Do you know if there are limitations of running it multithreaded?

@kvas-it Could we with @brechtm somehow contribute to make it easier to test uninstalled scripts, because I think that this is the beauty of this tool? Testing uninstalled scripts without the hassle of installing them.

I would use subprocess if the script was installed and was not mine (not from the same package that I am testing).

[kvas-it]

Hi, Mihail!

Do you know if there are limitations of running it multithreaded?

script_runner fixture is function scoped, so it should work.

Could we with @brechtm somehow contribute to make it easier to test uninstalled scripts, because I think that this is the beauty of this tool? Testing uninstalled scripts without the hassle of installing them.

I think it should be possible to use the approach from @brechtm's comment. Based on some research into this direction that I've done, I would probably do something like this:

1. Add a load_setup_py(path_to_setup_py) method to pytest_console_scripts.ScriptRunner class. In it:
   i. Use distutils.core.run_setup to load the entry points from setup.py as @brechtm explained.
   ii. Use EntryPoint.parse_map from pgk_resources (see doc) to parse the entry points dict. Unfortunately we can't just pass our distribution object to it as a second parameter because it needs pkg_resources.Distribution and we have setuptools.dist.Distribution, which is a different thing. We can try to figure out a way to create a sufficiently working pkg_resources.Distribution so that created entry points would have a working .load() method. If that fails, we can just add the directory of setup.py to sys.path here and then do the loading ourselves (see 2).
   iii. Store console scripts in a dict called self.loaded_console_scripts or something like that.
2. ScriptRunner._load_script can now also take console scripts from loaded_console_scripts if available. If we figured out how to create a functional pkg_resources.Distribution in 1.ii, we just call entry_point.load() otherwise we can load the console script by __import__-ing the module and getting the entry point function out of it.
3. To make subprocess mode work, we can dynamically generate a temporary wrapper that does 2 and run that.
4. Add a test for this functionality: we can put setup.py and another python file that contains the entry point function into tmpdir and load_setup_py on it.

Since script_runner is function scoped, we'd need to load_setup_py in every test. This should not be too bad because parsing setup.py is fast and importing the actual entry point will only be done once anyway. However, it would be more elegant to have a singleton registry instead of loaded_console_scripts attribute on ScriptRunner instances. I would not worry about this for now and just try to implement 1-4 first (maybe even just 1, 2 and 4).

Bonus: handle package_dir argument to setup() (see doc) -- if the package is doing crazy path remapping, you might have to symlink everything into some temporary directory to make it work, but simple cases should be easy.

If this sounds like too much, let me know, I might have time to work on this in the coming weeks.

Cheers,
Vasily

[brechtm]

We can try to figure out a way to create a sufficiently working pkg_resources.Distribution so that created entry points would have a working .load() method.

From distribution.entry_points['console_scripts'] in my earlier comment, you should be able to dynamically create entry points. This isn't documented, but I think it's fine for use in tests. After this, you can load the entry points like you do for installed packages.

[HexDecimal]

I think it's too much effort to support something non-standard. This could technically be done but the end result would have drawbacks similar to a development install, and at that point it's hard to justify why a development install shouldn't be performed instead of trying to extract the entry points from the setup metadata.

For running pytest locally use pip install -e . to install the entry points. Do this instead of setup.py develop since pip works for alternative build tools.

I think any solution which depends on setup.py existing is a bad idea. There are other places that console_scripts entry points can be found, including in pyproject.toml. Not all Python projects have a setup.py file but all projects with entry points support pip install -e ..

[kvas-it]

Yeah, I agree with @HexDecimal. Now with greater proliferation of build tools supporting testing while not installed for all kinds of packages is a lot of work. pip install -e . seems like the right way to go.

@brechtm: would you be ok with updating the documentation and closing this ticket or do you think there's something else to be done?

[brechtm]

I agree that it doesn't make sense to implement something that only works for setup.py.

I don't have the time to spare currently, sorry. Please go ahead and close this issue.

[HexDecimal]

Sorry for disturbing you with my bike-shedding. I'll close this for now since it doesn't make much sense for newer packagers. If someone insisted then I might try to get the console scripts from pyproject.toml which is the most standard place to put them these days.