Move code out of pip._internal.__init__

[chrahunt]

Related to #4497.

Previously our main function and its associated imports were in pip._internal.__init__ - this makes them unconditional for any access of pip internals. Now main is in pip._internal.main.

This is relevant to:

1. Improving test execution time - some of our longest tests are in tests.unit.test_build_env, which starts subprocesses and import internals which may not require everything that was in __init__.
2. Checking module import time - when we import so much in __init__ it is difficult to use -X importtime to isolate slow-loading modules or modules that cause a lot of other transitive imports

A test on my machine shows that import of pip._internal previously took around 120ms before this change, and is now around 60ms.

[cjerdonek]

Is there any reason why __init__.py can't be empty?

[pradyunsg]

Is there any reason why init.py can't be empty?

There's warning silencing (for warnings from _vendor) we're doing in there that would need to happen before we load any vendored dependencies. I think __init__ is the best spot for doing that.

---

I wonder if there's merit in doing the following:

• pip._internal.cli.main_parser -> pip._internal.cli.main (rename)
• pip._internal.__init__:main -> pip._internal.cli.main:main (move, like this PR)

[cjerdonek]

There's warning silencing (for warnings from _vendor) we're doing in there that would need to happen before we load any vendored dependencies.

If __init__.py is empty though, we won't have loaded any vendored dependencies. My point is just that @chrahunt says that it's better if __init__.py imports more quickly. So why not make it import as quickly as possible by making it empty. How many ms would that be? 😛

[chrahunt]

I think it can be empty, but expect that moving the warning silencing will require a little more thought or discussion. I figured it was better to submit the obvious change alone.

It is definitely worth doing though, you can see these are not cheap imports.

[pradyunsg]

If __init__.py is empty though, we won't have loaded any vendored dependencies.

Right, but we'd want to silence them before loading them. As @chrahunt pointed out, it's best for doing in a follow up.

None the less, I'm not 100% sure if that's the best place to investigate but, hey, I don't mean to block anyone from making improvements. :)

[pradyunsg]

LGTM.

[pelson]

Just a heads up: This change will mean an update to ensurepip when they upgrade to pip>=19.3.* as ensurepip uses internal pip interfaces: https://github.com/python/cpython/blob/3.8/Lib/ensurepip/__init__.py#L28

[xavfernandez]

Thanks :), python/cpython#16782 has been created to tackle this.

[chadwhitacre]

This also broke (some invocations of?) virtualenv: pypa/virtualenv#1436.

Maybe Pip should promote main to an external API, since other projects seem to depend on it?