Add mypy_types benchmark

[Fidget-Spinner]

Fixes #35. Ported from Pyston.

[Fidget-Spinner]

Huh, I forgot that mypy doesn't support pypy due to dependence on CPython AST. I wonder if there's a way to skip installation of mypy and this specific benchmark just for pypy?

[vstinner]

Can you try not not add the benchmark to DEFAULT_GROUP if you are running PyPy?

[Fidget-Spinner]

Yay the tests are finally passing! This should ready for review now. Please take a look when you have the time Victor.

[gvanrossum]

LGTM. @ericsnowcurrently take note!

[Fidget-Spinner]

Recently I remembered mypy uses mypyc for speedups. I'm inclined to pass --no-binary to pip so that the benchmark runs more python code (and less C extension code). WDYT?

Though I'm not sure how to pass in that argument just for mypy. Seems like pip-compile will add the argument to all other packages, which will affect django among others.

[gvanrossum]

I just was reminded of this PR again. Do we still need to do this when we have Eric's more general #109?

Recently I remembered mypy uses mypyc for speedups. I'm inclined to pass --no-binary to pip so that the benchmark runs more python code (and less C extension code). WDYT?

Oh, definitely. Benchmarking mypy with mypyc's speedups says more about mypyc than about the Python interpreter used to run it.

Though I'm not sure how to pass in that argument just for mypy. Seems like pip-compile will add the argument to all other packages, which will affect django among others.

Maybe #109 will help there too? It has a separate toml file per benchmark. I believe Eric also has prepared mods for the Pyston benchmarks that makes them runnable with his new infrastructure; I think it would be a shame if the Pyston benchmarks were forked. We should probably talk to the Pyston team about this before proceeding (@kmod, @undingen).

[pablogsal]

This is probably going to pick up the extension if the wheel is available and the pure python if not, which may give different numbers in different versions. Am I mistaken? Maybe I am missing something.

Edit: I saw this is being discussed already on the PR.

[Fidget-Spinner]

I just was reminded of this PR again. Do we still need to do this when we have Eric's more general #109?

Eric's solution is definitely technically superior (I'm a big fan of his work in that PR). This PR still serves a purpose in adding mypy to the default list of benchmarks in pyperformance. Which I'd assume shows up on speed.python.org (maybe?).

I'm not sure how speed.python.org's pyperformance is updated. If it's Pablo/Victor/Zach/Dong-hee manually updating the pyperformance in there, we can always add a curated list of additional benchmarks to run via Eric's work -- mypy being one of them. Then I can just close this PR.

[vstinner]

This is probably going to pick up the extension if the wheel is available and the pure python if not, which may give different numbers in different versions. Am I mistaken? Maybe I am missing something.

If you really don't want to measure the C implementation of mypy, the benchmark should have an option to decide which implementations is taken. See for example pyperformance/benchmarks/bm_pickle.py which implements such option and raises an error if it gets the wrong implementation.

[kmod]

fwiw in Pyston we run the benchmark without mypyc, but get that not 100% by choice because only the prebuilt mypy wheels are built with mypyc and building a fresh wheel yourself (like we need to do) will not use it.

[ericsnowcurrently]

FYI, I've merged my PR that allows running benchmarks that aren't part of pyperformance. I also have a PR up against the Pyston benchmarks repo to allow them to be run using pyperformance: pyston/python-macrobenchmarks#3.

[Fidget-Spinner]

@ericsnowcurrently awesome! I'll close this PR then since Pyston has the same.