Fix building with pip install

[hoxbro]

Same as #578, but moved the branch to the main repo and away from my fork.

[maximlt]

I've had to re-add the 'build' extra group in the setup.py file, it's needed by the conda recipe:

geoviews/conda.recipe/core/meta.yaml

Lines 23 to 25 in 9378694

	host:
	- python {{ sdata['python_requires'] }}
	{% for dep in sdata['extras_require']['build'] %}

I imagine there's a way for conda-build to pull the dependencies from the pyproject.toml file, I'd keep that for some later work on updating pyctdev.

I ran the docs and build workflows from this PR and they ran fine, merging then, thanks Simon!