Release 0.36.1 breaks arcgis import

[martijnlbos]

Importing the arcgis package with version 0.36.1 of referencing installed doesn't work.

Previously, with version 0.35.1 installed, the command import arcgis worked. Now it returns the error below.

Pinning this version of referencing worked for us, but since doing a pip install arcgis automatically uses the latest version of referencing I thought it might be useful to report this behaviour.

Error for import arcgis with the latest version:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File <command-3302183912106387>, line 5
      2 get_ipython().run_line_magic('pip', 'install geopandas==0.14.0')
      4 import os
----> 5 import arcgis
      6 import shapely
      8 import geopandas as gpd

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/__init__.py:4
      1 __version__ = "1.9.1"
      3 from . import env
----> 4 from . import features, geoanalytics, geocoding, geometry
      5 from . import geoprocessing, network, raster, realtime, schematics
      6 import os

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/features/__init__.py:33
     31 from ._validation import ValidationManager
     32 from . import analyze_patterns
---> 33 from . import enrich_data
     34 from . import find_locations
     35 from . import manage_data

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/features/enrich_data.py:10
      1 """
      2 These functions help you explore the character of areas. Detailed demographic data and statistics are returned for your
      3 chosen areas.
   (...)
      6 travel time or distance from a location.
      7 """
      9 import arcgis as _arcgis
---> 10 import arcgis.network as network
     11 from .._impl.common._utils import inspect_function_inputs
     13 # --------------------------------------------------------------------------

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/network/__init__.py:7
      1 """
      2 The arcgis.network module contains classes and functions for network analysis. Network layers and analysis can be used
      3 for operations such as finding the closest facility, the best route for a vehicle, the best routes for a fleet of
      4 vehicles, locating facilities using location allocation, calculating an OD cost matrix, and generating service areas.
      5 """
----> 7 from ._layer import (
      8     NetworkLayer,
      9     NetworkDataset,
     10     ClosestFacilityLayer,
     11     ServiceAreaLayer,
     12     RouteLayer,
     13     NAJob,
     14     ODCostMatrixLayer,
     15 )
     17 from . import analysis

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/network/_layer.py:10
      8 from arcgis.features import FeatureLayer, FeatureLayerCollection, Table
      9 from arcgis.network import _utils
---> 10 from arcgis.mapping import MapImageLayer
     11 from arcgis._impl.common._utils import _validate_url
     13 try:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/mapping/__init__.py:8
      1 """
      2 The arcgis.mapping module provides components for visualizing GIS data and analysis.
      3 This module includes WebMap and WebScene components that enable 2D and 3D
      4 mapping and visualization in the GIS. This module also includes mapping layers like
      5 MapImageLayer, SceneLayer and VectorTileLayer.
      6 """
----> 8 from ._types import (
      9     WebMap,
     10     WebScene,
     11     MapImageLayer,
     12     MapImageLayerManager,
     13     VectorTileLayer,
     14     OfflineMapAreaManager,
     15     PackagingJob,
     16 )
     17 from ._types import SceneLayer
     18 from .forms import (
     19     FormFieldElement,
     20     FormExpressionInfo,
   (...)
     24     FormCollection,
     25 )

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/mapping/_types.py:30
     28 try:
     29     from traitlets import HasTraits, observe
---> 30     from arcgis.widgets._mapview._traitlets_extension import ObservableDict
     31 except ImportError:
     33     class HasTraits:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/__init__.py:2
      1 try:
----> 2     from arcgis.widgets._mapview import MapView
      3 except ImportError as e:
      4     import logging

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/_mapview/__init__.py:1
----> 1 from arcgis.widgets._mapview._mapview import MapView

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/_mapview/_mapview.py:31
     29 from arcgis.widgets._mapview._webscene_utils import DEFAULT_WEBSCENE_TEXT_PROPERTY
     30 from arcgis.widgets._mapview._loading_icon_str import _loading_icon_str
---> 31 from arcgis.widgets._mapview._raster import LocalRasterOverlayManager
     32 from arcgis.widgets._mapview._raster._numpy_utils import *
     33 from arcgis import __version__ as py_api_version

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/_mapview/_raster/__init__.py:1
----> 1 from arcgis.widgets._mapview._raster.local_raster_overlay_manager import (
      2     LocalRasterOverlayManager,
      3     RasterOverlay,
      4 )

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/_mapview/_raster/local_raster_overlay_manager.py:7
      5 from arcgis.widgets._mapview._raster import *
      6 from arcgis.widgets._mapview._raster._numpy_utils import *
----> 7 from arcgis.widgets._mapview._raster._jupyter_utils import *
     10 class RasterOverlay:
     11     def __init__(self, id_: str, img_url: str, extent: dict, opacity: float):

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/arcgis/widgets/_mapview/_raster/_jupyter_utils.py:10
      7 log = logging.getLogger()
      9 import ipykernel
---> 10 from notebook.notebookapp import list_running_servers
     11 from IPython.display import HTML, display
     14 def get_dir_of_curr_exec_notebook():

File /databricks/python/lib/python3.10/site-packages/notebook/notebookapp.py:79
     77 from .services.kernels.kernelmanager import MappingKernelManager, AsyncMappingKernelManager
     78 from .services.config import ConfigManager
---> 79 from .services.contents.manager import ContentsManager
     80 from .services.contents.filemanager import FileContentsManager
     81 from .services.contents.largefilemanager import LargeFileManager

File /databricks/python/lib/python3.10/site-packages/notebook/services/contents/manager.py:17
     15 from .checkpoints import Checkpoints
     16 from traitlets.config.configurable import LoggingConfigurable
---> 17 from nbformat import sign, validate as validate_nb, ValidationError
     18 from nbformat.v4 import new_notebook
     19 from ipython_genutils.importstring import import_item

File /databricks/python/lib/python3.10/site-packages/nbformat/__init__.py:11
      6 # Copyright (c) IPython Development Team.
      7 # Distributed under the terms of the Modified BSD License.
      9 from traitlets.log import get_logger
---> 11 from . import v1, v2, v3, v4
     12 from ._version import __version__, version_info
     13 from .sentinel import Sentinel

File /databricks/python/lib/python3.10/site-packages/nbformat/v4/__init__.py:39
     36 writes_json = writes
     37 to_notebook_json = to_notebook
---> 39 from .convert import downgrade, upgrade

File /databricks/python/lib/python3.10/site-packages/nbformat/v4/convert.py:13
      9 from traitlets.log import get_logger
     11 from nbformat import v3
---> 13 from .. import validator
     14 from .nbbase import NotebookNode, nbformat, nbformat_minor, random_cell_id
     17 def _warn_if_invalid(nb, version):

File /databricks/python/lib/python3.10/site-packages/nbformat/validator.py:15
     13 from ._imports import import_item
     14 from .corpus.words import generate_corpus_id
---> 15 from .json_compat import ValidationError, _validator_for_name, get_current_validator
     16 from .reader import get_version
     17 from .warnings import DuplicateCellId, MissingIDFieldWarning

File /databricks/python/lib/python3.10/site-packages/nbformat/json_compat.py:11
      8 import os
     10 import fastjsonschema
---> 11 import jsonschema
     12 from fastjsonschema import JsonSchemaException as _JsonSchemaException
     13 from jsonschema import Draft4Validator as _JsonSchemaValidator

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/jsonschema/__init__.py:13
      1 """
      2 An implementation of JSON Schema for Python.
      3 
   (...)
      9 for you.
     10 """
     11 import warnings
---> 13 from jsonschema._format import FormatChecker
     14 from jsonschema._types import TypeChecker
     15 from jsonschema.exceptions import SchemaError, ValidationError

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/jsonschema/_format.py:11
      8 import typing
      9 import warnings
---> 11 from jsonschema.exceptions import FormatError
     13 _FormatCheckCallable = typing.Callable[[object], bool]
     14 #: A format checker callable.

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/jsonschema/exceptions.py:15
     12 import warnings
     14 from attrs import define
---> 15 from referencing.exceptions import Unresolvable as _Unresolvable
     17 from jsonschema import _utils
     19 if TYPE_CHECKING:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/referencing/__init__.py:5
      1 """
      2 Cross-specification, implementation-agnostic JSON referencing.
      3 """
----> 5 from referencing._core import Anchor, Registry, Resource, Specification
      7 __all__ = ["Anchor", "Registry", "Resource", "Specification"]

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/referencing/_core.py:18
     16 from referencing import exceptions
     17 from referencing._attrs import frozen
---> 18 from referencing.typing import URI, Anchor as AnchorType, D, Mapping, Retrieve
     20 EMPTY_UNCRAWLED: HashTrieSet[URI] = HashTrieSet()
     21 EMPTY_PREVIOUS_RESOLVERS: List[URI] = List()

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-7e2a9c4d-f1a6-4be0-a937-65fad8355043/lib/python3.10/site-packages/referencing/typing.py:22
     19 URI = str
     21 #: The type of documents within a registry.
---> 22 D = TypeVar("D", default=Any)
     25 class Retrieve(Protocol[D]):
     26     """
     27     A retrieval callable, usable within a `Registry` for resource retrieval.
     28 
     29     Does not make assumptions about where the resource might be coming from.
     30     """

TypeError: TypeVar.__init__() got an unexpected keyword argument 'default'

[ssbarnea]

@martijnlbos I have the impression that 0.35.2 does not have this problem.

[Julian]

I haven't really had time to look at this but a minimal reproducer would definitely help in making time -- specifically I can't reproduce when simply installing the package (nor of course can CI here) -- specifically, even on 3.10:

uv run --python 3.10 --with referencing python -c 'import referencing.typing'

has no errors.

And we do already pin to a version of typing-extensions which has support for these.

So yeah, example certainly welcome!

[fossdd]

A simple reproducer would be:

➜  ~ podman run -it --rm alpine:edge sh -c 'apk add py3-referencing; python'       
fetch https://dl-cdn.alpinelinux.org/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/edge/community/x86_64/APKINDEX.tar.gz
[...]
(23/23) Installing py3-referencing (0.36.2-r0)
Executing busybox-1.37.0-r10.trigger
OK: 47 MiB in 38 packages
Python 3.12.8 (main, Dec  7 2024, 04:56:22) [GCC 14.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import referencing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.12/site-packages/referencing/__init__.py", line 5, in <module>
    from referencing._core import Anchor, Registry, Resource, Specification
  File "/usr/lib/python3.12/site-packages/referencing/_core.py", line 18, in <module>
    from referencing.typing import URI, Anchor as AnchorType, D, Mapping, Retrieve
  File "/usr/lib/python3.12/site-packages/referencing/typing.py", line 22, in <module>
    D = TypeVar("D", default=Any)
        ^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'default' is an invalid keyword argument for typevar()
>>> 

[Julian]

That would sound like an alpine packaging bug, which doesn't appear to be installing the typing-extensions dependency.

[Julian]

Going to close this as as far as I can see this was a bug downstream in Alpine which it seems has now been fixed but if someone has a reproducer in some other environment feel free to share.

[fossdd]

Thank you very much for the quick information. That fixed the issue for Alpine edge.

[martijnlbos]

Apologies, I probably reported this in the wrong place, and/or am misunderstanding my issue.

I'm still experiencing the same behaviour, but I don't know how to provide a minimal reproducer since I am working on a Databricks platform

I am using their 13.3 LTS compute, which seems to use Ubuntu 22.04.2 LTS, Apache Spark 3.4.1, and Python 3.10.12 if that helps, and then I am installing arcgis==1.9.1 and importing it.

[Julian]

I'd ask for support from Databricks perhaps in that case.

Don't take this as being rude hopefully, it's not at all meant to be! -- but I don't do unpaid support for paid products, and it would seem initially quite likely that the issue is on Databricks's side (or your installing within Databricks rather than via normal means). Again no offense meant, just being realistic that I won't look at this if the only way to reproduce is in a commercial product. I'm of course happy to do paid support myself if you need it, or to come back to this if anyone else experiences the issue in some usual environment.

[martijnlbos]

@Julian No offence taken, thanks for the reply!

I was just reporting this because I initially thought I came across an issue in an open source project -and since we're using this freely- the least I can do is report it.

So thanks for looking at this and pointing out the problem is most likely on the Databricks platform side!