Python 3.11 compatibility

[saitcakmak]

Python 3.11 does not support mutable defaults in dataclasses. Role uses a mutable default for resource, which leads to a ValueError when importing torchx in Python 3.11. This prevents downstream packages from supporting Python 3.11. Example failure: https://github.com/facebook/Ax/actions/runs/5662890368/job/15343645668?pr=1744

Possible fixes:

• Make Resource a frozen dataclass, which would make it immutable, likely fixing the issue.
• Set the default value to MISSING and set the current default in a __post_init__ method.

[bryantbiggs]

related pytorch/pytorch#91795

[henrylhtsang]

Any update on this?

[kiukchung]

#768 oughta fix it.

[kiukchung]

@saitcakmak , @bryantbiggs could you guys try out torchx-nightly==2023.09.19 see if that fixes your issue?

[saitcakmak]

Hi @kiukchung. I am still getting an error with Py3.11.

(py311) saitcakmak@saitcakmak-mbp ~ % pip install torchx-nightly==2023.9.19
...
(py311) saitcakmak@saitcakmak-mbp ~ % python
Python 3.11.4 (main, Jul  5 2023, 08:40:20) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchx
>>> from torchx.specs.api import Role
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/torchx/specs/__init__.py", line 16, in <module>
    from torchx.specs.named_resources_aws import NAMED_RESOURCES as AWS_NAMED_RESOURCES
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/torchx/specs/named_resources_aws.py", line 34, in <module>
    from torchx.specs.api import Resource
  File "/opt/anaconda3/envs/py311/lib/python3.11/site-packages/torchx/specs/api.py", line 272, in <module>
    @dataclass
     ^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/dataclasses.py", line 1230, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/dataclasses.py", line 1220, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/anaconda3/envs/py311/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'torchx.specs.api.Resource'> for field resource is not allowed: use default_factory
>>> 

[saitcakmak]

Actually, it works if I install from GitHub directly using pip install --upgrade git+https://github.com/pytorch/torchx.git. My previous pip install ... did not work for some reason, I think.

(py311) saitcakmak@saitcakmak-mbp ~ % python                                                         
Python 3.11.4 (main, Jul  5 2023, 08:40:20) [Clang 14.0.6 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from torchx.specs.api import Role
>>> from torchx.specs.api import Role
>>> Role()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Role.__init__() missing 2 required positional arguments: 'name' and 'image'

[kiukchung]

@saitcakmak ah, I might have merged before after the 2023.09.19 nightly was pushed. My PR should be included in tonight's nightly release (torchx-nightly==2023.09.20). Let me know if 2023.09.20 doesn't work and I'll look into it. Thanks!

[kiukchung]

@saitcakmak I validated that torchx-nightly==2023.09.20 works with python 3.11.4

[saitcakmak]

Thanks @kiukchung! I'll move Ax CI to Python 3.11 after the next torchx release.