Use syntax updated with Python 3.7 and avoid unnecessary list creations

[bryant1410]

I ran the following, which updates the code to syntax to Python >= 3.7:

pyupgrade $(find . -name "*.py" -type f) --py37-plus

This includes, for example, the Python-2 style of calling super and using f-strings where possible. I reviewed these changes manually.

Also, I looked for unnecessary list comprehension creations, where a generator expression could have been created instead, thus saving memory and compute (though small if the lists were small).

[Jasha10]

Thanks @bryant1410.
Overall it looks good, with a few issues:

[bryant1410]

BTW, the tests are failing for another reason, right?

[Jasha10]

BTW, the tests are failing for another reason, right?

I think so. The failing test should be fixed by PR #2677, which was recently merged into main. Could you please rebase this PR against the main branch?

[bryant1410]

I think so. The failing test should be fixed by PR #2677, which was recently merged into main. Could you please rebase this PR against the main branch?

Okay! I just merged instead of rebasing, hope it's fine. You can always rebase everything at the end, and I think it should remove the merge-type commits.

[bryant1410]

BTW, I see the latest in main is still failing. You sure the tests are gonna work?

[Jasha10]

Hmm, no. It seems there's an issue with the diff. See comment below:

[Jasha10]

Currently three CI jobs are failing on this branch:

• test_tools-3.10: caused by this diff's change to tools/configen/tests/test_modules/future_annotations.py.
• lint-3.10: seems unrelated to this diff (see [Bug] Failing lint: Need type annotation for "matches" #2693)
• test_core-3.10: seems unrelated to this diff (see [Bug] Failing tests due to DeprecationWarning: pkg_resources is deprecated as an API. #2692)

I recommend dropping this diff's change to tools/configen/tests/test_modules/future_annotations.py.

Output from the test_tools CI job:

nox > [2023-06-22 20:03:57,598] Running session test_tools-3.10
nox > [2023-06-22 20:03:57,599] Creating virtual environment (virtualenv) using python3.10 in .nox/test_tools-3-10
nox > [2023-06-22 20:03:57,968] python -m pip install --upgrade pip
nox > [2023-06-22 20:03:59,701] python -m pip install --upgrade setuptools
nox > [2023-06-22 20:04:01,110] python -m pip install pytest
nox > [2023-06-22 20:04:02,348] python -m pip install read-version
nox > [2023-06-22 20:04:02,961] cd /home/circleci/project
nox > [2023-06-22 20:04:02,962] pip install .
nox > [2023-06-22 20:04:07,666] pip list
Installed omegaconf version: omegaconf              2.3.0
nox > [2023-06-22 20:04:08,104] cd /home/circleci/project
nox > [2023-06-22 20:04:08,104] cd /home/circleci/project
nox > [2023-06-22 20:04:08,104] pip install -e tools/configen
nox > [2023-06-22 20:04:09,384] pytest tools/configen
============================= test session starts ==============================
platform linux -- Python 3.10.11, pytest-7.3.2, pluggy-1.2.0
rootdir: /home/circleci/project
configfile: pytest.ini
plugins: hydra-core-1.4.0.dev0
collecting ... 
collected 22 items                                                             

tools/configen/tests/test_generate.py .F....................             [100%]

=================================== FAILURES ===================================
________________________ test_generated_code_future_ann ________________________

    @mark.skipif(sys.version_info < (3, 7), reason="requires Python 3.7")
    def test_generated_code_future_ann() -> None:
        classes = ["ExampleClass"]
        expected_file = (
            Path(MODULE_NAME.replace(".", "/")) / "generated_future_annotations.py"
        )
        expected = expected_file.read_text()
    
        generated = generate_module(
            cfg=conf,
            module=ModuleConf(
                name=MODULE_NAME + ".future_annotations",
                classes=classes,
            ),
        )
>       _assert_expected_output(generated, expected, expected_file)

/home/circleci/project/tools/configen/tests/test_generate.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

generated = '# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved\n# Generated by configen, do not edit.\n# See ...] = field(default_factory=lambda: [])\n    default_str: Any = "Bond, James Bond"\n    none_str: Optional[str] = None\n'
expected = '# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved\n# Generated by configen, do not edit.\n# See ...] = field(default_factory=lambda: [])\n    default_str: Any = "Bond, James Bond"\n    none_str: Optional[str] = None\n'
expected_file = PosixPath('tests/test_modules/generated_future_annotations.py')

    def _assert_expected_output(generated: str, expected: str, expected_file: Path) -> None:
        lines = [
            line
            for line in unified_diff(
                expected.splitlines(),
                generated.splitlines(),
                fromfile=str(expected_file),
                tofile="Generated",
            )
        ]
    
        diff = "\n".join(lines)
        if generated != expected:
            print(diff)
>           assert False, f"Mismatch between {expected_file} and generated code"
E           AssertionError: Mismatch between tests/test_modules/generated_future_annotations.py and generated code
E           assert False

/home/circleci/project/tools/configen/tests/test_generate.py:122: AssertionError
----------------------------- Captured stdout call -----------------------------
--- tests/test_modules/generated_future_annotations.py

+++ Generated

@@ -17,9 +17,9 @@

 class ExampleClassConf:
     _target_: str = "tests.test_modules.future_annotations.ExampleClass"
     no_default: float = MISSING
-    lst: List[str] = MISSING
-    passthrough_list: Any = MISSING  # List[LibraryClass]
-    dataclass_val: List[User] = MISSING
-    def_value: List[str] = field(default_factory=lambda: [])
+    lst: list[str] = MISSING
+    passthrough_list: Any = MISSING  # list[LibraryClass]
+    dataclass_val: list[User] = MISSING
+    def_value: list[str] = field(default_factory=lambda: [])
     default_str: Any = "Bond, James Bond"
     none_str: Optional[str] = None
=========================== short test summary info ============================
FAILED tools/configen/tests/test_generate.py::test_generated_code_future_ann - AssertionError: Mismatch between tests/test_modules/generated_future_annotations.py and generated code
assert False
========================= 1 failed, 21 passed in 0.75s =========================
nox > [2023-06-22 20:04:10,842] Command pytest tools/configen failed with exit code 1
nox > [2023-06-22 20:04:10,842] Session test_tools-3.10 failed.

[Jasha10]

Looks good. Thanks @bryant1410!