Move the macOS ARM64 build job into the existing test workflows. #15805

benjyw · 2022-06-11T19:16:26Z

Also ensure it runs specifically on macOS 11.

Also build for arm64, not arm64e, as the latter isn't intended
for non-Apple-provided binaries yet (at least on macOS 11).

Also some further refactoring to facilitate qualifying cache keys
by system version.

Also fix some unrelated docstrings that didn't justify their own change.

benjyw · 2022-06-11T19:31:07Z

Will figure out what is misconfigured with the new macOS 11 runner I just set up, and is apparently not working properly.

benjyw · 2022-06-12T14:20:31Z

OK, this is ready for review. Thanks!

Also ensure it runs specifically on macOS 11. Also build for arm64, not arm64e, as the latter isn't intended for non-Apple-provided binaries yet (at least not on macOS 11). Also some further refactoring to facilitate qualifying cache keys by system version. Also fix some unrelated docstrings that didn't justify their own change.

benjyw · 2022-06-12T21:06:20Z

Will change the names of the expected jobs right as I merge this.

benjyw · 2022-06-13T06:48:16Z

build-support/bin/generate_github_workflows.py

@@ -243,30 +240,6 @@ def deploy_to_s3() -> Step:
    }


-def bootstrap_caches() -> Sequence[Step]:


Moved into Helper.

benjyw · 2022-06-13T06:48:34Z

build-support/bin/generate_github_workflows.py

+                "uses": "actions/cache@v3",
+                "with": {
+                    "path": "\n".join(NATIVE_FILES),
+                    "key": f"{self.platform_name()}-engine-${{ steps.get-engine-hash.outputs.hash }}-v1\n",


This is why it was moved into Helper.

benjyw · 2022-06-13T06:49:45Z

src/python/pants/backend/python/target_types.py

@@ -892,8 +892,8 @@ class PythonTestsOverrideField(OverridesField):
        PythonTestTarget.alias,
        """
        overrides={
-            "foo_test.py": {"timeout": 120]},
-            "bar_test.py": {"timeout": 200]},
+            "foo_test.py": {"timeout": 120},


Unrelated, but not worth a CI cycle on its own.

stuhood

Thanks!: just a comment on factoring.

stuhood · 2022-06-13T18:54:52Z

build-support/bin/generate_github_workflows.py

                        upload_log_artifacts(name="wheels-macos-x86_64"),
                        deploy_to_s3(),
                    ],
                },
            }
        )
+    jobs.update(**macos_arm64_jobs())


Rather than breaking out only the macos_arm64_jobs, it seems like there would be better symmetry to breaking out all of wheel building (i.e. lines 605-690) into a separate function wheel_building_jobs. As it stands, the parallels (and differences) between them aren't as obvious as they could be.

Yeah, I have a TODO for ~that above.

stuhood · 2022-06-13T18:56:10Z

build-support/bin/generate_github_workflows.py

+        "build_wheels_macos_arm64": {
+            "name": f"Bootstrap Pants, build wheels and fs_util ({Platform.MACOS11_ARM64.value})",
+            "runs-on": helper_macos_arm64.runs_on(),
+            "needs": "check_labels",


Eric-Arellano

Thanks!

benjyw · 2022-06-14T22:48:51Z

@Eric-Arellano can you try and consume the next dev release on your M1, to see that the wheels work?

Internal changes: * Add `@alonsodomin` to maintainers ([pantsbuild#15869](pantsbuild#15869)) * Scala parser improvements ([pantsbuild#15839](pantsbuild#15839)) * Refactor valid field aliases calculation for targets into a classmethod for easier re-use. ([pantsbuild#15861](pantsbuild#15861)) * Update Media page with Pycon ([pantsbuild#15852](pantsbuild#15852)) * Use Readme.com's new Markdown syntax ([pantsbuild#15845](pantsbuild#15845)) * [internal] better align test JVM lockfile support with pytest concepts ([pantsbuild#15804](pantsbuild#15804)) * Fix broken `engine_error` testutil decorator. ([pantsbuild#15818](pantsbuild#15818)) * Add `description_of_origin` for Docker address parsing ([pantsbuild#15814](pantsbuild#15814)) * Use `MultiGet` in Go rule ([pantsbuild#15838](pantsbuild#15838)) * Switch from warn to warning ([pantsbuild#15837](pantsbuild#15837)) * Bump clap from 3.1.18 to 3.2.4 in /src/rust/engine ([pantsbuild#15834](pantsbuild#15834)) * Bump reqwest from 0.11.10 to 0.11.11 in /src/rust/engine ([pantsbuild#15831](pantsbuild#15831)) * Bump strum from 0.24.0 to 0.24.1 in /src/rust/engine ([pantsbuild#15833](pantsbuild#15833)) * Bump strum_macros from 0.24.0 to 0.24.1 in /src/rust/engine ([pantsbuild#15835](pantsbuild#15835)) * Remove flake8-pantsbuild plugin ([pantsbuild#15830](pantsbuild#15830)) * go: support third-party package embeds ([pantsbuild#15827](pantsbuild#15827)) * Move `gshuflin` to Maintainers Emeritus ([pantsbuild#15828](pantsbuild#15828)) * Adds current contents of readme.com docs to repo and adds notes on how to sync. ([pantsbuild#15810](pantsbuild#15810)) * Remove unnecessary `WrappedNode` method indirection ([pantsbuild#15813](pantsbuild#15813)) * Handle anchors in doc_url() correctly. ([pantsbuild#15812](pantsbuild#15812)) * Move the macOS ARM64 build job into the existing test workflows. ([pantsbuild#15805](pantsbuild#15805)) * Add Lablup to users list. ([pantsbuild#15807](pantsbuild#15807)) * Add `description_of_origin` for several places finding Addresses ([pantsbuild#15797](pantsbuild#15797)) * [internal] Register `platform_specific_behavior` marker with Pytest ([pantsbuild#12679](pantsbuild#12679)) * Bump http from 0.2.7 to 0.2.8 in /src/rust/engine ([pantsbuild#15780](pantsbuild#15780)) * Update the `notify` crate. ([pantsbuild#15783](pantsbuild#15783)) * Bump tokio-stream from 0.1.8 to 0.1.9 in /src/rust/engine ([pantsbuild#15777](pantsbuild#15777)) * Bump indexmap from 1.8.1 to 1.8.2 in /src/rust/engine ([pantsbuild#15778](pantsbuild#15778)) * Bump uuid from 1.0.0 to 1.1.1 in /src/rust/engine ([pantsbuild#15779](pantsbuild#15779)) * Bump hyper from 0.14.18 to 0.14.19 in /src/rust/engine ([pantsbuild#15781](pantsbuild#15781)) * Update `crossbeam-utils` to `0.8.8`. ([pantsbuild#15782](pantsbuild#15782))

Internal changes: * Add `@alonsodomin` to maintainers ([#15869](#15869)) * Scala parser improvements ([#15839](#15839)) * Refactor valid field aliases calculation for targets into a classmethod for easier re-use. ([#15861](#15861)) * Update Media page with Pycon ([#15852](#15852)) * Use Readme.com's new Markdown syntax ([#15845](#15845)) * [internal] better align test JVM lockfile support with pytest concepts ([#15804](#15804)) * Fix broken `engine_error` testutil decorator. ([#15818](#15818)) * Add `description_of_origin` for Docker address parsing ([#15814](#15814)) * Use `MultiGet` in Go rule ([#15838](#15838)) * Switch from warn to warning ([#15837](#15837)) * Bump clap from 3.1.18 to 3.2.4 in /src/rust/engine ([#15834](#15834)) * Bump reqwest from 0.11.10 to 0.11.11 in /src/rust/engine ([#15831](#15831)) * Bump strum from 0.24.0 to 0.24.1 in /src/rust/engine ([#15833](#15833)) * Bump strum_macros from 0.24.0 to 0.24.1 in /src/rust/engine ([#15835](#15835)) * Remove flake8-pantsbuild plugin ([#15830](#15830)) * go: support third-party package embeds ([#15827](#15827)) * Move `gshuflin` to Maintainers Emeritus ([#15828](#15828)) * Adds current contents of readme.com docs to repo and adds notes on how to sync. ([#15810](#15810)) * Remove unnecessary `WrappedNode` method indirection ([#15813](#15813)) * Handle anchors in doc_url() correctly. ([#15812](#15812)) * Move the macOS ARM64 build job into the existing test workflows. ([#15805](#15805)) * Add Lablup to users list. ([#15807](#15807)) * Add `description_of_origin` for several places finding Addresses ([#15797](#15797)) * [internal] Register `platform_specific_behavior` marker with Pytest ([#12679](#12679)) * Bump http from 0.2.7 to 0.2.8 in /src/rust/engine ([#15780](#15780)) * Update the `notify` crate. ([#15783](#15783)) * Bump tokio-stream from 0.1.8 to 0.1.9 in /src/rust/engine ([#15777](#15777)) * Bump indexmap from 1.8.1 to 1.8.2 in /src/rust/engine ([#15778](#15778)) * Bump uuid from 1.0.0 to 1.1.1 in /src/rust/engine ([#15779](#15779)) * Bump hyper from 0.14.18 to 0.14.19 in /src/rust/engine ([#15781](#15781)) * Update `crossbeam-utils` to `0.8.8`. ([#15782](#15782))

### Internal (put these in a PR comment for review, not the release notes) * Bump uuid from 1.1.1 to 1.1.2 in /src/rust/engine ([#15832](#15832)) * Tests for Docker from arg not a upstream target ([#15874](#15874)) * Fix outstanding todo for moved fields defaults support. ([#15923](#15923)) * Move generic debug adapter settings to dedicated subsystem ([#15928](#15928)) * Prepare `2.12.0rc3`. ([#15913](#15913)) * Prepare `2.11.1rc3`. ([#15912](#15912)) * Upgrade default mypy to 0.961 and default pylint to v2.13.9 ([#15886](#15886)) * Bump clap from 3.2.4 to 3.2.6 in /src/rust/engine ([#15909](#15909)) * Bump indexmap from 1.8.2 to 1.9.1 in /src/rust/engine ([#15894](#15894)) * Bump time from 0.3.9 to 0.3.11 in /src/rust/engine ([#15895](#15895)) * Bump tower-service from 0.3.1 to 0.3.2 in /src/rust/engine ([#15896](#15896)) * Bump crossbeam-channel from 0.5.4 to 0.5.5 in /src/rust/engine ([#15893](#15893)) * [internal] remove TestCoursierWrapper ([#15902](#15902)) * Silence `[scala-infer].force_add_siblings_as_dependencies` deprecation. ([#15898](#15898)) * Add unit tests to GraphQL queries ([#15792](#15792)) * Release script fixups ([#15875](#15875)) * [internal] jvm: port remaining tests to use test lockfiles ([#15870](#15870)) * Bump to 2.13.0a0 ([#15871](#15871)) * Add `@alonsodomin` to maintainers ([#15869](#15869)) * Scala parser improvements ([#15839](#15839)) * Refactor valid field aliases calculation for targets into a classmethod for easier re-use. ([#15861](#15861)) * Update Media page with Pycon ([#15852](#15852)) * Use Readme.com's new Markdown syntax ([#15845](#15845)) * [internal] better align test JVM lockfile support with pytest concepts ([#15804](#15804)) * Fix broken `engine_error` testutil decorator. ([#15818](#15818)) * Add `description_of_origin` for Docker address parsing ([#15814](#15814)) * Use `MultiGet` in Go rule ([#15838](#15838)) * Switch from warn to warning ([#15837](#15837)) * Bump clap from 3.1.18 to 3.2.4 in /src/rust/engine ([#15834](#15834)) * Bump reqwest from 0.11.10 to 0.11.11 in /src/rust/engine ([#15831](#15831)) * Bump strum from 0.24.0 to 0.24.1 in /src/rust/engine ([#15833](#15833)) * Bump strum_macros from 0.24.0 to 0.24.1 in /src/rust/engine ([#15835](#15835)) * Remove flake8-pantsbuild plugin ([#15830](#15830)) * go: support third-party package embeds ([#15827](#15827)) * Move `gshuflin` to Maintainers Emeritus ([#15828](#15828)) * Adds current contents of readme.com docs to repo and adds notes on how to sync. ([#15810](#15810)) * Remove unnecessary `WrappedNode` method indirection ([#15813](#15813)) * Handle anchors in doc_url() correctly. ([#15812](#15812)) * Move the macOS ARM64 build job into the existing test workflows. ([#15805](#15805)) * Add Lablup to users list. ([#15807](#15807)) * Add `description_of_origin` for several places finding Addresses ([#15797](#15797)) * [internal] Register `platform_specific_behavior` marker with Pytest ([#12679](#12679)) * Bump http from 0.2.7 to 0.2.8 in /src/rust/engine ([#15780](#15780)) * Update the `notify` crate. ([#15783](#15783)) * Bump tokio-stream from 0.1.8 to 0.1.9 in /src/rust/engine ([#15777](#15777)) * Bump indexmap from 1.8.1 to 1.8.2 in /src/rust/engine ([#15778](#15778)) * Bump uuid from 1.0.0 to 1.1.1 in /src/rust/engine ([#15779](#15779)) * Bump hyper from 0.14.18 to 0.14.19 in /src/rust/engine ([#15781](#15781)) * Update `crossbeam-utils` to `0.8.8`. ([#15782](#15782))

benjyw requested review from Eric-Arellano and stuhood June 11, 2022 19:16

benjyw added the category:internal CI, fixes for not-yet-released features, etc. label Jun 11, 2022

benjyw force-pushed the macos11_arm64 branch 4 times, most recently from fc62c25 to b160038 Compare June 12, 2022 09:55

benjyw force-pushed the macos11_arm64 branch from b160038 to 8006e82 Compare June 12, 2022 14:41

benjyw commented Jun 13, 2022

View reviewed changes

kaos approved these changes Jun 13, 2022

View reviewed changes

stuhood approved these changes Jun 13, 2022

View reviewed changes

Fix typo

58e3872

benjyw merged commit c1b1eaa into pantsbuild:main Jun 13, 2022

benjyw deleted the macos11_arm64 branch June 13, 2022 21:09

Eric-Arellano reviewed Jun 13, 2022

View reviewed changes

illicitonion mentioned this pull request Jun 17, 2022

Bump to 2.13.0a0 #15871

Merged

wisechengyi mentioned this pull request Jun 25, 2022

Prepare 2.14.0.dev0 #15944

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the macOS ARM64 build job into the existing test workflows. #15805

Move the macOS ARM64 build job into the existing test workflows. #15805

benjyw commented Jun 11, 2022 •

edited

Loading

benjyw commented Jun 11, 2022

benjyw commented Jun 12, 2022

benjyw commented Jun 12, 2022

benjyw Jun 13, 2022

benjyw Jun 13, 2022

benjyw Jun 13, 2022

stuhood left a comment

stuhood Jun 13, 2022

benjyw Jun 13, 2022

stuhood Jun 13, 2022

Eric-Arellano left a comment

benjyw commented Jun 14, 2022

		@@ -243,30 +240,6 @@ def deploy_to_s3() -> Step:
		}


		def bootstrap_caches() -> Sequence[Step]:

Move the macOS ARM64 build job into the existing test workflows. #15805

Move the macOS ARM64 build job into the existing test workflows. #15805

Conversation

benjyw commented Jun 11, 2022 • edited Loading

benjyw commented Jun 11, 2022

benjyw commented Jun 12, 2022

benjyw commented Jun 12, 2022

benjyw Jun 13, 2022

Choose a reason for hiding this comment

benjyw Jun 13, 2022

Choose a reason for hiding this comment

benjyw Jun 13, 2022

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

stuhood Jun 13, 2022

Choose a reason for hiding this comment

benjyw Jun 13, 2022

Choose a reason for hiding this comment

stuhood Jun 13, 2022

Choose a reason for hiding this comment

Eric-Arellano left a comment

Choose a reason for hiding this comment

benjyw commented Jun 14, 2022

benjyw commented Jun 11, 2022 •

edited

Loading