[NET-5622] build: consolidate Envoy version management #21245

zalimeni · 2024-06-03T15:07:31Z

Description

Consolidate Envoy versions into a single canonical file to enable simpler management and avoid missed updates in the future.

This effort is motivated by repeated experience struggling to ensure that we updated Envoy "everywhere" we needed to. Even moreso than managing our Go versions, there's a significant amount of sprawl when it comes to Envoy, and addressing that will enable us to more confidently update versions as well as automate the toil of doing so when new releases are available.

Note that if it weren't for returning full x.y.z Envoy versions in v1/agent/self output, I'm pretty sure we could just store the "major" x.y versions going forward and dynamically test against the latest of each in CI, since we only validate major versions in the consul connect envoy command. For now, I've kept the existing pattern of explicitly source-controlled minor Envoy releases that we'll have to keep updated - but we can at least automate that more effectively now.

I've opened manual backports through 1.15 Ent to ensure that the ENVOY_VERSIONS file is available on all actively tested branches; this is necessary for sourcing in nightly integration tests on main, so required for this PR prior to merge.

This change also includes:

Fix to Go version branch sourcing aligned with this PR (missed in previous Go version work).
Some missed component version updates in integration tests.

I have not removed or changed defunct uses of Envoy in Windows integration tests as I think that would be best accomplished as a separate holistic Windows cleanup PR.

Testing & Reproduction steps

get-envoy-versions output matches file
Integration tests use 1.29.5
Nightly main versions match expected
Nightly 1.19 versions match expected (workflow dispatch test branch used to target 1.19 PR branch for sourcing Envoy version to emulate post-1.19-backport nightly run)
Nightly 1.15 versions match expected (workflow dispatch test branch used to target 1.15 PR branch for sourcing Envoy version to emulate post-1.15-backport nightly run)

Local agent test:
main

❯ curl -s localhost:8500/v1/agent/self | jq '.xDS.SupportedProxies'
{
  "envoy": [
    "1.29.5",
    "1.28.4",
    "1.27.6",
    "1.26.8"
  ]
}

PR branch

❯ curl -s localhost:8500/v1/agent/self | jq '.xDS.SupportedProxies'
{
  "envoy": [
    "1.29.5",
    "1.28.4",
    "1.27.6",
    "1.26.8"
  ]
}

Follow-ups

Update release guide to call out upgrade test Envoy version adjustments
Verify nightly Ent tests are not broken after merge

Links

Manual backport PRs:

PR Checklist

updated test coverage
external facing docs updated
appropriate backport labels added
not a security concern

zalimeni · 2024-06-03T18:47:23Z

.github/workflows/nightly-test-integrations-1.15.x.yml

-          # this is further going to multiplied in envoy-integration tests by the 
-          # other dimensions in the matrix.  Currently TOTAL_RUNNERS would be
-          # 14 based on these values:
-          # envoy-version: ["1.22.11", "1.23.12", "1.24.12", "1.25.11", "1.26.8", "1.27.5", "1.28.3"]
-          # xds-target: ["server", "client"]
-          TOTAL_RUNNERS: 7
+          # TEST_SPLITS sets the number of test case splits to use in the matrix. This will be
+          # further multiplied in envoy-integration tests by the other dimensions in the matrix
+          # to determine the total number of runners used.
+          TEST_SPLITS: 4


It took me until this PR to realize that this number does not need to be changed after every Envoy version introduction - the comment was simply noting that the total runners is one of 3 factors in the matrix that multiply to get the total effective runner count. I've updated the env var here to TEST_SPLITS to try to make that more obvious, and restored the original count of 4.

zalimeni · 2024-06-03T18:48:15Z

.github/workflows/nightly-test-integrations-1.18.x.yml

-        consul-version: ["1.15", "1.16", "1.17"]
+        consul-version: ["1.15", "1.16", "1.17", "1.18"]


Noticed this was missing, so added here

zalimeni · 2024-06-05T15:33:58Z

cc @dduzgun-security @sarahalsmiller that this PR is in draft. I'm not sure it'll be ready by the time you do Envoy bumps for the latest set of patches, but wanted to keep you in the loop esp. if I'm OOO and this looks worth pushing through. I'll do my best to kick it along for now.

dduzgun-security · 2024-06-05T15:38:57Z

@zalimeni this is awesome and will save a ton of time while bringing better visibility on the envoy versions we use 🙌 Thanks a lot!

sarahalsmiller · 2024-06-10T20:49:24Z

envoyextensions/xdscommon/ENVOY_VERSIONS

@@ -0,0 +1,14 @@
+# This file represents the canonical list of supported Envoy versions for this version of Consul.


zalimeni · 2024-06-17T18:40:29Z

.github/workflows/nightly-test-integ-peering_commontopo.yml

+    with:
+      ref: ${{ inputs.branch }}


Added in this PR to come along for the backport ride since I was already making changes to support nightly branch targeting, and noticed these were missing.

zalimeni · 2024-06-17T19:10:51Z

envoyextensions/xdscommon/proxysupport.go

@@ -27,18 +79,28 @@ var EnvoyVersions = []string{
 // see: https://www.consul.io/docs/connect/proxies/envoy#supported-versions
 var UnsupportedEnvoyVersions = []string{}

-// GetMaxEnvoyMinorVersion grabs the first value in EnvoyVersions and strips the patch number off in order
-// to return the maximum supported Envoy minor version
+// GetMaxEnvoyMajorVersion grabs the first value in EnvoyVersions and strips the last number off in order


Here and below, aligning these functions to Envoy's own "major/minor" non-semver naming conventions, similar to Consul and Go.

These are not used in Ent-specific code.

This will result in a (small) breaking change in the envoyextensions submodule for anyone who's consuming these fields; the submodule is v0.x so this is allowed.

Supports nightly tests.

Manual backport of #21245 into release/1.19.x.

hc-github-team-consul-core · 2024-07-06T09:09:20Z

@zalimeni, a backport is missing for this PR [21245] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

hc-github-team-consul-core · 2024-07-07T09:09:37Z

@zalimeni, a backport is missing for this PR [21245] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

hc-github-team-consul-core · 2024-07-08T09:09:45Z

@zalimeni, a backport is missing for this PR [21245] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

* build: consolidate Envoy version management Simplify Envoy version management by consolidating all runtime, build, and CI sources of Envoy versions into a single plaintext file. The goal of this change is to avoid common mistakes missing an update of some Envoy versions (both in general and due to release branch inconsistency), and enable automated Envoy version updates in the future. * ci: add missing ref argument for get-go-version Supports nightly tests.

hc-github-team-consul-core · 2024-07-09T09:09:45Z

@zalimeni, a backport is missing for this PR [21245] for versions [1.15] please perform the backport manually and add the following snippet to your backport PR description:

<details>
	<summary> Overview of commits </summary>
		- <<backport commit 1>>
		- <<backport commit 2>>
		...
</details>

* build: consolidate Envoy version management Simplify Envoy version management by consolidating all runtime, build, and CI sources of Envoy versions into a single plaintext file. The goal of this change is to avoid common mistakes missing an update of some Envoy versions (both in general and due to release branch inconsistency), and enable automated Envoy version updates in the future. * ci: add missing ref argument for get-go-version Supports nightly tests.

… release/1.19.x (#21292) build: consolidate Envoy version management Manual backport of #21245 into release/1.19.x. Co-authored-by: sarahalsmiller <100602640+sarahalsmiller@users.noreply.github.com>

github-actions bot added theme/cli Flags and documentation for the CLI interface type/ci Relating to continuous integration (CI) tooling for testing or releases theme/contributing Additions and enhancements to community contributing materials labels Jun 3, 2024

zalimeni added pr/no-changelog PR does not need a corresponding .changelog entry pr/no-backport labels Jun 3, 2024

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch from e5a280f to cd32bc8 Compare June 3, 2024 18:43

zalimeni commented Jun 3, 2024

View reviewed changes

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch 2 times, most recently from 9e86e92 to 42efa0f Compare June 3, 2024 19:02

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch from 42efa0f to 8aeae52 Compare June 10, 2024 19:54

zalimeni changed the title ~~chore: consolidate Envoy version management~~ [NET-5622] build: consolidate Envoy version management Jun 10, 2024

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch from 8aeae52 to 6dbe438 Compare June 10, 2024 20:08

This was referenced Jun 10, 2024

Backport of [NET-5622] build: consolidate Envoy version management to release/1.19.x #21292

Merged

Backport of [NET-5622] build: consolidate Envoy version management to release/1.18.x #21293

Closed

sarahalsmiller reviewed Jun 10, 2024

View reviewed changes

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch from 6dbe438 to 94699b2 Compare June 10, 2024 21:19

zalimeni mentioned this pull request Jun 10, 2024

ci: add ENVOY_VERSIONS file for nightly int tests (1.17) #21297

Closed

4 tasks

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch 8 times, most recently from 234c8f9 to e9779c4 Compare June 17, 2024 19:09

zalimeni commented Jun 17, 2024

View reviewed changes

ci: add missing ref argument for get-go-version

91d1fbf

Supports nightly tests.

zalimeni force-pushed the zalimeni/net-5622-consolidate-envoy-version-mgmt branch from cb498be to 91d1fbf Compare June 18, 2024 21:28

zalimeni added a commit that referenced this pull request Jun 18, 2024

build: consolidate Envoy version management

9ca89bc

Manual backport of #21245 into release/1.19.x.

sarahalsmiller merged commit 40ca4ad into main Jul 5, 2024
132 checks passed

sarahalsmiller deleted the zalimeni/net-5622-consolidate-envoy-version-mgmt branch July 5, 2024 19:19

sarahalsmiller added backport/all Apply backports for all active releases per .release/versions.hcl and removed pr/no-backport labels Jul 5, 2024

hc-github-team-consul-core added backport/1.19 Changes are backported to 1.19 backport/ent/1.17 Changes are backported to 1.17 ent backport/ent/1.15 Changes are backported to 1.15 ent backport/ent/1.18 Changes are backported to 1.18 ent labels Jul 5, 2024

hc-github-team-consul-core mentioned this pull request Jul 5, 2024

Backport of [NET-5622] build: consolidate Envoy version management into release/1.19.x #21521

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NET-5622] build: consolidate Envoy version management #21245

[NET-5622] build: consolidate Envoy version management #21245

zalimeni commented Jun 3, 2024 •

edited

Loading

zalimeni Jun 3, 2024 •

edited

Loading

zalimeni Jun 3, 2024 •

edited

Loading

zalimeni commented Jun 5, 2024 •

edited

Loading

dduzgun-security commented Jun 5, 2024

sarahalsmiller Jun 10, 2024

zalimeni Jun 17, 2024

zalimeni Jun 17, 2024 •

edited

Loading

hc-github-team-consul-core commented Jul 6, 2024

hc-github-team-consul-core commented Jul 7, 2024

hc-github-team-consul-core commented Jul 8, 2024

hc-github-team-consul-core commented Jul 9, 2024

		consul-version: ["1.15", "1.16", "1.17"]
		consul-version: ["1.15", "1.16", "1.17", "1.18"]

		@@ -0,0 +1,14 @@
		# This file represents the canonical list of supported Envoy versions for this version of Consul.

[NET-5622] build: consolidate Envoy version management #21245

[NET-5622] build: consolidate Envoy version management #21245

Conversation

zalimeni commented Jun 3, 2024 • edited Loading

Description

Testing & Reproduction steps

Follow-ups

Links

PR Checklist

zalimeni Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

zalimeni Jun 3, 2024 • edited Loading

Choose a reason for hiding this comment

zalimeni commented Jun 5, 2024 • edited Loading

dduzgun-security commented Jun 5, 2024

sarahalsmiller Jun 10, 2024

Choose a reason for hiding this comment

zalimeni Jun 17, 2024

Choose a reason for hiding this comment

zalimeni Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

hc-github-team-consul-core commented Jul 6, 2024

hc-github-team-consul-core commented Jul 7, 2024

hc-github-team-consul-core commented Jul 8, 2024

hc-github-team-consul-core commented Jul 9, 2024

zalimeni commented Jun 3, 2024 •

edited

Loading

zalimeni Jun 3, 2024 •

edited

Loading

zalimeni Jun 3, 2024 •

edited

Loading

zalimeni commented Jun 5, 2024 •

edited

Loading

zalimeni Jun 17, 2024 •

edited

Loading