backport of commit 92d31cd

.changelog/17107.txt

@@ -0,0 +1,3 @@
+```release-note:breaking-change
+api: RaftLeaderTransfer now requires an id string. An empty string can be specified to keep the old behavior.
+```

.changelog/17936.txt

@@ -0,0 +1,3 @@
+```release-note:feature
+acl: Add new `acl.tokens.dns` config field which specifies the token used implicitly during dns checks.
+```

.changelog/18322.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+catalog api: fixes a bug with catalog api where filter query parameter was not working correctly for the `/v1/catalog/services` endpoint
+```

.changelog/18573.txt

@@ -0,0 +1,3 @@
+```release-note:enhancement
+xds: Use downstream protocol when connecting to local app
+```

.changelog/18769.txt

@@ -0,0 +1,3 @@
+```release-note:feature
+acl: Adds a new ACL rule for workload identities
+```

.changelog/18773.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+ca: Vault provider now cleans up the previous Vault issuer and key when generating a new leaf signing certificate [[GH-18779](https://github.com/hashicorp/consul/issues/18779)]
+```

.changelog/18797.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+command: Adds -since flag in consul debug command which internally calls hcdiag for debug information in the past.
+```

.changelog/18813.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+acl: Use templated policy to generate synthetic policies for tokens/roles with node and/or service identities
+```

.changelog/18816.txt

@@ -0,0 +1,3 @@
+```release-note:feature
+cli: Add `consul acl templated-policy` commands to read, list and preview templated policies. 
+```

.changelog/18831.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+gateways: Fix a bug where gateway to service mappings weren't being cleaned up properly when externally registered proxies were being deregistered.
+```

.changelog/18943.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+api: added `CheckRegisterOpts` to Agent API
+```

.changelog/18959.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+gateways: Fix a bug where a service in a peered datacenter could not access an external node service through a terminating gateway
+```

.changelog/18983.txt

@@ -0,0 +1,3 @@
+```release-note:improvement
+api: added `Token` field to `ServiceRegisterOpts` type in Agent API
+```

.changelog/18994.txt

@@ -0,0 +1,26 @@
+```release-note:feature
+# Catalog v2 feature preview
+This release provides the ability to preview Consul's v2 Catalog and Resource API if enabled. The new model supports
+multi-port application deployments with only a single Envoy proxy. Note that the v1 and v2 catalogs are not cross
+compatible, and not all Consul features are available within this v2 feature preview. See the [v2 Catalog and Resource
+API documentation](https://developer.hashicorp.com/consul/docs/architecture/v2) for more information. The v2 Catalog and
+Resources API should be considered a feature preview within this release and should not be used in production
+environments.
+
+### Limitations
+* The v2 catalog API feature preview does not support connections with client agents. As a result, it is only available for Kubernetes deployments, which use [Consul dataplanes](consul/docs/connect/dataplane) instead of client agents.
+* The v1 and v2 catalog APIs cannot run concurrently.
+* The Consul UI does not support multi-port services or the v2 catalog API in this release.
+* HCP Consul does not support multi-port services or the v2 catalog API in this release.
+* The v2 API only supports transparent proxy mode where services that have permissions to connect to each other can use
+  Kube DNS to connect.
+
+### Known Issues
+* When using the v2 API with transparent proxy, Kubernetes pods cannot use L7 liveness, readiness, or startup probes.
+
+
+[[Catalog resource controllers]](https://github.com/hashicorp/consul/tree/e6b724d06249d3e62cd75afe3ee6042ba1fd5415/internal/catalog/internal/controllers)
+[[Mesh resource controllers]](https://github.com/hashicorp/consul/tree/e6b724d06249d3e62cd75afe3ee6042ba1fd5415/internal/mesh/internal/controllers)
+[[Auth resource controllers]](https://github.com/hashicorp/consul/tree/e6b724d06249d3e62cd75afe3ee6042ba1fd5415/internal/auth/internal)
+[[V2 Protobufs]](https://github.com/hashicorp/consul/tree/e6b724d06249d3e62cd75afe3ee6042ba1fd5415/proto-public)
+```

.changelog/19031.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+api: add custom marshal/unmarshal for ServiceResolverConfigEntry.RequestTimeout so config entries that set this field can be read using the API.
+```

.changelog/19077.txt

@@ -0,0 +1,3 @@
+```release-note:feature
+acl: Adds workload identity templated policy
+```

.changelog/19095.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+ca: ensure Vault CA provider respects Vault Enterprise namespace configuration.
+```

.changelog/19225.txt

@@ -0,0 +1,9 @@
+```release-note:security
+Upgrade Go to 1.20.10.
+This resolves vulnerability [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
+/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487)(`net/http`).
+```
+```release-note:security
+Update `golang.org/x/net` to v0.17.0 to address [CVE-2023-39325](https://nvd.nist.gov/vuln/detail/CVE-2023-39325)
+/ [CVE-2023-44487](https://nvd.nist.gov/vuln/detail/CVE-2023-44487)(`x/net/http2`).
+```

.changelog/_6074.txt

@@ -0,0 +1,3 @@
+```release-note:bug
+connect: **(Enterprise only)** Fix bug where incorrect service-defaults entries were fetched to determine an upstream's protocol whenever the upstream did not explicitly define the namespace / partition. When this bug occurs, upstreams would use the protocol from a service-default entry in the default namespace / partition, rather than their own namespace / partition.
+```

.copywrite.hcl

@@ -27,6 +27,9 @@ project {
     "agent/grpc-middleware/rate_limit_mappings.gen.go",
     "agent/uiserver/dist/**",
 
+    # ignoring policy embedded files
+    "agent/structs/acltemplatedpolicy/policies/ce/**",
+
     # licensed under MPL - ignoring for now until the copywrite tool can support
     # multiple licenses per repo.
     "sdk/**",

.github/scripts/filter_changed_files_go_test.sh

@@ -2,35 +2,41 @@
 # Copyright (c) HashiCorp, Inc.
 # SPDX-License-Identifier: BUSL-1.1
 
+set -euo pipefail
 
 # Get the list of changed files
-files_to_check=$(git diff --name-only origin/$GITHUB_BASE_REF)
+# Using `git merge-base` ensures that we're always comparing against the correct branch point. 
+#For example, given the commits:
+#
+# A---B---C---D---W---X---Y---Z # origin/main
+#             \---E---F         # feature/branch
+#
+# ... `git merge-base origin/$SKIP_CHECK_BRANCH HEAD` would return commit `D`
+# `...HEAD` specifies from the common ancestor to the latest commit on the current branch (HEAD)..
+files_to_check=$(git diff --name-only "$(git merge-base origin/$SKIP_CHECK_BRANCH HEAD~)"...HEAD)
 
 # Define the directories to check
 skipped_directories=("docs/" "ui/" "website/" "grafana/")
 
-# Initialize a variable to track directories outside the skipped ones
-other_directories=""
-trigger_ci=false
-
 # Loop through the changed files and find directories/files outside the skipped ones
-for file_to_check in $files_to_check; do
+for file_to_check in "${files_to_check[@]}"; do
 	file_is_skipped=false
 	for dir in "${skipped_directories[@]}"; do
 		if [[ "$file_to_check" == "$dir"* ]] || [[ "$file_to_check" == *.md && "$dir" == *"/" ]]; then
 			file_is_skipped=true
 			break
 		fi
 	done
-	if [ "$file_is_skipped" = "false" ]; then
-		other_directories+="$(dirname "$file_to_check")\n"
-		trigger_ci=true
-		echo "Non doc file(s) changed - triggered ci: $trigger_ci"
-		echo -e $other_directories
-		echo "trigger-ci=$trigger_ci" >>"$GITHUB_OUTPUT"
+	if [ "$file_is_skipped" != "true" ]; then
+		echo -e $file_to_check
+        SKIP_CI=false
+		echo "Changes detected in non-documentation files - skip-ci: $SKIP_CI"
+        echo "skip-ci=$SKIP_CI" >> "$GITHUB_OUTPUT"
 		exit 0 ## if file is outside of the skipped_directory exit script
 	fi
 done
 
-echo "Only doc file(s) changed - triggered ci: $trigger_ci"
-echo "trigger-ci=$trigger_ci" >>"$GITHUB_OUTPUT"
+echo -e "$files_to_check"
+SKIP_CI=true
+echo "Changes detected in only documentation files - skip-ci: $SKIP_CI"
+echo "skip-ci=$SKIP_CI" >> "$GITHUB_OUTPUT"

.github/scripts/get_runner_classes_windows.sh

@@ -10,11 +10,11 @@ set -euo pipefail
 case "$GITHUB_REPOSITORY" in
 *-enterprise)
 	# shellcheck disable=SC2129
-	echo "compute-small=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.4xlarge']" >>"$GITHUB_OUTPUT"
-	echo "compute-medium=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.8xlarge']" >>"$GITHUB_OUTPUT"
-	echo "compute-large=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.12xlarge']" >>"$GITHUB_OUTPUT"
+	echo "compute-small=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.2xlarge']" >>"$GITHUB_OUTPUT"
+	echo "compute-medium=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.4xlarge']" >>"$GITHUB_OUTPUT"
+	echo "compute-large=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.8xlarge']" >>"$GITHUB_OUTPUT"
 	# m5d.8xlarge is equivalent to our xl custom runner in CE
-	echo "compute-xl=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.16xlarge']" >>"$GITHUB_OUTPUT"
+	echo "compute-xl=['self-hosted', 'ondemand', 'os=windows-2019', 'type=m6a.12xlarge']" >>"$GITHUB_OUTPUT"
 	;;
 *)
 	# shellcheck disable=SC2129

.github/scripts/license_checker.sh

@@ -2,15 +2,19 @@
 # Copyright (c) HashiCorp, Inc.
 # SPDX-License-Identifier: BUSL-1.1
 
+if [[ ${GITHUB_BASE_REF} == release/1.14.* ]] || [[ ${GITHUB_BASE_REF} == release/1.15.* ]] || [[ ${GITHUB_BASE_REF} == release/1.16.* ]]; then
+    busl_files=$(grep -r 'SPDX-License-Identifier: BUSL' . --exclude-dir .github)
 
-busl_files=$(grep -r 'SPDX-License-Identifier: BUSL' . --exclude-dir .github)
-
-# If we do not find a file in .changelog/, we fail the check
-if [ -n "$busl_files" ]; then
-    echo "Found BUSL occurrences in the PR branch! (See NET-5258 for details)"
-    echo -n "$busl_files"
-    exit 1
+    if [ -n "$busl_files" ]; then
+        echo "Found BUSL occurrences in the PR branch! (See NET-5258 for details)"
+        echo -n "$busl_files"
+        exit 1
+    else
+        echo "Did not find any occurrences of BUSL in the PR branch"
+        exit 0
+    fi
+    echo "The variable starts with release/1.14, release/1.15, or release/1.17."
 else
-    echo "Did not find any occurrences of BUSL in the PR branch"
+    echo "Skipping BUSL check since ${GITHUB_BASE_REF} not one of release/1.14.*, release/1.15.*, or release/1.16.*."
     exit 0
-fi
+fi

.github/workflows/build.yml

@@ -85,15 +85,15 @@ jobs:
     strategy:
       matrix:
         include:
-          - {go: "1.20.8", goos: "linux", goarch: "386"}
-          - {go: "1.20.8", goos: "linux", goarch: "amd64"}
-          - {go: "1.20.8", goos: "linux", goarch: "arm"}
-          - {go: "1.20.8", goos: "linux", goarch: "arm64"}
-          - {go: "1.20.8", goos: "freebsd", goarch: "386"}
-          - {go: "1.20.8", goos: "freebsd", goarch: "amd64"}
-          - {go: "1.20.8", goos: "windows", goarch: "386"}
-          - {go: "1.20.8", goos: "windows", goarch: "amd64"}
-          - {go: "1.20.8", goos: "solaris", goarch: "amd64"}
+          - {go: "1.20.10", goos: "linux", goarch: "386"}
+          - {go: "1.20.10", goos: "linux", goarch: "amd64"}
+          - {go: "1.20.10", goos: "linux", goarch: "arm"}
+          - {go: "1.20.10", goos: "linux", goarch: "arm64"}
+          - {go: "1.20.10", goos: "freebsd", goarch: "386"}
+          - {go: "1.20.10", goos: "freebsd", goarch: "amd64"}
+          - {go: "1.20.10", goos: "windows", goarch: "386"}
+          - {go: "1.20.10", goos: "windows", goarch: "amd64"}
+          - {go: "1.20.10", goos: "solaris", goarch: "amd64"}
       fail-fast: true
 
     name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build
@@ -182,7 +182,7 @@ jobs:
     strategy:
       matrix:
         include:
-          - {go: "1.20.8", goos: "linux", goarch: "s390x"}
+          - {go: "1.20.10", goos: "linux", goarch: "s390x"}
       fail-fast: true
 
     name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build
@@ -233,7 +233,7 @@ jobs:
       matrix:
         goos: [ darwin ]
         goarch: [ "amd64", "arm64" ]
-        go: [ "1.20.8" ]
+        go: [ "1.20.10" ]
       fail-fast: true
 
     name: Go ${{ matrix.go }} ${{ matrix.goos }} ${{ matrix.goarch }} build

.github/workflows/frontend.yml

@@ -5,10 +5,8 @@ name: frontend
 
 on:
   push:
-    branches:
-      - main
+    paths:
       - ui/**
-      - backport/ui/**
 
 permissions:
   contents: read

.github/workflows/go-tests.yml

@@ -22,6 +22,7 @@ permissions:
 env:
   TEST_RESULTS: /tmp/test-results
   GOPRIVATE: github.com/hashicorp # Required for enterprise deps
+  SKIP_CHECK_BRANCH: ${{ github.head_ref || github.ref_name }}
 
 # concurrency
 concurrency:
@@ -33,7 +34,7 @@ jobs:
     runs-on: ubuntu-latest 
     name: Get files changed and conditionally skip CI
     outputs:
-      trigger-ci: ${{ steps.read-files.outputs.trigger-ci }}
+      skip-ci: ${{ steps.read-files.outputs.skip-ci }}
     steps:
       - uses: actions/checkout@c85c95e3d7251135ab7dc9ce3241c5835cc595a9 # v3.5.3
         with:
@@ -45,7 +46,7 @@ jobs:
   setup:
     needs: [conditional-skip]
     name: Setup
-    if: needs.conditional-skip.outputs.trigger-ci == 'true'
+    if: needs.conditional-skip.outputs.skip-ci != 'true'
     runs-on: ubuntu-latest
     outputs:
       compute-small: ${{ steps.setup-outputs.outputs.compute-small }}
@@ -506,7 +507,7 @@ jobs:
     - go-test-32bit
     # - go-test-s390x
     runs-on: ${{ fromJSON(needs.setup.outputs.compute-small) }}
-    if: always() && needs.conditional-skip.outputs.trigger-ci == 'true'
+    if: always() && needs.conditional-skip.outputs.skip-ci != 'true'
     steps:
       - name: evaluate upstream job results
         run: |

.github/workflows/jira-pr.yaml

@@ -40,7 +40,7 @@ jobs:
         id: is-team-member
         run: |
           TEAM=consul
-          ROLE="$(hub api orgs/hashicorp/teams/${TEAM}/memberships/${{ github.actor }} | jq -r '.role | select(.!=null)')"
+          ROLE="$(gh api orgs/hashicorp/teams/${TEAM}/memberships/${{ github.actor }} | jq -r '.role | select(.!=null)')"
           if [[ -n ${ROLE} ]]; then
             echo "Actor ${{ github.actor }} is a ${TEAM} team member"
             echo "MESSAGE=true" >> $GITHUB_OUTPUT

.github/workflows/license-checker.yml

@@ -7,11 +7,8 @@ name: License Checker
 
 on:
   pull_request:
+    # Logic to only apply check 1.1[4,5,6].x branches is in license_checker.sh
     types: [opened, synchronize]
-    branches:
-      - release/1.14.*
-      - release/1.15.*
-      - release/1.16.*
 
 jobs:
   # checks that the diff does not contain any reference to

.github/workflows/reusable-lint.yml

@@ -34,6 +34,7 @@ jobs:
         - "envoyextensions"
         - "troubleshoot"
         - "test/integration/consul-container"
+        - "test-integ"
         - "testing/deployer"
       fail-fast: true
     name: lint ${{ matrix.directory }}

.github/workflows/reusable-unit-split.yml

@@ -129,8 +129,7 @@ jobs:
           --rerun-fails-report=/tmp/gotestsum-rerun-fails \
           --packages="$PACKAGE_NAMES" \
           --junitfile ${{env.TEST_RESULTS}}/gotestsum-report.xml -- \
-          -tags="${{env.GOTAGS}}" -p 2 \
-          ${GO_TEST_FLAGS-} \
+          -tags="${{env.GOTAGS}}" \
           -cover -coverprofile=coverage.txt
 
       # NOTE: ENT specific step as we store secrets in Vault.