prepare 6.7.2 release (#179)

* more logging + Docker helper improvements * (offline mode #4) full implementation of offline mode in Relay app (#202) * fix goroutine leak related to stream heartbeats * use correct import path for gcfg * use correct import path for gcfg * added whitesource to build * removed broken environment references from build * Updating the minimum go version in the contribution guidelines (#213) * change offline mode env schema to separate dataId from other properties * offline mode in-repo docs (#214) * (#6) offline mode integration test + offline mode fixes (#204) * only run scheduled tests on private repo (#216) * fix "expiring-but-still-valid SDK key" logic so requests with old key are accepted * add note about non-support for clustered Redis and Redis Sentinel * disable diagnostic events & metrics events in offline mode * add accept-events-but-discard-them behavior for offline mode * clean up imports * fix product name in docs * fix the broken images in our repo docs (#217) * in auto-config mode, return 503 for client requests till configuration is complete (#219) * always send stream updates to clients regardless of version checking (#224) * Enable hourly Relay integration tests against Production (#223) * Record response body when relay archive download test fails (#225) There is some additional information provided in the response body for certain 504 error codes and this should allow us to see it. * Fix the broken production integration tests (#226) * improve metrics documentation and fix route strings in docs (#227) * improve metrics documentation and fix route strings in docs * clarify mobile + browser terminology * use more efficient jsonstream encoding/decoding for stream data and evaluations (#228) * recognize alias events from non-v3-schema payloads and forward them unchanged * environment should still be usable even if the client timed out * bump dependency versions for SDK fixes * update go-server-sdk-dynamodb to get newer AWS SDK * minor clarification about Relay.Close (#234) * Use the Go releaser template (#233) * [ch102248] big segment sync with redis (#235) * fix makefile so it tries building all the test code first before running any of it (#239) * (big segments #1) add basic big segments configuration for SDK clients (#237) * (big segments #2) add more abstraction around big segments implementation for testability (#238) * [102253] bigsegment status / config (#242) * big segments integration test + misc fixes (#240) * use latest URL paths for big segments endpoints * add SDK DynamoDB integration for big segments (#241) * fix broken link in Markdown docs (#246) * make sure newly added credentials for existing environments are accepted in requests (#244) * don't return 503 if SDK initialization has timed out * add in-repo docs about error/503 behavior (#249) * [ch102255] BigSegments DynamoDB (#245) * add init timeout config option + better test coverage + misc refactoring (#250) * fix example build command * use public prerelease tags instead of private dependencies * fix Go installation in CI * update SDK dependencies for JSON number parsing bugfix * update gorilla/mux to 1.8.0 * update OpenCensus packages * add Go 1.16 CI + "latest Go" CI + use latest 1.15 patch for release * cimg images use "current", not "latest" * seems there isn't any cimg/go "latest" or "current" * add daily package build test in CI * job names * bump SDK version for traffic allocation feature * [ch113491] update alpine base image (#258) * use latest prerelease SDK * fix enabling of test tags in CI * add DynamoDB docker image in CI * set a polling base URI in end-to-end tests since big segments logic will use it * fix initialization logic so SDK client creation errors aren't lost when big segments are enabled * fix use of prefix key in DynamoDB + improve tests (#260) * more debug logging, less info logging for big segments logic * make logging of big segments patch version mismatch clearer and use Warn level * fix log parameter * fix DynamoDB updates for big segments metadata * add test to make sure sync time and cursor can be updated independently * only start big seg synchronizer if necessary * use SDK GA releases * change applyPatch to exit early on version mismatch; go back to restarting stream in this case * add unit tests for version mismatch behavior + DRY tests * add log assertion * fix retry logic on big segments stream failure * add more logging for big segments connection status * fix logging assertion * add more big segments integration tests * fix overly-time-sensitive file data tests * fix more flaky tests * run big segments tests with DynamoDB too * Migrate transitive dep (jwt-go) to use modern version without vulnerability. * Edit doc * move Relay release logic to .ldrelease script * suppress SDK big segments status query if we've never synced big segments * dump Relay logs including debug logs if integration test fails * include environment prefix in BigSegmentSynchronizer logging * increase big segment integration test timeout (#274) * generate client-side stream pings if big segments have changed * clear big segments cache as needed + simplify state management * fix tests and simplify component creation * use GA releases of SDK packages * disable CI package-build-test in Go 1.16+ * Migrate Relay release to Releaser v2 and support dry run (#278) * Adding degraded doc blurb for big segments (#280) * respect Redis password & TLS options for big segments; add Redis password integration tests * redact Redis URL password in logs and status resource * update go-server-sdk-redis-redigo to 1.2.1 for Redis URL logging fix * Part 1, add the config and the documentation for the new config * Part 2, Add the configuration validation and test * Part 3, the actual logic to include the headers in the CORS Access-Control-Allow-Headers * Linter * update Alpine version to 3.14.2 to fix openssl CVEs * Fix the global variable modification * Go format * turn off unnecessary metrics integrations in config for Docker smoke test * rename test.env to smoke-test.env to clarify what it's for * fix setting of custom Access-Control-Allow-Origin and add test (#285) * add more explanatory test output and more verbose debugging for big segments integration tests (#287) * update to Go 1.16.10 + Alpine 3.14.3; add some docs about releases (#288) * update go-server-sdk-consul version for Consul API version update * override x/crypto dependency version for CVE-2020-29652 * bump Prometheus dependency to eliminate jwt-go vulnerability * drop support for Go 1.14 & 1.15 * make sure defaults are always applied for base URL properties * rm unused * rm unnecessary linter directive * add separate configuration for server-side/client-side SDK base URLs & update the defaults * remove Whitesource CI job + remove obsolete dependency issue note * don't include any big segment status info in status resource unless that feature is active (#296) * don't include any big segment status info in status resource unless that feature is active * fix Big Segments staleness logic in status resource * documentation * update x/text package for vulnerability GO-2021-0113 * add Trivy security scan to CI (#297) * add daily re-scan with Trivy * update Go version to 1.17.6 (#301) * always terminate if auto-config stream fails with a fatal error * pass along tags header when proxying events * comments, rm debugging * fix auth header logic * fix auth header logic some more * comments * add tags header to CORS header whitelist (#304) * update to Alpine 3.14.4 for CVE-2022-0778 fix * force upgrade of openssl in Alpine * also upgrade libretls * fix it in both files * update to Alpine 3.14.5 for CVE-2022-0778/CVE-2018-25032 (#308) * update to Alpine 3.14.5 for CVE-2022-0778 * revert patches that are now included in Alpine 3.14.5 * add scripts for checking and updating Go/Alpine versions (#309) * update to Alpine 3.14.5 for CVE-2022-0778 * add scripts for checking and updating Go/Alpine versions * also make sure the Docker images really exist * update CONTRIBUTING.md * fix file rename * revert patches that are now included in Alpine 3.14.5 * update Alpine to 3.14.6 for CVE-2022-28391 Co-authored-by: Eli Bishop <eli@launchdarkly.com> Co-authored-by: Patrick Kaeding <pkaeding@launchdarkly.com> Co-authored-by: Ben Woskow <48036130+bwoskow-ld@users.noreply.github.com> Co-authored-by: LaunchDarklyCI <dev@launchdarkly.com> Co-authored-by: Ben Woskow <bwoskow@launchdarkly.com> Co-authored-by: Andrew Shannon Brown <ashanbrown@users.noreply.github.com> Co-authored-by: hroederld <hroeder@launchdarkly.com> Co-authored-by: LaunchDarklyReleaseBot <launchdarklyreleasebot@launchdarkly.com> Co-authored-by: Dan Richelson <drichelson@launchdarkly.com> Co-authored-by: Dan Richelson <drichelson@users.noreply.github.com> Co-authored-by: Louis Chan <lchan@launchdarkly.com> Co-authored-by: Louis Chan <91093020+louis-launchdarkly@users.noreply.github.com>
launchdarkly · Apr 5, 2022 · 9c1decd · 9c1decd
1 parent 4007231
commit 9c1decd
Show file tree

Hide file tree

Showing 8 changed files with 196 additions and 24 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -269,6 +269,10 @@ jobs:
       - checkout
       - setup_remote_docker  # start docker engine
 
+      - run:
+          name: verify Go and Alpine versions
+          command: ./scripts/verify-release-versions.sh
+
       - run:
           name: add package source for Trivy
           command: |

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -66,12 +66,9 @@ As this is a larger codebase than most LaunchDarkly open-source projects, we hav
 
 The published `ld-relay` Docker image embeds specific versions of the Alpine OS and the Go runtime. We update these to take advantage of patch releases for both Alpine and Go.
 
-These versions are specified in several places. For the published `ld-relay` image:
+These versions are specified in several places. To ensure that they're updated consistently, and that the CI builds are using the same, use the following scripts:
 
-* The Alpine version is specified by the `FROM` line in `Dockerfile.goreleaser`.
-* The Go version is specified by the `image` property in `.ldrelease/config.yml`. Basically, we run a Docker container with some version of Go in it, and within that container we will be running `goreleaser`. Then the `goreleaser` tool will look at `Dockerfile.goreleaser` to provide the base image, and it will embed whatever version of the Go runtime it is running on in the published executable.
+* `./scripts/update-go-version.sh VERSION` - updates CI and Docker configuration to use the specified Go version (do not include the `v` prefix).
+* `./scripts/update-alpine-version.sh VERSION` - updates CI and Docker configuration to use the specified Alpine version.
 
-When we change these versions, we should also update our test builds to match the versions we are releasing with:
-
-* In `.circleci/config.yml`, update the default value of `go-release-version`.
-* In `Dockerfile` (which is used for CI tests, not for the release), update the `FROM` line to an image in the format `golang:$(GO_VERSION)-alpine$(ALPINE_VERSION)`.
+The script `./scripts/verify-release-versions.sh` verifies that the current configurations are consistent. This is run automatically in CI, and also whenever you use the scripts above.
diff --git a/Dockerfile b/Dockerfile
@@ -21,7 +21,7 @@ ENV GOPATH=/go
 
 RUN go build -a -o ldr .
 
-FROM alpine:3.14.5
+FROM alpine:3.14.6
 
 RUN addgroup -g 1000 -S ldr-user && \
     adduser -u 1000 -S ldr-user -G ldr-user && \

diff --git a/Dockerfile.goreleaser b/Dockerfile.goreleaser
@@ -2,7 +2,7 @@
 
 # See .ldrelease/config.yml for an explanation of the build/release process.
 
-FROM alpine:3.14.5
+FROM alpine:3.14.6
 # See "Runtime platform versions" in CONTRIBUTING.md
 
 RUN apk add --no-cache \

diff --git a/scripts/release.sh b/scripts/release.sh
diff --git a/scripts/update-alpine-version.sh b/scripts/update-alpine-version.sh
@@ -0,0 +1,49 @@
+#!/bin/bash
+
+set -e
+
+# update-alpine-version <Alpine version string (without a "v")>
+# This script updates all configuration files in the repository that reference the Alpine version that
+# will be used to build Docker images.
+
+VERSION=$1
+if [ -z "${VERSION}" ]; then
+  echo must specify Alpine version
+  exit 1
+fi
+
+cd $(dirname $0)/..
+
+dockerfile_for_tests=Dockerfile
+dockerfile_for_releases=Dockerfile.goreleaser
+
+function ensure_file_was_changed() {
+  filename=$1
+  if (( $(diff ${filename} ${filename}.tmp | grep '^>' | wc -l) == 0 )); then
+    echo "failed to update Alpine version in ${filename}; sed expression did not match any lines"
+    for f in ${ldrelease_config_file} ${circleci_config_file} ${dockerfile_for_tests}; do
+      rm -r ${f}.tmp
+    done
+    exit 1
+  fi
+}
+
+# the golang: Docker images are only specific to an Alpine minor version, not a patch version
+MINOR_VERSION=${VERSION%.*}
+sed <${dockerfile_for_tests} >${dockerfile_for_tests}.tmp \
+  -e "s#-alpine.* as builder#-alpine${MINOR_VERSION} as builder#" \
+  -e "s#FROM alpine:.*#FROM alpine:${VERSION}#"
+ensure_file_was_changed ${dockerfile_for_tests}
+
+sed <${dockerfile_for_releases} >${dockerfile_for_releases}.tmp \
+  -e "s#FROM *alpine:.*#\FROM alpine:${VERSION}#"
+ensure_file_was_changed ${dockerfile_for_releases}
+
+for f in ${dockerfile_for_tests} ${dockerfile_for_releases}; do
+  mv ${f}.tmp ${f}
+  echo "updated ${f}"
+done
+
+echo
+
+$(dirname $0)/verify-release-versions.sh
diff --git a/scripts/update-go-release-version.sh b/scripts/update-go-release-version.sh
@@ -0,0 +1,51 @@
+#!/bin/bash
+
+set -e
+
+# update-go-release-version <Go version string (without a "v")>
+# This script updates all configuration files in the repository that reference the Go version that
+# will be used to compile releases.
+
+VERSION=$1
+if [ -z "${VERSION}" ]; then
+  echo must specify Go version
+  exit 1
+fi
+
+cd $(dirname $0)/..
+
+ldrelease_config_file=.ldrelease/config.yml
+circleci_config_file=.circleci/config.yml
+dockerfile_for_tests=Dockerfile
+
+function ensure_file_was_changed() {
+  filename=$1
+  if (( $(diff ${filename} ${filename}.tmp | grep '^>' | wc -l) != 1 )); then
+    echo "failed to update Go version in ${filename}; sed expression did not match any lines or matched more than one line"
+    for f in ${ldrelease_config_file} ${circleci_config_file} ${dockerfile_for_tests}; do
+      rm -r ${f}.tmp
+    done
+    exit 1
+  fi
+}
+
+sed <${ldrelease_config_file} >${ldrelease_config_file}.tmp \
+  -e "/image:/s#cimg/go:[^ ]*#cimg/go:${VERSION}#"
+ensure_file_was_changed ${ldrelease_config_file}
+
+sed <${circleci_config_file} >${circleci_config_file}.tmp \
+  -e "/go-release-version:/,/default:/s#default: *\"[^\"]*\"#default: ${VERSION}#"
+ensure_file_was_changed ${circleci_config_file}
+
+sed <${dockerfile_for_tests} >${dockerfile_for_tests}.tmp \
+  -e "s#FROM *golang:[^-]#FROM golang:${VERSION}#"
+ensure_file_was_changed ${dockerfile_for_tests}
+
+for f in ${ldrelease_config_file} ${circleci_config_file}; do
+  mv ${f}.tmp ${f}
+  echo "updated ${f}"
+done
+
+echo
+
+$(dirname $0)/verify-release-versions.sh
diff --git a/scripts/verify-release-versions.sh b/scripts/verify-release-versions.sh
@@ -0,0 +1,86 @@
+#!/bin/bash
+
+# verify-release-versions.sh  (no parameters)
+# This script checks all of the configuration files where a Go version and/or Alpine version is
+# mentioned in the context of producing releases, and makes sure they are consistent with each other.
+# Unfortunately it's not possible to have these be sourced from just one file, because the CircleCI
+# configuration and the .ldrelease configuration require Docker image names to be hard-coded in their
+# respective config files.
+
+set -e
+
+cd $(dirname $0)/..
+
+ldrelease_config_file=.ldrelease/config.yml
+circleci_config_file=.circleci/config.yml
+dockerfile_for_tests=Dockerfile
+dockerfile_for_releases=Dockerfile.goreleaser
+
+function fail_for_file() {
+  echo "failed to parse $1 version from $2"
+  exit 1
+}
+
+LDRELEASE_GO_VERSION=$(sed <${ldrelease_config_file} -n 's#.*image: *cimg/go:\([1-9.]*\).*#\1#p')
+if [ -z "${LDRELEASE_GO_VERSION}" ]; then
+  fail_for_file Go ${ldrelease_config_file}
+fi
+echo "${ldrelease_config_file} (for building releases) is using Go ${LDRELEASE_GO_VERSION}"
+
+CIRCLECI_GO_VERSION=$(sed <${circleci_config_file} -n '/go-release-version:/,/default/{/default:/{p;q;};}' | \
+    sed 's/.*default:.*"\(.*\)".*/\1/')
+if [ -z "${CIRCLECI_GO_VERSION}" ]; then
+  fail_for_file Go ${circleci_config_file}
+fi
+echo "${circleci_config_file} (for CI tests) is using Go ${CIRCLECI_GO_VERSION}"
+
+DOCKERFILE_TESTS_GO_VERSION=$(sed <${dockerfile_for_tests} -n 's/FROM *golang:\([^-]*\)-.*/\1/p')
+if [ -z "${DOCKERFILE_TESTS_GO_VERSION}" ]; then
+  fail_for_file ${dockerfile_for_tests}
+fi
+echo "${dockerfile_for_tests} (for images in CI tests) is using Go ${DOCKERFILE_TESTS_GO_VERSION}"
+
+if [[ "${CIRCLECI_GO_VERSION}" != "${LDRELEASE_GO_VERSION}" || \
+     "${DOCKERFILE_TESTS_GO_VERSION}" != "${LDRELEASE_GO_VERSION}" ]]; then
+  echo; echo "Go versions are out of sync!"
+  exit 1
+fi
+
+echo "Go versions are in sync"
+
+DOCKERFILE_TESTS_ALPINE_VERSION=$(sed <${dockerfile_for_tests} -n 's/FROM alpine:\(.*\).*/\1/p')
+if [ -z "${DOCKERFILE_TESTS_ALPINE_VERSION}" ]; then
+  fail_for_file Alpine ${dockerfile_for_tests}
+fi
+DOCKERFILE_TESTS_ALPINE_MINOR_VERSION=$(sed <${dockerfile_for_tests} -n 's/FROM *golang:.*alpine\([^ ]*\).*/\1/p')
+if [ -z "${DOCKERFILE_TESTS_ALPINE_MINOR_VERSION}" ]; then
+  fail_for_file "Alpine minor" ${dockerfile_for_tests}
+fi
+echo "${dockerfile_for_tests} (for images in CI tests) is using Alpine ${DOCKERFILE_TESTS_ALPINE_VERSION} (and minor version ${DOCKERFILE_TESTS_ALPINE_MINOR_VERSION})"
+
+DOCKERFILE_RELEASES_ALPINE_VERSION=$(sed <${dockerfile_for_releases} -n 's/FROM *alpine:\([^ ]*\).*/\1/p')
+if [ -z "${DOCKERFILE_RELEASES_ALPINE_VERSION}" ]; then
+  fail_for_file Alpine ${dockerfile_for_releases}
+fi
+echo "${dockerfile_for_releases} (for building releases) is using Alpine ${DOCKERFILE_RELEASES_ALPINE_VERSION}"
+
+if [[ "${DOCKERFILE_TESTS_ALPINE_VERSION}" != "${DOCKERFILE_RELEASES_ALPINE_VERSION}" ]]; then
+  echo; echo "Alpine versions are out of sync!"
+  exit 1
+fi
+
+if [[ "${DOCKERFILE_TESTS_ALPINE_MINOR_VERSION}" != "${DOCKERFILE_TESTS_ALPINE_VERSION%.*}" ]]; then
+  echo; echo "Alpine minor version is out of sync!"
+  exit 1
+fi
+
+echo "Alpine versions are in sync"
+
+for docker_image in \
+  "cimg/go:${LDRELEASE_GO_VERSION}" \
+  "alpine:${DOCKERFILE_RELEASES_ALPINE_VERSION}" \
+  "golang:${LDRELEASE_GO_VERSION}-alpine${DOCKERFILE_TESTS_ALPINE_MINOR_VERSION}"
+do
+  docker pull ${docker_image} >/dev/null 2>/dev/null || { echo; echo "Docker image ${docker_image} is not available!"; exit 1; }
+  echo "verified that Docker image ${docker_image} is available"
+done