Skip to content

Commit

Permalink
prepare 6.7.0 release (#174)
Browse files Browse the repository at this point in the history
* fix data race

* (offline mode #2) add configuration for offline mode (#199)

* (offline mode #3) factor out types that are shared between autoconfig and filedata (#200)

* add full integration test of Relay in standard config mode

* add missing functions

* also run database integrations test

* better logging

* even better logging

* comments, misc cleanup

* more logging + Docker helper improvements

* (offline mode #4) full implementation of offline mode in Relay app (#202)

* fix goroutine leak related to stream heartbeats

* use correct import path for gcfg

* use correct import path for gcfg

* added whitesource to build

* removed broken environment references from build

* Updating the minimum go version in the contribution guidelines (#213)

* change offline mode env schema to separate dataId from other properties

* offline mode in-repo docs (#214)

* (#6) offline mode integration test + offline mode fixes (#204)

* only run scheduled tests on private repo (#216)

* fix "expiring-but-still-valid SDK key" logic so requests with old key are accepted

* add note about non-support for clustered Redis and Redis Sentinel

* disable diagnostic events & metrics events in offline mode

* add accept-events-but-discard-them behavior for offline mode

* clean up imports

* fix product name in docs

* fix the broken images in our repo docs (#217)

* in auto-config mode, return 503 for client requests till configuration is complete (#219)

* always send stream updates to clients regardless of version checking (#224)

* Enable hourly Relay integration tests against Production (#223)

* Record response body when relay archive download test fails (#225)

There is some additional information provided in the response body for certain 504 error codes and this should allow us to see it.

* Fix the broken production integration tests (#226)

* improve metrics documentation and fix route strings in docs (#227)

* improve metrics documentation and fix route strings in docs

* clarify mobile + browser terminology

* use more efficient jsonstream encoding/decoding for stream data and evaluations (#228)

* recognize alias events from non-v3-schema payloads and forward them unchanged

* environment should still be usable even if the client timed out

* bump dependency versions for SDK fixes

* update go-server-sdk-dynamodb to get newer AWS SDK

* minor clarification about Relay.Close (#234)

* Use the Go releaser template (#233)

* [ch102248] big segment sync with redis (#235)

* fix makefile so it tries building all the test code first before running any of it (#239)

* (big segments #1) add basic big segments configuration for SDK clients (#237)

* (big segments #2) add more abstraction around big segments implementation for testability (#238)

* [102253] bigsegment status / config (#242)

* big segments integration test + misc fixes (#240)

* use latest URL paths for big segments endpoints

* add SDK DynamoDB integration for big segments (#241)

* fix broken link in Markdown docs (#246)

* make sure newly added credentials for existing environments are accepted in requests (#244)

* don't return 503 if SDK initialization has timed out

* add in-repo docs about error/503 behavior (#249)

* [ch102255] BigSegments DynamoDB (#245)

* add init timeout config option + better test coverage + misc refactoring (#250)

* fix example build command

* use public prerelease tags instead of private dependencies

* fix Go installation in CI

* update SDK dependencies for JSON number parsing bugfix

* update gorilla/mux to 1.8.0

* update OpenCensus packages

* add Go 1.16 CI + "latest Go" CI + use latest 1.15 patch for release

* cimg images use "current", not "latest"

* seems there isn't any cimg/go "latest" or "current"

* add daily package build test in CI

* job names

* bump SDK version for traffic allocation feature

* [ch113491] update alpine base image (#258)

* use latest prerelease SDK

* fix enabling of test tags in CI

* add DynamoDB docker image in CI

* set a polling base URI in end-to-end tests since big segments logic will use it

* fix initialization logic so SDK client creation errors aren't lost when big segments are enabled

* fix use of prefix key in DynamoDB + improve tests (#260)

* more debug logging, less info logging for big segments logic

* make logging of big segments patch version mismatch clearer and use Warn level

* fix log parameter

* fix DynamoDB updates for big segments metadata

* add test to make sure sync time and cursor can be updated independently

* only start big seg synchronizer if necessary

* use SDK GA releases

* change applyPatch to exit early on version mismatch; go back to restarting stream in this case

* add unit tests for version mismatch behavior + DRY tests

* add log assertion

* fix retry logic on big segments stream failure

* add more logging for big segments connection status

* fix logging assertion

* add more big segments integration tests

* fix overly-time-sensitive file data tests

* fix more flaky tests

* run big segments tests with DynamoDB too

* Migrate transitive dep (jwt-go) to use modern version without vulnerability.

* Edit doc

* move Relay release logic to .ldrelease script

* suppress SDK big segments status query if we've never synced big segments

* dump Relay logs including debug logs if integration test fails

* include environment prefix in BigSegmentSynchronizer logging

* increase big segment integration test timeout (#274)

* generate client-side stream pings if big segments have changed

* clear big segments cache as needed + simplify state management

* fix tests and simplify component creation

* use GA releases of SDK packages

* disable CI package-build-test in Go 1.16+

* Migrate Relay release to Releaser v2 and support dry run (#278)

* Adding degraded doc blurb for big segments (#280)

* respect Redis password & TLS options for big segments; add Redis password integration tests

* redact Redis URL password in logs and status resource

* update go-server-sdk-redis-redigo to 1.2.1 for Redis URL logging fix

* Part 1, add the config and the documentation for the new config

* Part 2, Add the configuration validation and test

* Part 3, the actual logic to include the headers in the CORS Access-Control-Allow-Headers

* Linter

* update Alpine version to 3.14.2 to fix openssl CVEs

* Fix the global variable modification

* Go format

* turn off unnecessary metrics integrations in config for Docker smoke test

* rename test.env to smoke-test.env to clarify what it's for

* fix setting of custom Access-Control-Allow-Origin and add test (#285)

* add more explanatory test output and more verbose debugging for big segments integration tests (#287)

* update to Go 1.16.10 + Alpine 3.14.3; add some docs about releases (#288)

* update go-server-sdk-consul version for Consul API version update

* override x/crypto dependency version for CVE-2020-29652

* bump Prometheus dependency to eliminate jwt-go vulnerability

* drop support for Go 1.14 & 1.15

* make sure defaults are always applied for base URL properties

* rm unused

* rm unnecessary linter directive

* add separate configuration for server-side/client-side SDK base URLs & update the defaults

* remove Whitesource CI job + remove obsolete dependency issue note

* don't include any big segment status info in status resource unless that feature is active (#296)

* don't include any big segment status info in status resource unless that feature is active

* fix Big Segments staleness logic in status resource

* documentation

* update x/text package for vulnerability GO-2021-0113

* add Trivy security scan to CI (#297)

* add daily re-scan with Trivy

* update Go version to 1.17.6 (#301)

* always terminate if auto-config stream fails with a fatal error

* pass along tags header when proxying events

* comments, rm debugging

* fix auth header logic

* fix auth header logic some more

* comments

* add tags header to CORS header whitelist (#304)

* update to Alpine 3.14.4 for CVE-2022-0778 fix

Co-authored-by: Eli Bishop <eli@launchdarkly.com>
Co-authored-by: Patrick Kaeding <pkaeding@launchdarkly.com>
Co-authored-by: Ben Woskow <48036130+bwoskow-ld@users.noreply.github.com>
Co-authored-by: LaunchDarklyCI <dev@launchdarkly.com>
Co-authored-by: Ben Woskow <bwoskow@launchdarkly.com>
Co-authored-by: Andrew Shannon Brown <ashanbrown@users.noreply.github.com>
Co-authored-by: hroederld <hroeder@launchdarkly.com>
Co-authored-by: LaunchDarklyReleaseBot <launchdarklyreleasebot@launchdarkly.com>
Co-authored-by: Dan Richelson <drichelson@launchdarkly.com>
Co-authored-by: Dan Richelson <drichelson@users.noreply.github.com>
Co-authored-by: Louis Chan <lchan@launchdarkly.com>
Co-authored-by: Louis Chan <91093020+louis-launchdarkly@users.noreply.github.com>
  • Loading branch information
13 people authored Mar 24, 2022
1 parent 80e1e6f commit 893b0a3
Show file tree
Hide file tree
Showing 13 changed files with 585 additions and 186 deletions.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ require (
github.com/launchdarkly/go-server-sdk-consul v1.0.1
github.com/launchdarkly/go-server-sdk-dynamodb v1.1.0
github.com/launchdarkly/go-server-sdk-redis-redigo v1.2.1
github.com/launchdarkly/go-test-helpers/v2 v2.2.0
github.com/launchdarkly/go-test-helpers/v2 v2.3.1
github.com/mattn/go-isatty v0.0.14 // indirect
github.com/miekg/dns v1.1.43 // indirect
github.com/mitchellh/mapstructure v1.4.2 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -291,8 +291,9 @@ github.com/launchdarkly/go-server-sdk-dynamodb v1.1.0 h1:fE0OwYjsItOOHR2/kq5StGG
github.com/launchdarkly/go-server-sdk-dynamodb v1.1.0/go.mod h1:zKUi5j+UnvISac34dgjWtZ/N0rwlLrdAwWTBtu9rIPY=
github.com/launchdarkly/go-server-sdk-redis-redigo v1.2.1 h1:5KhwXcx+0sqxjDf4m/irLCohe/8Fh72zzsC6XU3aTMc=
github.com/launchdarkly/go-server-sdk-redis-redigo v1.2.1/go.mod h1:rcydnSjPuE8w5HYeOg/l98QSFUT/lM9Txk9/pbyU30k=
github.com/launchdarkly/go-test-helpers/v2 v2.2.0 h1:L3kGILP/6ewikhzhdNkHy1b5y4zs50LueWenVF0sBbs=
github.com/launchdarkly/go-test-helpers/v2 v2.2.0/go.mod h1:L7+th5govYp5oKU9iN7To5PgznBuIjBPn+ejqKR0avw=
github.com/launchdarkly/go-test-helpers/v2 v2.3.1 h1:KXUAQVTeHNcWVDVQ94uEkybI+URXI9rEd7E553EsZFw=
github.com/launchdarkly/go-test-helpers/v2 v2.3.1/go.mod h1:L7+th5govYp5oKU9iN7To5PgznBuIjBPn+ejqKR0avw=
github.com/mailru/easyjson v0.7.6 h1:8yTIVnZgCoiM1TgqoeTl+LfU5Jg6/xL3QhGQnimLYnA=
github.com/mailru/easyjson v0.7.6/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
Expand Down Expand Up @@ -605,7 +606,6 @@ golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.5/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
Expand Down
1 change: 1 addition & 0 deletions internal/core/internal/browser/cors.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ var DefaultAllowedHeaders = strings.Join([]string{ //nolint:gochecknoglobals
"X-LaunchDarkly-Payload-ID",
"X-LaunchDarkly-Wrapper",
events.EventSchemaHeader,
events.TagsHeader,
}, ",")

// CORSContext represents a scope that has a specific set of allowed origins for CORS requests. This
Expand Down
81 changes: 45 additions & 36 deletions internal/core/internal/events/event-relay.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ import (
"io/ioutil"
"net/http"
"reflect"
"strconv"
"strings"
"sync"
"time"

c "github.com/launchdarkly/ld-relay/v6/config"
"github.com/launchdarkly/ld-relay/v6/internal/basictypes"
Expand All @@ -25,12 +25,18 @@ type eventVerbatimRelay struct {
publisher EventPublisher
}

const defaultEventQueueCleanupInterval = time.Hour

const (
// SummaryEventsSchemaVersion is the minimum event schema that supports summary events
// SummaryEventsSchemaVersion is the minimum event schema that supports summary events.
SummaryEventsSchemaVersion = 3

// EventSchemaHeader is an HTTP header that describes the schema version for event requests
// EventSchemaHeader is an HTTP header that describes the schema version for event requests.
EventSchemaHeader = "X-LaunchDarkly-Event-Schema"

// TagsHeader is an HTTP header that may be sent by SDKs that support application metadata.
// We copy the value of this header when proxying events.
TagsHeader = "X-LaunchDarkly-Tags"
)

// EventDispatcher relays events to LaunchDarkly for an environment
Expand All @@ -40,16 +46,17 @@ type EventDispatcher struct {
}

type analyticsEventEndpointDispatcher struct {
config c.EventsConfig
httpClient *http.Client
httpConfig httpconfig.HTTPConfig
authKey c.SDKCredential
remotePath string
verbatimRelay *eventVerbatimRelay
summarizingRelay *eventSummarizingRelay
storeAdapter *store.SSERelayDataStoreAdapter
loggers ldlog.Loggers
mu sync.Mutex
config c.EventsConfig
httpClient *http.Client
httpConfig httpconfig.HTTPConfig
authKey c.SDKCredential
remotePath string
verbatimRelay *eventVerbatimRelay
summarizingRelay *eventSummarizingRelay
storeAdapter *store.SSERelayDataStoreAdapter
eventQueueCleanupInterval time.Duration
loggers ldlog.Loggers
mu sync.Mutex
}

type diagnosticEventEndpointDispatcher struct {
Expand Down Expand Up @@ -82,16 +89,14 @@ func (r *analyticsEventEndpointDispatcher) dispatch(w http.ResponseWriter, req *
return
}

payloadVersion, _ := strconv.Atoi(req.Header.Get(EventSchemaHeader))
if payloadVersion == 0 {
payloadVersion = 1
}
r.loggers.Debugf("Received %d events (v%d) to be proxied to %s", len(evts), payloadVersion, r.remotePath)
if payloadVersion >= SummaryEventsSchemaVersion {
metadata := GetEventPayloadMetadata(req)

r.loggers.Debugf("Received %d events (v%d) to be proxied to %s", len(evts), metadata.SchemaVersion, r.remotePath)
if metadata.SchemaVersion >= SummaryEventsSchemaVersion {
// New-style events that have already gone through summarization - deliver them as-is
r.getVerbatimRelay().enqueue(evts)
r.getVerbatimRelay().enqueue(metadata, evts)
} else {
r.getSummarizingRelay().enqueue(evts, payloadVersion)
r.getSummarizingRelay().enqueue(metadata, evts)
}
})
}
Expand All @@ -102,7 +107,7 @@ func (r *analyticsEventEndpointDispatcher) replaceCredential(newCredential c.SDK
if reflect.TypeOf(r.authKey) == reflect.TypeOf(newCredential) {
r.authKey = newCredential
if r.summarizingRelay != nil {
r.summarizingRelay.eventSender.replaceCredential(newCredential)
r.summarizingRelay.replaceCredential(newCredential)
}
if r.verbatimRelay != nil {
r.verbatimRelay.publisher.ReplaceCredential(newCredential)
Expand Down Expand Up @@ -176,7 +181,8 @@ func (r *analyticsEventEndpointDispatcher) getSummarizingRelay() *eventSummarizi
r.mu.Lock()
defer r.mu.Unlock()
if r.summarizingRelay == nil {
r.summarizingRelay = newEventSummarizingRelay(r.config, r.httpConfig, r.authKey, r.storeAdapter, r.loggers, r.remotePath)
r.summarizingRelay = newEventSummarizingRelay(r.config, r.httpConfig, r.authKey, r.storeAdapter,
r.loggers, r.remotePath, r.eventQueueCleanupInterval)
}
return r.summarizingRelay
}
Expand All @@ -188,7 +194,7 @@ func (r *analyticsEventEndpointDispatcher) flush() { //nolint:unused // used onl
r.verbatimRelay.publisher.Flush()
}
if r.summarizingRelay != nil {
r.summarizingRelay.eventProcessor.Flush()
r.summarizingRelay.flush()
}
}

Expand All @@ -201,24 +207,25 @@ func NewEventDispatcher(
config c.EventsConfig,
httpConfig httpconfig.HTTPConfig,
storeAdapter *store.SSERelayDataStoreAdapter,
eventQueueCleanupInterval time.Duration, // normally zero to use the default; overridden in tests
) *EventDispatcher {
ep := &EventDispatcher{
analyticsEndpoints: map[basictypes.SDKKind]*analyticsEventEndpointDispatcher{
basictypes.ServerSDK: newAnalyticsEventEndpointDispatcher(sdkKey,
config, httpConfig, storeAdapter, loggers, "/bulk"),
config, httpConfig, storeAdapter, loggers, "/bulk", eventQueueCleanupInterval),
},
diagnosticEndpoints: map[basictypes.SDKKind]*diagnosticEventEndpointDispatcher{
basictypes.ServerSDK: newDiagnosticEventEndpointDispatcher(config, httpConfig, loggers, "/diagnostic"),
},
}
if mobileKey != "" {
ep.analyticsEndpoints[basictypes.MobileSDK] = newAnalyticsEventEndpointDispatcher(mobileKey,
config, httpConfig, storeAdapter, loggers, "/mobile")
config, httpConfig, storeAdapter, loggers, "/mobile", eventQueueCleanupInterval)
ep.diagnosticEndpoints[basictypes.MobileSDK] = newDiagnosticEventEndpointDispatcher(config, httpConfig, loggers, "/mobile/events/diagnostic")
}
if envID != "" {
ep.analyticsEndpoints[basictypes.JSClientSDK] = newAnalyticsEventEndpointDispatcher(envID, config, httpConfig, storeAdapter, loggers,
"/events/bulk/"+string(envID))
"/events/bulk/"+string(envID), eventQueueCleanupInterval)
ep.diagnosticEndpoints[basictypes.JSClientSDK] = newDiagnosticEventEndpointDispatcher(config, httpConfig, loggers,
"/events/diagnostic/"+string(envID))
}
Expand Down Expand Up @@ -273,15 +280,17 @@ func newAnalyticsEventEndpointDispatcher(
storeAdapter *store.SSERelayDataStoreAdapter,
loggers ldlog.Loggers,
remotePath string,
eventQueueCleanupInterval time.Duration,
) *analyticsEventEndpointDispatcher {
return &analyticsEventEndpointDispatcher{
authKey: authKey,
config: config,
httpClient: httpConfig.Client(),
httpConfig: httpConfig,
storeAdapter: storeAdapter,
loggers: loggers,
remotePath: remotePath,
authKey: authKey,
config: config,
httpClient: httpConfig.Client(),
httpConfig: httpConfig,
storeAdapter: storeAdapter,
loggers: loggers,
remotePath: remotePath,
eventQueueCleanupInterval: eventQueueCleanupInterval,
}
}

Expand Down Expand Up @@ -310,8 +319,8 @@ func newEventVerbatimRelay(
return res
}

func (er *eventVerbatimRelay) enqueue(evts []json.RawMessage) {
er.publisher.Publish(evts...)
func (er *eventVerbatimRelay) enqueue(metadata EventPayloadMetadata, evts []json.RawMessage) {
er.publisher.Publish(metadata, evts...)
}

func (er *eventVerbatimRelay) close() {
Expand Down
18 changes: 17 additions & 1 deletion internal/core/internal/events/event-relay_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ import (

"github.com/launchdarkly/go-configtypes"
"github.com/launchdarkly/go-test-helpers/v2/httphelpers"
m "github.com/launchdarkly/go-test-helpers/v2/matchers"
"gopkg.in/launchdarkly/go-sdk-common.v2/ldlog"
"gopkg.in/launchdarkly/go-sdk-common.v2/ldlogtest"
ldevents "gopkg.in/launchdarkly/go-sdk-events.v1"
Expand Down Expand Up @@ -59,6 +60,10 @@ var testJSClientEndpointInfo = testEndpointInfo{
}
var allTestEndpoints = []testEndpointInfo{testServerEndpointInfo, testMobileEndpointInfo, testJSClientEndpointInfo}

type eventRelayTestOptions struct {
eventQueueCleanupInterval time.Duration
}

type eventRelayTestParams struct {
dispatcher *EventDispatcher
requestsCh <-chan httphelpers.HTTPRequestInfo
Expand All @@ -71,6 +76,16 @@ func eventRelayTest(
testEnv st.TestEnv,
eventsConfig config.EventsConfig,
fn func(eventRelayTestParams),
) {
eventRelayTestWithOptions(t, testEnv, eventsConfig, eventRelayTestOptions{}, fn)
}

func eventRelayTestWithOptions(
t *testing.T,
testEnv st.TestEnv,
eventsConfig config.EventsConfig,
opts eventRelayTestOptions,
fn func(eventRelayTestParams),
) {
mockLog := ldlogtest.NewMockLog()
mockLog.Loggers.SetMinLevel(ldlog.Debug)
Expand All @@ -97,6 +112,7 @@ func eventRelayTest(
eventsConfig,
httpConfig,
makeStoreAdapterWithExistingStore(store),
opts.eventQueueCleanupInterval,
)
defer dispatcher.Close()

Expand Down Expand Up @@ -182,7 +198,7 @@ func TestSummarizingEventHandlers(t *testing.T) {
assert.Equal(t, e.analyticsPath, r.Request.URL.Path)
assert.Equal(t, e.authKey, r.Request.Header.Get("Authorization"))
assert.Equal(t, strconv.Itoa(SummaryEventsSchemaVersion), r.Request.Header.Get(EventSchemaHeader))
assert.JSONEq(t, expectedSummarizedFeatureEventsOutputUnknownFlagWithVersion, string(r.Body))
m.In(t).Assert(r.Body, m.JSONStrEqual(expectedSummarizedFeatureEventsOutputUnknownFlagWithVersion))
})
})
}
Expand Down
Loading

0 comments on commit 893b0a3

Please sign in to comment.