Skip to content

Commit

Permalink
master pull
Browse files Browse the repository at this point in the history
  • Loading branch information
@achettyiitr
achettyiitr committed May 25, 2023
2 parents eb26f72 + b245915 commit 25a24d8
Show file tree
Hide file tree
Showing 15 changed files with 179 additions and 62 deletions.
63 changes: 63 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,68 @@
# Changelog

## [1.9.0](https://github.com/rudderlabs/rudder-server/compare/v1.8.0...v1.9.0) (2023-05-25)


### Features

* **batchrouter:** introduce isolation levels and concurrency limiters ([#3248](https://github.com/rudderlabs/rudder-server/issues/3248)) ([d90ea68](https://github.com/rudderlabs/rudder-server/commit/d90ea68178d8ad6175d0145568fd7e2d651ed760))
* support for incremental updates while fetching backend config ([#3175](https://github.com/rudderlabs/rudder-server/issues/3175)) ([9de904d](https://github.com/rudderlabs/rudder-server/commit/9de904d68ee122f4c2df4f1bf9335f9a851470fa))
* wh Endpoint To Fetch Tables Per Connection ([#3279](https://github.com/rudderlabs/rudder-server/issues/3279)) ([ea7d5ce](https://github.com/rudderlabs/rudder-server/commit/ea7d5ce8b8fe0b5e8f3f3e32b665d9b94232c58c))


### Bug Fixes

* batchrouter acquiring a read lock twice ([#3363](https://github.com/rudderlabs/rudder-server/issues/3363)) ([6fd8552](https://github.com/rudderlabs/rudder-server/commit/6fd855258daf1a4fbbd7130072d66e4e2e3d3d55))
* deltalake parquet datatype support ([#3393](https://github.com/rudderlabs/rudder-server/issues/3393)) ([0a6c85e](https://github.com/rudderlabs/rudder-server/commit/0a6c85ec913158169e341957c2bdbc2ffba3ba64))
* external table support for databricks validations ([#3378](https://github.com/rudderlabs/rudder-server/issues/3378)) ([acf0c47](https://github.com/rudderlabs/rudder-server/commit/acf0c47ece7195a445dab52029e773f41845e3f9))
* include source definition type when destination has UT ([#3338](https://github.com/rudderlabs/rudder-server/issues/3338)) ([1ab2f55](https://github.com/rudderlabs/rudder-server/commit/1ab2f553dec3b34f341fdcc707540d5758d69b09))
* kafka secret comes first ([#3307](https://github.com/rudderlabs/rudder-server/issues/3307)) ([19ceebb](https://github.com/rudderlabs/rudder-server/commit/19ceebb6f20c26630a407d33490ed13b72ccf083))
* limiter not respecting WithLimiterTags option ([#3367](https://github.com/rudderlabs/rudder-server/issues/3367)) ([9a3e6fc](https://github.com/rudderlabs/rudder-server/commit/9a3e6fc9432b5ab8d1d26b8ac4ec7e6b1bcdde10))
* max connections for warehouse slaves ([#3341](https://github.com/rudderlabs/rudder-server/issues/3341)) ([1ab2f55](https://github.com/rudderlabs/rudder-server/commit/1ab2f553dec3b34f341fdcc707540d5758d69b09))
* processor panicking during shutdown ([#3396](https://github.com/rudderlabs/rudder-server/issues/3396)) ([4e3981f](https://github.com/rudderlabs/rudder-server/commit/4e3981f31f669af127d88c5f5a78173db000d81d))
* schema forwarder records invalid json in statuses ([#3350](https://github.com/rudderlabs/rudder-server/issues/3350)) ([91b1902](https://github.com/rudderlabs/rudder-server/commit/91b1902973bd22be0994a7b3a3518b1778d79877))
* staging files status when insert ([#3332](https://github.com/rudderlabs/rudder-server/issues/3332)) ([fb7277f](https://github.com/rudderlabs/rudder-server/commit/fb7277f00ded6ccfeb4cd61becc6cdc9aef905aa))
* stash sleep ([#3312](https://github.com/rudderlabs/rudder-server/issues/3312)) ([19ceebb](https://github.com/rudderlabs/rudder-server/commit/19ceebb6f20c26630a407d33490ed13b72ccf083))


### Miscellaneous

* adaptive processor worker sleep time ([#3334](https://github.com/rudderlabs/rudder-server/issues/3334)) ([4a4f293](https://github.com/rudderlabs/rudder-server/commit/4a4f2931ccdd9266597845c5ee7b713424c34bc4))
* add toggle for backendconfig db caching ([#3320](https://github.com/rudderlabs/rudder-server/issues/3320)) ([0d198b8](https://github.com/rudderlabs/rudder-server/commit/0d198b81db518cc5a57ee35669560856b136a952))
* added logs to help debug suppression backup service issue ([#3249](https://github.com/rudderlabs/rudder-server/issues/3249)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* cleanup fetch schema ([#3242](https://github.com/rudderlabs/rudder-server/issues/3242)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* dedup based on message ID ([#3289](https://github.com/rudderlabs/rudder-server/issues/3289)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* dedup based on message ID ([#3289](https://github.com/rudderlabs/rudder-server/issues/3289)) ([a2679d5](https://github.com/rudderlabs/rudder-server/commit/a2679d59d1c42e7fb289fa82e1d338753ddc3a75))
* **deps:** bump cloud.google.com/go/bigquery from 1.51.0 to 1.51.1 ([#3303](https://github.com/rudderlabs/rudder-server/issues/3303)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump cloud.google.com/go/bigquery from 1.51.1 to 1.51.2 ([#3309](https://github.com/rudderlabs/rudder-server/issues/3309)) ([465ad41](https://github.com/rudderlabs/rudder-server/commit/465ad41c2d8a4695bf00923b3b04cad0b31c686f))
* **deps:** bump github.com/aws/aws-sdk-go from 1.44.256 to 1.44.259 ([#3302](https://github.com/rudderlabs/rudder-server/issues/3302)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump github.com/aws/aws-sdk-go from 1.44.256 to 1.44.259 ([#3302](https://github.com/rudderlabs/rudder-server/issues/3302)) ([2476215](https://github.com/rudderlabs/rudder-server/commit/24762152d85c99a501b365844b44fa113ea8ecd1))
* **deps:** bump github.com/aws/aws-sdk-go from 1.44.259 to 1.44.261 ([#3316](https://github.com/rudderlabs/rudder-server/issues/3316)) ([f4abcb1](https://github.com/rudderlabs/rudder-server/commit/f4abcb17a4a2037411daf9d1dcbe65e2b31b86ab))
* **deps:** bump github.com/aws/aws-sdk-go from 1.44.261 to 1.44.262 ([#3322](https://github.com/rudderlabs/rudder-server/issues/3322)) ([a2dd313](https://github.com/rudderlabs/rudder-server/commit/a2dd313c97a127bf6d62e91ccb306518e79b07e5))
* **deps:** bump github.com/aws/aws-sdk-go from 1.44.262 to 1.44.264 ([#3343](https://github.com/rudderlabs/rudder-server/issues/3343)) ([de5c605](https://github.com/rudderlabs/rudder-server/commit/de5c605d7cc8633310ba630426af76cd6f3ae52e))
* **deps:** bump github.com/confluentinc/confluent-kafka-go/v2 from 2.1.0 to 2.1.1 ([#3266](https://github.com/rudderlabs/rudder-server/issues/3266)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump github.com/onsi/ginkgo/v2 from 2.9.4 to 2.9.5 ([#3336](https://github.com/rudderlabs/rudder-server/issues/3336)) ([8aefa7a](https://github.com/rudderlabs/rudder-server/commit/8aefa7a7fd0b1d54238a9964d85047a8d17401a0))
* **deps:** bump github.com/prometheus/common from 0.42.0 to 0.43.0 ([#3293](https://github.com/rudderlabs/rudder-server/issues/3293)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump github.com/rudderlabs/rudder-go-kit from 0.13.0 to 0.13.1 ([#3284](https://github.com/rudderlabs/rudder-server/issues/3284)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump github.com/rudderlabs/rudder-go-kit from 0.13.1 to 0.13.3 ([#3296](https://github.com/rudderlabs/rudder-server/issues/3296)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump github.com/segmentio/kafka-go from 0.4.39 to 0.4.40 ([#3294](https://github.com/rudderlabs/rudder-server/issues/3294)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump go.etcd.io/etcd/client/v3 from 3.5.8 to 3.5.9 ([#3323](https://github.com/rudderlabs/rudder-server/issues/3323)) ([c54f7d4](https://github.com/rudderlabs/rudder-server/commit/c54f7d496aaec106384a2d4f1a4b23c277a0a78f))
* **deps:** bump golang.org/x/oauth2 from 0.7.0 to 0.8.0 ([#3300](https://github.com/rudderlabs/rudder-server/issues/3300)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump golang.org/x/sync from 0.1.0 to 0.2.0 ([#3301](https://github.com/rudderlabs/rudder-server/issues/3301)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump golang.org/x/sync from 0.1.0 to 0.2.0 ([#3301](https://github.com/rudderlabs/rudder-server/issues/3301)) ([feb07aa](https://github.com/rudderlabs/rudder-server/commit/feb07aacc34f9bc0386728d900da441684ce52fd))
* **deps:** bump google.golang.org/api from 0.120.0 to 0.121.0 ([#3286](https://github.com/rudderlabs/rudder-server/issues/3286)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **deps:** bump google.golang.org/api from 0.121.0 to 0.122.0 ([#3310](https://github.com/rudderlabs/rudder-server/issues/3310)) ([d5a506a](https://github.com/rudderlabs/rudder-server/commit/d5a506a4743df6910a375fc7802eb317649c88eb))
* **deps:** bump google.golang.org/grpc from 1.54.0 to 1.55.0 ([#3283](https://github.com/rudderlabs/rudder-server/issues/3283)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* fix flaky test ([#3325](https://github.com/rudderlabs/rudder-server/issues/3325)) ([22f2510](https://github.com/rudderlabs/rudder-server/commit/22f251062ad7225196eb4ef4773dffe25278a20b))
* go-kit v0.13.5 ([#3375](https://github.com/rudderlabs/rudder-server/issues/3375)) ([7352e64](https://github.com/rudderlabs/rudder-server/commit/7352e640813d448727404c7e0816945b36c296fe))
* include error message in error log ([#3348](https://github.com/rudderlabs/rudder-server/issues/3348)) ([bb01437](https://github.com/rudderlabs/rudder-server/commit/bb01437d77b8bac180b27bad3fdeea6d4418a517))
* increase default jobsdb operation timeouts ([#3172](https://github.com/rudderlabs/rudder-server/issues/3172)) ([a28984a](https://github.com/rudderlabs/rudder-server/commit/a28984a96b5550f3a3e79550f09d813f262a4a8b))
* **jobsdb:** granular table count and cache hit stats ([#3372](https://github.com/rudderlabs/rudder-server/issues/3372)) ([a481131](https://github.com/rudderlabs/rudder-server/commit/a481131eb0eddfa504d16cff591e22725ae72fe3))
* moved from gorilla to chi ([#3263](https://github.com/rudderlabs/rudder-server/issues/3263)) ([57231be](https://github.com/rudderlabs/rudder-server/commit/57231befe58b30ebba2b3a64ca1a60f8d749a679))
* rudder-go-kit v0.13.4 ([#3365](https://github.com/rudderlabs/rudder-server/issues/3365)) ([dfb8745](https://github.com/rudderlabs/rudder-server/commit/dfb8745166d5ecfb54a89398bfc15e5affe4ff8f))
* upgraded chi v1 middleware to v5 ([#3353](https://github.com/rudderlabs/rudder-server/issues/3353)) ([a1b37d1](https://github.com/rudderlabs/rudder-server/commit/a1b37d1e839cb0a6cb42cc6efd93c6c0f0c008ab))
* warehouse integration test improvements ([#3264](https://github.com/rudderlabs/rudder-server/issues/3264)) ([4f8f54d](https://github.com/rudderlabs/rudder-server/commit/4f8f54d16ccd4395b3b13b834f509aaabef032c9))

## [1.8.5](https://github.com/rudderlabs/rudder-server/compare/v1.8.4...v1.8.5) (2023-05-17)


Expand Down
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ require (
github.com/prometheus/client_model v0.4.0
github.com/rs/cors v1.9.0
github.com/rudderlabs/analytics-go v3.3.3+incompatible
github.com/rudderlabs/rudder-go-kit v0.13.3
github.com/rudderlabs/rudder-go-kit v0.13.5
github.com/rudderlabs/sql-tunnels v0.1.3
github.com/samber/lo v1.38.1
github.com/segmentio/kafka-go v0.4.40
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1841,8 +1841,8 @@ github.com/rudderlabs/analytics-go v3.3.3+incompatible h1:OG0XlKoXfr539e2t1dXtTB
github.com/rudderlabs/analytics-go v3.3.3+incompatible/go.mod h1:LF8/ty9kUX4PTY3l5c97K3nZZaX5Hwsvt+NBaRL/f30=
github.com/rudderlabs/compose-test v0.1.1 h1:YJn30Fg0+pk9abKTBbWssiofwPuOEfe7Nb2UxKkC+FA=
github.com/rudderlabs/compose-test v0.1.1/go.mod h1:z2dUBgcXaOhhMUcG09lZpqdz5S8bYOIX2wAx4itEr1o=
github.com/rudderlabs/rudder-go-kit v0.13.3 h1:Tl1yN7TZXjVXYe7TBkMYV2yv6tfdOZpDvRxUrH74C/0=
github.com/rudderlabs/rudder-go-kit v0.13.3/go.mod h1:3P7g4yt8TxiN6zSGvl2h/9dh1ae6GQi5zRP02rva+28=
github.com/rudderlabs/rudder-go-kit v0.13.5 h1:Zyfu4b1mfxenHKWsMPyksVQmOBSn+K1O+bgwydsvnM0=
github.com/rudderlabs/rudder-go-kit v0.13.5/go.mod h1:3P7g4yt8TxiN6zSGvl2h/9dh1ae6GQi5zRP02rva+28=
github.com/rudderlabs/sql-tunnels v0.1.3 h1:o7/MX4Yj0WpAaw0uxkRmkagtzedGxUPRwyho4SMbWMQ=
github.com/rudderlabs/sql-tunnels v0.1.3/go.mod h1:1TolUkSsrQxdXS0iyGlbLADsgkebmPcz1MxU5xBl6dE=
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
Expand Down
31 changes: 17 additions & 14 deletions integration_test/kafka_batching/kafka_batching_test.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -294,6 +294,16 @@ func TestKafkaBatching(t *testing.T) {
metrics, err := testhelper.ParsePrometheusMetrics(bytes.NewBuffer(buf))
require.NoError(t, err)

expectedDefaultAttrs := []*promClient.LabelPair{
{Name: ptr("job"), Value: ptr(app.EMBEDDED)},
{Name: ptr("service_name"), Value: ptr(app.EMBEDDED)},
{Name: ptr("service_version"), Value: ptr("Not an official release. Get the latest release from the github repo.")},
{Name: ptr("instanceName"), Value: &serverInstanceID},
{Name: ptr("telemetry_sdk_language"), Value: ptr("go")},
{Name: ptr("telemetry_sdk_name"), Value: ptr("opentelemetry")},
{Name: ptr("telemetry_sdk_version"), Value: ptr("1.14.0")},
}

requireHistogramEqual(t, metrics["router_kafka_batch_size"], histogram{
name: "router_kafka_batch_size", count: 1, sum: 10,
buckets: []*promClient.Bucket{
Expand All @@ -320,31 +330,24 @@ func TestKafkaBatching(t *testing.T) {
{CumulativeCount: ptr(uint64(1)), UpperBound: ptr(1209600.0)},
{CumulativeCount: ptr(uint64(1)), UpperBound: ptr(math.Inf(1))},
},
labels: []*promClient.LabelPair{
{Name: ptr("job"), Value: ptr(app.EMBEDDED)},
{Name: ptr("instance"), Value: &serverInstanceID},
},
labels: expectedDefaultAttrs,
})

require.EqualValues(t, ptr("router_batch_num_input_jobs"), metrics["router_batch_num_input_jobs"].Name)
require.EqualValues(t, ptr(promClient.MetricType_COUNTER), metrics["router_batch_num_input_jobs"].Type)
require.Len(t, metrics["router_batch_num_input_jobs"].Metric, 1)
require.EqualValues(t, &promClient.Counter{Value: ptr(10.0)}, metrics["router_batch_num_input_jobs"].Metric[0].Counter)
require.ElementsMatch(t, []*promClient.LabelPair{
{Name: ptr("destType"), Value: ptr("KAFKA")},
{Name: ptr("job"), Value: ptr(app.EMBEDDED)},
{Name: ptr("instance"), Value: &serverInstanceID},
}, metrics["router_batch_num_input_jobs"].Metric[0].Label)
require.ElementsMatch(t, append(expectedDefaultAttrs,
&promClient.LabelPair{Name: ptr("destType"), Value: ptr("KAFKA")},
), metrics["router_batch_num_input_jobs"].Metric[0].Label)

require.EqualValues(t, ptr("router_batch_num_output_jobs"), metrics["router_batch_num_output_jobs"].Name)
require.EqualValues(t, ptr(promClient.MetricType_COUNTER), metrics["router_batch_num_output_jobs"].Type)
require.Len(t, metrics["router_batch_num_output_jobs"].Metric, 1)
require.EqualValues(t, &promClient.Counter{Value: ptr(1.0)}, metrics["router_batch_num_output_jobs"].Metric[0].Counter)
require.ElementsMatch(t, []*promClient.LabelPair{
{Name: ptr("destType"), Value: ptr("KAFKA")},
{Name: ptr("job"), Value: ptr(app.EMBEDDED)},
{Name: ptr("instance"), Value: &serverInstanceID},
}, metrics["router_batch_num_output_jobs"].Metric[0].Label)
require.ElementsMatch(t, append(expectedDefaultAttrs,
&promClient.LabelPair{Name: ptr("destType"), Value: ptr("KAFKA")},
), metrics["router_batch_num_output_jobs"].Metric[0].Label)
}

func requireHistogramEqual(t *testing.T, mf *promClient.MetricFamily, h histogram) {
Expand Down
Loading

0 comments on commit 25a24d8

Please sign in to comment.