Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DPDV-6415: Remove SIGSEV #100

Merged
merged 37 commits into from
Jun 28, 2024
Merged

DPDV-6415: Remove SIGSEV #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
8cd508a
DPDV-6415: Remove SIGSEV
martin-majlis-s1 Jun 24, 2024
bd81cc3
Improve Finish go routine
martin-majlis-s1 Jun 25, 2024
538d688
Fix few more tests
martin-majlis-s1 Jun 25, 2024
78bc383
Add more logging related to locks in Memo
martin-majlis-s1 Jun 25, 2024
acdc0e0
add_events_long_running_test are succeeding!!!
martin-majlis-s1 Jun 25, 2024
6b5a13f
All tests are passing now
martin-majlis-s1 Jun 25, 2024
7d931fc
Add timeout restrictions so that it doesn't run for 6 hours
martin-majlis-s1 Jun 26, 2024
eb45132
Enable skipped buffer tests
martin-majlis-s1 Jun 26, 2024
1496687
Reset tickers when event is received
martin-majlis-s1 Jun 26, 2024
0086ae7
Fix test
martin-majlis-s1 Jun 26, 2024
03026b6
Add logging to the pub method
martin-majlis-s1 Jun 26, 2024
b777d21
Add even more logging
martin-majlis-s1 Jun 26, 2024
155cc3e
Add even more logging
martin-majlis-s1 Jun 26, 2024
65437ac
Build and run commands with -race flag
martin-majlis-s1 Jun 26, 2024
bec553a
Also use channel for publishing messages
martin-majlis-s1 Jun 26, 2024
e0413f8
Remove pubsub library
martin-majlis-s1 Jun 26, 2024
6faa910
Increase time interval between purges
martin-majlis-s1 Jun 26, 2024
ffd4ff2
Move SessionManager to separate module
martin-majlis-s1 Jun 26, 2024
7c1999a
Update buffer_config tests
martin-majlis-s1 Jun 26, 2024
275b52d
Add tests related to the buffer
martin-majlis-s1 Jun 26, 2024
a9f9128
Add more buffer tests
martin-majlis-s1 Jun 26, 2024
e08c832
Make number of parallel buffer sendings configurable.
martin-majlis-s1 Jun 26, 2024
21118c6
Simplify SessionManager
martin-majlis-s1 Jun 27, 2024
c126cba
Add tests for SessionManager
martin-majlis-s1 Jun 27, 2024
cd35f4a
Bump version
martin-majlis-s1 Jun 27, 2024
5699fd5
Fix session_manager tests
martin-majlis-s1 Jun 27, 2024
2e160b7
Add sleep to make sure that everything is processed
martin-majlis-s1 Jun 27, 2024
ae82c19
Reduce logging in the client
martin-majlis-s1 Jun 27, 2024
d4561f4
Add more tests for statistics
martin-majlis-s1 Jun 27, 2024
c8551e3
Remove unused variable publishAsap
martin-majlis-s1 Jun 27, 2024
607624b
Make number of outgoing connections configurable
martin-majlis-s1 Jun 27, 2024
33175e2
Shutdown sequence finishes on time
martin-majlis-s1 Jun 27, 2024
4b8c932
Incorporates comments from CR
martin-majlis-s1 Jun 27, 2024
5c1361c
Improve some function names
martin-majlis-s1 Jun 27, 2024
1bf201f
Incorporate comments from CR
martin-majlis-s1 Jun 28, 2024
a1c5365
Rename private functions in the session manager
martin-majlis-s1 Jun 28, 2024
83030c4
To be safe stop tickers
martin-majlis-s1 Jun 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/code-quality.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ jobs:
run: |
pre-commit run -a
- name: Code coverage
timeout-minutes: 15
run: |
make coverage
- name: Upload coverage reports to Codecov
Expand All @@ -77,5 +78,6 @@ jobs:
run: |
make build-examples
- name: Check SSL Certificates
timeout-minutes: 10
run: |
make test-ssl-certificates
1 change: 1 addition & 0 deletions .github/workflows/stress-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ jobs:
go: ['1.21', '1.22']
os: [ubuntu-latest, windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 15
needs: pre_job
if: ${{ needs.pre_job.outputs.should_skip != 'true' || github.ref_name == 'main' }}

Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/unit-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,11 @@ jobs:
run: |
go mod vendor
- name: Code tests
timeout-minutes: 15
run: |
make test
- name: Run Tests Many Times
timeout-minutes: 30
# Run test multiple times to check for flaky tests
if: github.ref_name == 'main'
run: |
Expand Down
4 changes: 2 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ endif

GO_BUILD_TAGS=""
GOTEST_OPT?= -v -race -timeout 300s -parallel 4 -count=1 --tags=$(GO_BUILD_TAGS)
GOTEST_LONG_RUNNING_OPT?= -v -race -timeout 1200s -parallel 4 -count=1 -tags=long_running,$(GO_BUILD_TAGS)
GOTEST_LONG_RUNNING_OPT?= -v -race -timeout 600s -parallel 4 -count=1 -tags=long_running,$(GO_BUILD_TAGS)
GOTEST_OPT_WITH_COVERAGE = $(GOTEST_OPT) -coverprofile=coverage.txt -covermode=atomic
GOTEST_OPT_WITH_COVERAGE_LONG_RUNNING=$(GOTEST_LONG_RUNNING_OPT) -coverprofile=coverage.txt -covermode=atomic
GOCMD?= go
Expand Down Expand Up @@ -93,7 +93,7 @@ coverage-all:
build-examples:
for d in examples/*; do \
echo "Build example: $${d}"; \
(cd $${d}; go mod tidy && go build) || exit 1; \
(cd $${d}; go mod tidy && go build -race) || exit 1; \
done;

.PHONY: test-ssl-certificates
Expand Down
5 changes: 5 additions & 0 deletions RELEASE_NOTES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
# Release Notes

## 0.20.0

* fix: fix SIGSEV that was caused by releasing resources associated with inactive sessions
* feat: make number of parallel outgoing requests configurable

## 0.19.0

* fix: retry also when server is not returning JSON
Expand Down
1 change: 0 additions & 1 deletion examples/client/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ require (

require (
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cskr/pubsub v1.0.2 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/uuid v1.6.0 // indirect
Expand Down
2 changes: 0 additions & 2 deletions examples/client/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cskr/pubsub v1.0.2 h1:vlOzMhl6PFn60gRlTQQsIfVwaPB/B/8MziK8FhEPt/0=
github.com/cskr/pubsub v1.0.2/go.mod h1:/8MzYXk/NJAz782G8RPkFzXTZVu63VotefPnR9TIRis=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
Expand Down
1 change: 0 additions & 1 deletion examples/readme/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ require (

require (
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cskr/pubsub v1.0.2 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/google/uuid v1.6.0 // indirect
Expand Down
2 changes: 0 additions & 2 deletions examples/readme/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cskr/pubsub v1.0.2 h1:vlOzMhl6PFn60gRlTQQsIfVwaPB/B/8MziK8FhEPt/0=
github.com/cskr/pubsub v1.0.2/go.mod h1:/8MzYXk/NJAz782G8RPkFzXTZVu63VotefPnR9TIRis=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
Expand Down
1 change: 0 additions & 1 deletion examples/stress/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ require (

require (
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cskr/pubsub v1.0.2 // indirect
github.com/google/uuid v1.6.0 // indirect
go.opentelemetry.io/otel v1.27.0 // indirect
go.opentelemetry.io/otel/metric v1.27.0 // indirect
Expand Down
2 changes: 0 additions & 2 deletions examples/stress/go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
github.com/cenkalti/backoff/v4 v4.3.0/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
github.com/cskr/pubsub v1.0.2 h1:vlOzMhl6PFn60gRlTQQsIfVwaPB/B/8MziK8FhEPt/0=
github.com/cskr/pubsub v1.0.2/go.mod h1:/8MzYXk/NJAz782G8RPkFzXTZVu63VotefPnR9TIRis=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-logr/logr v1.4.1 h1:pKouT5E8xu9zeFC39JXRDukb6JFQPXM5p5I91188VAQ=
Expand Down
18 changes: 14 additions & 4 deletions examples/stress/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,13 +40,14 @@ import (
)

const (
MaxLifeTimeMultiplier = 5
PurgeOlderThanMultiplier = 15
MaxLifeTimeMultiplier = 50
PurgeOlderThanMultiplier = 150
)

func main() {
eventsCount := flag.Int("events", 1e5, "number of events")
bucketsCount := flag.Int("buckets", 1e5, "number of buckets")
parallel := flag.Int("parallel", 20, "number of parallel outgoing connections")
sleep := flag.Duration("sleep", 10*time.Millisecond, "sleep between sending two events")
logFile := flag.String("log", fmt.Sprintf("log-%s-%d.log", version.Version, time.Now().UnixMilli()), "log file for stats")
logEvery := flag.Duration("log-every", time.Second, "how often log statistics")
Expand All @@ -60,6 +61,7 @@ func main() {
logger.Info("Running stress test - input:",
zap.Int("events", *eventsCount),
zap.Int("buckets", *bucketsCount),
zap.Int("parallel", *parallel),
zap.Duration("sleep", *sleep),
zap.String("log", *logFile),
zap.Duration("log-every", *logEvery),
Expand All @@ -74,6 +76,7 @@ func main() {
logger.Info("Running stress test - adjusted:",
zap.Int("events", *eventsCount),
zap.Int("buckets", *bucketsCount),
zap.Int("parallel", *parallel),
zap.Duration("sleep", *sleep),
zap.String("log", *logFile),
zap.Duration("log-every", *logEvery),
Expand Down Expand Up @@ -109,6 +112,7 @@ func main() {
buffer_config.WithGroupBy([]string{"body.str"}),
buffer_config.WithMaxLifetime(MaxLifeTimeMultiplier**sleep),
buffer_config.WithPurgeOlderThan(PurgeOlderThanMultiplier**sleep),
buffer_config.WithMaxParallelOutgoing(*parallel),
)
check(err)
cfgUpdated, err := cfg.WithOptions(
Expand Down Expand Up @@ -167,8 +171,14 @@ func main() {
eventBundle := &add_events.EventBundle{Event: event, Thread: thread, Log: log}

batch = append(batch, eventBundle)
err := dataSetClient.AddEvents(batch)
check(err)
go func(batch []*add_events.EventBundle) {
err := dataSetClient.AddEvents(batch)
check(err)
}(batch)
Comment on lines 173 to +177
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it an intention to call dataSetClient.AddEvents with bigger and bigger batch?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the top of the loop there is always new batch created:

	for i := 0; i < *eventsCount; i++ {
		batch := make([]*add_events.EventBundle, 0)

Furthermore, we can see in the logs, that if you ask for 1k events, there is 1k events processed - not 500k.


if i%*bucketsCount == 0 {
time.Sleep(PurgeOlderThanMultiplier * *sleep)
}
time.Sleep(*sleep)
}

Expand Down
Loading
Loading