Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LimitReader copies bytes instead of slicing so the underlying big array can be freed #11524

Merged
merged 5 commits into from
Apr 16, 2019
Merged

LimitReader copies bytes instead of slicing so the underlying big array can be freed #11524

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
82579fa
copy bytes instead of slicing so the underlying big array can be freed
kvch Mar 29, 2019
a1d9fa5
make it work
kvch Apr 5, 2019
6921dc2
add changelog entry
kvch Apr 15, 2019
cb6718a
rly fix it
kvch Apr 15, 2019
7fd2824
reword changelog entry
kvch Apr 16, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix goroutine leak happening when harvesters are dynamically stopped. {pull}11263[11263]
- Fix `add_docker_metadata` source matching, using `log.file.path` field now. {pull}11577[11577]
- Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. {pull}11591[11591]
- Reduce memory usage if long lines are truncated to fit `max_bytes` limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. {pull}11524[11524]

*Heartbeat*

Expand Down
9 changes: 8 additions & 1 deletion libbeat/reader/readfile/limit.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
package readfile

import (
"fmt"

"github.com/elastic/beats/libbeat/reader"
)

Expand All @@ -37,7 +39,12 @@ func NewLimitReader(r reader.Reader, maxBytes int) *LimitReader {
func (r *LimitReader) Next() (reader.Message, error) {
message, err := r.reader.Next()
if len(message.Content) > r.maxBytes {
message.Content = message.Content[:r.maxBytes]
tmp := make([]byte, r.maxBytes)
n := copy(tmp, message.Content)
if n != r.maxBytes {
return message, fmt.Errorf("unexpected number of bytes were copied, %d instead of limit %d", n, r.maxBytes)
}
message.Content = tmp
message.AddFlagsWithKey("log.flags", "truncated")
}
return message, err
Expand Down