Migration to new state structure #1703
Conversation
ruflin
added
in progress
ruflin
added
review
and removed
in progress
| spoolerChan: spoolerChan, | ||
| harvesterChan: make(chan *input.FileEvent), | ||
| done: make(chan struct{}), | ||
| states: input.NewStates(), | ||
| states: &states, |
There was a problem hiding this comment.
the states parameter given to NewProspector is a copy of original one, but here we allocate the copied state onto heap. You want input.States to be shared or to be a copy in here?
There was a problem hiding this comment.
It should be a copy as from here on, each prospector treats the state independent.
There was a problem hiding this comment.
states is defined like this:
// States handles list of FileState
type States struct {
states []FileState
mutex sync.Mutex
}
The copy of 'states' is mostly shallow. Some code checkers (newer go vet should) will complain about copy on mutex. The underlying byte-buffer is shared when slice is copied. The (*States).Update method can override a file its state, if file is member of slice. Due to mutex being copied, any 2 go-routines attempting to updates a file state for the same file might actually race.
| @@ -55,6 +55,7 @@ https://github.com/elastic/beats/compare/v5.0.0-alpha2...master[Check the HEAD d | |||
| *Topbeat* | |||
|
|
|||
| *Filebeat* | |||
| - The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. {pull}1703[1703] | |||
There was a problem hiding this comment.
+++ for doing the migration automatically!
The previous state list was dependent on a file path. This had the potential to lead to overwrite and conflicts on file rotation. As the file path is also stored inside the state object this information was duplicated. Now a pure array is stored in the registry which makes the format more flexibel and brings it close to the format used by Logstash. Changes: * Not storing an index with paths as this leaded to duplicates * Introduction of last_seen to differentiate between entries with same bath which show up multiple times. This introduces also the base to clean up the registry file. Currently it is only used in the prospector, no registry yet. * Registry can now contain multiple entries for a path * Refactor registrar to use the same state array as prospector does * Introduce migration check to migrate from old to new state. Make it backward compatible by checking for old structure on startup * Decouple registrar from prospector * Update tests to follow new model
|
LGTM |
The previous state list was dependent on a file path. This had the potential to lead to overwrite and conflicts on file rotation. As the file path is also stored inside the state object this information was duplicated. Now a pure array is stored in the registry which makes the format more flexibel and brings it close to the format used by Logstash.
Changes: