Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Time normalize" bag written from API so that splits are in order #842

Open
1 task
emersonknapp opened this issue Aug 9, 2021 · 1 comment
Open
1 task
Labels
enhancement New feature or request

Comments

@emersonknapp
Copy link
Collaborator

Description

When writing a bag from the API (instead of recording), with splitting enabled, it is easily possible to write messages out-of-order in time. This can lead to the timestamp bounds of these API-generated split bags to not have an increasing order.

Pseudocode example:

msg = Message()
bag = Bag()
bag.max_bagfile_size = 1
bag.write(msg, ts=5)
# splits
bag.write(msg, ts=3)
# splits
bag.write(msg, ts=10)
# splits
bag.write(msg, ts=0)

Now there are 4 split bags, in the totally wrong order for meaningful playback. Within a single split, out-of-order writes don't matter because the sqlite implementation sorts by timestamp - but once a split occurs, we start a new context and cannot maintain consistency.

Related Issues

Noticed this as a serious problem for seek behavioral consistency #821

Completion Criteria

  • There is a way for a user to make a bag have monotonic increasing start/end durations. This does not have to be an "online" process, it may be a postprocess step.

Implementation Notes / Suggestions

This could be a special case of the #831 functionality - simultaneously opening all split files within a bag and rewriting them ordered into a new bag.

Testing Notes / Suggestions

@emersonknapp emersonknapp added the enhancement New feature or request label Aug 9, 2021
@emersonknapp emersonknapp changed the title "Normalize" bag written from API "Time normalize" bag written from API so that splits are in order Aug 9, 2021
@emersonknapp
Copy link
Collaborator Author

Related: this could also have a utility to just check for timestamp ordering without doing any modification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant