Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Queries against files that have just been compacted need to point to new files #5501

Closed
jwilder opened this issue Feb 1, 2016 · 0 comments · Fixed by #7038
Closed

Queries against files that have just been compacted need to point to new files #5501

jwilder opened this issue Feb 1, 2016 · 0 comments · Fixed by #7038
Labels
area/tsm
Milestone
1.0.0

Comments

@jwilder
Copy link
Contributor

jwilder commented Feb 1, 2016

If a query is run and is executing against a TSM file that is compacted and replaced while the query is executing, we need to use the new file.

Original issue from #4977
@jwilder jwilder modified the milestone: 1.0.0 Jul 19, 2016
@jwilder jwilder mentioned this issue Jul 20, 2016
Merged
3 tasks
jwilder added a commit that referenced this issue Jul 20, 2016
@jwilder
Allow queries to complete before closing TSM files
1d0decb 
If a query was running against a file being compacted, we close the file
and the query would end wherever it had read up to.  This could result
in queries that randomly lost data, but running them again showed the
full results.

We now use a reference counting approach and move the in-use files out
of the way in the filestore and allow the queries to complete against
the old tsm files.  The new files are installed and new queries will
use them.

To move the files out of the way, the existing file is hardlink and
remap with the existing TSMReader which is quick as we do not need
to reload the index (which is very slow).  This should also work
cross platform better than just renaming the old file (which breaks
on windows if the file is still open).

Fixes #5501
jwilder added a commit that referenced this issue Jul 20, 2016
@jwilder
Allow queries to complete before closing TSM files
0c9175e 
If a query was running against a file being compacted, we close the file
and the query would end wherever it had read up to.  This could result
in queries that randomly lost data, but running them again showed the
full results.

We now use a reference counting approach and move the in-use files out
of the way in the filestore and allow the queries to complete against
the old tsm files.  The new files are installed and new queries will
use them.

Fixes #5501
jwilder added a commit that referenced this issue Jul 21, 2016
@jwilder
Allow queries to complete before closing TSM files
822f409 
If a query was running against a file being compacted, we close the file
and the query would end wherever it had read up to.  This could result
in queries that randomly lost data, but running them again showed the
full results.

We now use a reference counting approach and move the in-use files out
of the way in the filestore and allow the queries to complete against
the old tsm files.  The new files are installed and new queries will
use them.

Fixes #5501
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/tsm
Projects
None yet
Milestone
1.0.0
Development

Successfully merging a pull request may close this issue.

Fix queries against compacted files influxdata/influxdb
1 participant
@jwilder