-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix weird NRT bug #13353 #13369
Fix weird NRT bug #13353 #13369
Conversation
Pinging some folks on review. This is a weird place in the codebase I haven't messed with before. This deserves careful review. Checks are green, but I may have missed some corners. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Your approach makes sense to me, we should use field infos to decide whether to flush, not whether we buffered docs. This is consistent with what happens when all values get merged away, and with what happens when we flush postings, norms or doc values.
The issue outlines the problem. When we have point value dimensions, segment core readers assume that there will be point files. However, when allowing soft deletes and a document fails indexing failed before a point field could be written, this assumption fails. Consequently, the NRT fails to open. I settled on always flushing a point file if the field info says there are point fields, even if there aren't any docs in the buffer. closes #13353
This commit updates the writer to handle the case where there are no values. Previously (before #13369), there was a check that there were some points values before trying to write, this is no longer the case. The code in writeFieldNDims has an assumption that the values is not empty - an empty values will result in calculating a negative number of splits, and a negate array size to hold the splits. The fix is trivial, return null when values is empty - null is an allowable return value from this method. Note: writeField1Dim is able to handle an empty values.
This commit updates the writer to handle the case where there are no values. Previously (before apache#13369), there was a check that there were some points values before trying to write, this is no longer the case. The code in writeFieldNDims has an assumption that the values is not empty - an empty values will result in calculating a negative number of splits, and a negate array size to hold the splits. The fix is trivial, return null when values is empty - null is an allowable return value from this method. Note: writeField1Dim is able to handle an empty values.
The issue outlines the problem. When we have point value dimensions, segment core readers assume that there will be point files.
However, when allowing soft deletes and a document fails indexing failed before a point field could be written, this assumption fails. Consequently, the NRT fails to open.
I tried many different ways of fixing this issue.
So, I settled on always flushing a point file if the field info says there are point fields, even if there aren't any docs in the buffer.
I am happy to consider other options.
closes #13353