Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dist-txns] IntentAwareIterator is missing intents even during a forward scan #444

Closed
ttyusupov opened this issue Aug 14, 2018 · 0 comments
Closed

[dist-txns] IntentAwareIterator is missing intents even during a forward scan #444

ttyusupov opened this issue Aug 14, 2018 · 0 comments

Comments

@ttyusupov
Copy link
Contributor

IntentAwareIterator can miss intents due to IntentAwareIterator::SetIntentUpperbound optimization. This can happen in case when we have a mix of regular records and intents and reading at the time before regular record was written, but after intent transaction was committed, for example:

I0813 05:28:34.821775 2378527616 doc_operation-test.cc:652] Dump:
SubDocKey(DocKey(0x0000, [-884876226], [-44045842]), [SystemColumnId(0); HT{ physical: 1244 }]) -> null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [-44045842]), [ColumnId(2); HT{ physical: 1244 w: 1 }]) -> 69919023; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [374338310]), [SystemColumnId(0); HT{ physical: 1077 }]) -> null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [374338310]), [ColumnId(2); HT{ physical: 1077 w: 1 }]) -> -758946760; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [635390489]), [SystemColumnId(0); HT{ physical: 1260 }]) -> null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [635390489]), [ColumnId(2); HT{ physical: 1260 w: 1 }]) -> 84751635; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1044445579]), [SystemColumnId(0); HT{ physical: 738 }]) -> null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1044445579]), [ColumnId(2); HT{ physical: 738 w: 1 }]) -> -265099079; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), []) kWeakSnapshotWrite HT{ physical: 1054 w: 2 } -> TransactionId(a7e24881-5ef6-4f62-aa62-b5acd8ef9066) none
SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1054 } -> TransactionId(a7e24881-5ef6-4f62-aa62-b5acd8ef9066) WriteId(4) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1054 w: 1 } -> TransactionId(a7e24881-5ef6-4f62-aa62-b5acd8ef9066) WriteId(5) 2141591066; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), []) kWeakSnapshotWrite HT{ physical: 1252 w: 2 } -> TransactionId(f21367c2-4be2-485d-b748-6e5661ef2399) none
SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1252 } -> TransactionId(f21367c2-4be2-485d-b748-6e5661ef2399) WriteId(10) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1252 w: 1 } -> TransactionId(f21367c2-4be2-485d-b748-6e5661ef2399) WriteId(11) 2128040225; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [541857129]), []) kWeakSnapshotWrite HT{ physical: 651 w: 2 } -> TransactionId(8042fe72-1d47-453e-a740-dff009291f9e) none
SubDocKey(DocKey(0x0000, [-884876226], [541857129]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 651 } -> TransactionId(8042fe72-1d47-453e-a740-dff009291f9e) WriteId(6) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [541857129]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 651 w: 1 } -> TransactionId(8042fe72-1d47-453e-a740-dff009291f9e) WriteId(7) -1663296584; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [588293043]), []) kWeakSnapshotWrite HT{ physical: 579 w: 2 } -> TransactionId(07bbdef9-7009-4303-b87b-8364090c758e) none
SubDocKey(DocKey(0x0000, [-884876226], [588293043]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 579 } -> TransactionId(07bbdef9-7009-4303-b87b-8364090c758e) WriteId(8) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [588293043]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 579 w: 1 } -> TransactionId(07bbdef9-7009-4303-b87b-8364090c758e) WriteId(9) 1291699207; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), []) kWeakSnapshotWrite HT{ physical: 958 w: 2 } -> TransactionId(1444e225-b536-424a-af0d-a721b8b2ebb0) none
SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 958 } -> TransactionId(1444e225-b536-424a-af0d-a721b8b2ebb0) WriteId(2) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 958 w: 1 } -> TransactionId(1444e225-b536-424a-af0d-a721b8b2ebb0) WriteId(3) 1577165154; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), []) kWeakSnapshotWrite HT{ physical: 1168 w: 2 } -> TransactionId(1c0b5430-1105-4d8a-812e-d4952f9ce1a9) none
SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1168 } -> TransactionId(1c0b5430-1105-4d8a-812e-d4952f9ce1a9) WriteId(0) null; ttl: 1.000s
SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1168 w: 1 } -> TransactionId(1c0b5430-1105-4d8a-812e-d4952f9ce1a9) WriteId(1) -1437677451; ttl: 1.000s
TXN REV 07bbdef9-7009-4303-b87b-8364090c758e HT{ physical: 579 } -> SubDocKey(DocKey(0x0000, [-884876226], [588293043]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 579 }
TXN REV 07bbdef9-7009-4303-b87b-8364090c758e HT{ physical: 579 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [588293043]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 579 w: 1 }
TXN REV 07bbdef9-7009-4303-b87b-8364090c758e HT{ physical: 579 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [588293043]), []) kWeakSnapshotWrite HT{ physical: 579 w: 2 }
TXN REV 1444e225-b536-424a-af0d-a721b8b2ebb0 HT{ physical: 958 } -> SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 958 }
TXN REV 1444e225-b536-424a-af0d-a721b8b2ebb0 HT{ physical: 958 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 958 w: 1 }
TXN REV 1444e225-b536-424a-af0d-a721b8b2ebb0 HT{ physical: 958 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [1710663921]), []) kWeakSnapshotWrite HT{ physical: 958 w: 2 }
TXN REV 1c0b5430-1105-4d8a-812e-d4952f9ce1a9 HT{ physical: 1168 } -> SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1168 }
TXN REV 1c0b5430-1105-4d8a-812e-d4952f9ce1a9 HT{ physical: 1168 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1168 w: 1 }
TXN REV 1c0b5430-1105-4d8a-812e-d4952f9ce1a9 HT{ physical: 1168 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [1721694025]), []) kWeakSnapshotWrite HT{ physical: 1168 w: 2 }
TXN REV 8042fe72-1d47-453e-a740-dff009291f9e HT{ physical: 651 } -> SubDocKey(DocKey(0x0000, [-884876226], [541857129]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 651 }
TXN REV 8042fe72-1d47-453e-a740-dff009291f9e HT{ physical: 651 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [541857129]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 651 w: 1 }
TXN REV 8042fe72-1d47-453e-a740-dff009291f9e HT{ physical: 651 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [541857129]), []) kWeakSnapshotWrite HT{ physical: 651 w: 2 }
TXN REV a7e24881-5ef6-4f62-aa62-b5acd8ef9066 HT{ physical: 1054 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1054 }
TXN REV a7e24881-5ef6-4f62-aa62-b5acd8ef9066 HT{ physical: 1054 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1054 w: 1 }
TXN REV a7e24881-5ef6-4f62-aa62-b5acd8ef9066 HT{ physical: 1054 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1762085585]), []) kWeakSnapshotWrite HT{ physical: 1054 w: 2 }
TXN REV f21367c2-4be2-485d-b748-6e5661ef2399 HT{ physical: 1252 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), [SystemColumnId(0)]) kStrongSnapshotWrite HT{ physical: 1252 }
TXN REV f21367c2-4be2-485d-b748-6e5661ef2399 HT{ physical: 1252 w: 1 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), [ColumnId(2)]) kStrongSnapshotWrite HT{ physical: 1252 w: 1 }
TXN REV f21367c2-4be2-485d-b748-6e5661ef2399 HT{ physical: 1252 w: 2 } -> SubDocKey(DocKey(0x0000, [-884876226], [-1438805598]), []) kWeakSnapshotWrite HT{ physical: 1252 w: 2 }
I0813 05:28:34.824223 2378527616 doc_operation-test.cc:670] Testing: QL_OP_LESS_THAN_EQUAL
I0813 05:28:34.825400 2378527616 doc_operation-test.cc:680] Bound: { data: { k: -884876226 r: 1044445579 v: -265099079 } ht: { physical: 738 } }
I0813 05:28:34.825475 2378527616 doc_operation-test.cc:691] Read time: { read: { physical: 737 } local_limit: { physical: 737 } global_limit: { physical: 737 } serial_no: 0}
I0813 05:28:34.827278 2378527616 doc_operation-test.cc:718] Expected rows: [{ data: { k: -884876226 r: 541857129 v: -1663296584 } ht: { physical: 651 } }, { data: { k: -884876226 r: 588293043 v: 1291699207 } ht: { physical: 579 } }]
../../src/yb/docdb/doc_operation-test.cc:733: Failure
      Expected: expected_rows.end()
      Which is: 8-byte object <40-B7 2E-0F 01-00 00-00>
To be equal to: it
      Which is: 8-byte object <10-B7 2E-0F 01-00 00-00>

The reason is the due to incorrect setting of intent upper bound based regular record with ht>read_time, iterator skips intents which are later than this record, but were committed before read_time.
yugabyte-ci pushed a commit that referenced this issue Aug 18, 2018
@ttyusupov
#444: ENG-3782: IntentAwareIterator is missing intents even during a …
5a460b8 
…forward scan

Summary:
IntentAwareIterator can miss intents in case when we have a mix of regular records and
intents and reading at the time before regular record was written, but after intent transaction was
committed. The reason is the due to incorrect setting of intent upper bound based regular record with ht>read_time, iterator skips intents which are later than this record, but were committed before read_time.
The fix is to do lazy setting of upper bound for intents and lazy scanning for intents inside
valid() function. Also added tests to catch issue which was fixed.

Test Plan: Jenkins

Reviewers: mikhail, robert, sergei

Reviewed By: sergei

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D5344
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ttyusupov