YQL-16402: Implement key cache for Top and TopSort nodes #4761
Conversation
|
⚪
|
|
⚪
|
igormunkin
force-pushed
the
YQL-16402-keeptop-key-cache
branch
from
229df8b to
38dd5da
Compare
|
⚪
|
|
⚪
|
igormunkin
force-pushed
the
YQL-16402-keeptop-key-cache
branch
from
38dd5da to
b08dbd6
Compare
|
⚪
|
|
⚪
|
igormunkin
force-pushed
the
YQL-16402-keeptop-key-cache
branch
from
b08dbd6 to
ceecfe7
Compare
|
⚪
|
|
⚪
|
This patch fixes the issue with the excess calculations of the sorting
keys for TopSort computation node. While compiling TopSort runtime node,
the input source (either stream or flow) is transformed into {key: item}
mapping, and the resulting iterable is processed by both KeepTop and
TopSort nodes using the trivial key extractor, that obtains the key
value as the first component from the item of the resulting iterable.
The result yielded by TopSort is transformed back returning only the
second component from the item of the mapping being processed.
As a result of the changed described above, <keyExtractor> callable is
invoked once for each item of the given input iterable.
A static UDF module is used to test the fix. It provides an echo
function, that increments TLS counter each time being called. When the
processing is finished, the value of this counter is compared with the
number of the items in the Stream/Flow.
This patch fixes the issue with the excess calculations of the sorting
keys for Top computation node. While compiling Top runtime node, the
input source (either stream or flow) is transformed into {key: item}
mapping, and the resulting iterable is processed by both KeepTop and Top
nodes using the trivial key extractor, that obtains the key value as the
first component from the item of the resulting iterable. The result
yielded by Top is transformed back returning only the second component
from the item of the mapping being processed.
As a result of the changed described above, <keyExtractor> callable is
invoked once for each item of the given input iterable.
A static UDF module is used to test the fix. It provides an echo
function, that increments TLS counter each time being called. When the
processing is finished, the value of this counter is compared with the
number of the items in the Stream/Flow.
igormunkin
force-pushed
the
YQL-16402-keeptop-key-cache
branch
from
ceecfe7 to
43b854b
Compare
igormunkin
changed the title
igormunkin
changed the title
|
⚪
|
|
⚪
|
This patchset fixes the issue with the excess calculations of the sorting keys for
Top/TopSortcomputation node. While compilingTop/TopSortruntime node, the input source (either stream or flow) is transformed into{key: item}mapping, and the resulting iterable is processed by bothKeepTopandTop/TopSortnodes using the trivial key extractor, that obtains the key value as the first component from the item of the resulting iterable. The result yielded byTop/TopSortis transformed back returning only the second component from the item of the mapping being processed.As a result of the changed described above,
<keyExtractor>callable is invoked once for each item of the given input iterable.A static UDF module is used to test the fix. It provides an echo function, that increments TLS counter each time being called. When the processing is finished, the value of this counter is compared with the number of the items in the Stream/Flow.
Changelog category