[SPARK-6326][SQL] Improve castStruct to be faster #5017
Conversation
|
Test build #28575 timed out for PR 5017 at commit |
|
Please post benchmark results for PRs with performance improvements. |
|
Simple benchmark conducting 1000000 the before pr: 59.149s Conducting 1000000 the before pr: 33.975s It is slightly improved. |
| // TODO: Could be faster? | ||
| buildCast[Row](_, row => { | ||
| var i = 0 | ||
| val fields = row.toSeq.map {(v) => |
There was a problem hiding this comment.
Will there be faster if we use while loop instead of the .map? At least we can save the intermediate object creation(fields).
There was a problem hiding this comment.
Improved with new commit. It is faster.
|
With new commit. Conducting 1000000 the before pr: 59.149s Conducting 1000000 the before pr: 33.975s |
| var i = 0 | ||
| while (i < row.length) { | ||
| val v = row(i) | ||
| newRow.update(i, if (v == null) null else casts(i)(v)) |
There was a problem hiding this comment.
Nit: newRow(i) = if (v == null) null else casts(i)(v)
|
Unrelated failure. retest it please. |
|
test this please |
|
Test build #28763 has finished for PR 5017 at commit
|
|
/cc @marmbrus Any other concerns? |
|
Merged to master. Thanks! |
Current
castStructshould be very slow. This pr slightly improves it.