[SPARK-26379][SS][BRANCH-2.3] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp #23656

HeartSaVioR · 2019-01-25T23:40:52Z

What changes were proposed in this pull request?

Spark replaces CurrentTimestamp with CurrentBatchTimestamp.
However, CurrentBatchTimestamp is TimeZoneAwareExpression while CurrentTimestamp isn't.
Without TimeZoneId, CurrentBatchTimestamp becomes unresolved and raises UnresolvedException.

Since CurrentDate is TimeZoneAwareExpression, there is no problem with CurrentDate.

How was this patch tested?

Pass the Jenkins with the updated test cases.

…to streaming query ## What changes were proposed in this pull request? This patch proposes to fix issue on adding `current_timestamp` / `current_date` with streaming query. The root reason is that Spark transforms `CurrentTimestamp`/`CurrentDate` to `CurrentBatchTimestamp` in MicroBatchExecution which makes transformed attributes not-yet-resolved. They will be resolved by IncrementalExecution. (In ContinuousExecution, Spark doesn't allow using `current_timestamp` and `current_date` so it has been OK.) It's OK for DataSource V1 sink because it simply leverages transformed logical plan and don't evaluate until they're resolved, but for DataSource V2 sink, Spark tries to extract the schema of transformed logical plan in prior to IncrementalExecution, and unresolved attributes will raise errors. This patch fixes the issue via having separate pre-resolved logical plan to pass the schema to StreamingWriteSupport safely. ## How was this patch tested? Added UT. Closes apache#23609 from HeartSaVioR/SPARK-26379. Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

dongjoon-hyun · 2019-01-25T23:57:12Z

Thank you, @HeartSaVioR . :)

SparkQA · 2019-01-26T02:01:00Z

Test build #101696 has finished for PR 23656 at commit cec965c.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-26T02:19:52Z

We will revisit this PR because the original PR needs a more verification and a followup PR is expected.

[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query #23609

HeartSaVioR · 2019-01-26T13:00:44Z

FYI: follow-up PR is available in #23660 submitted by Dongjoon.

dongjoon-hyun · 2019-01-27T18:06:02Z

Hi, @HeartSaVioR . Could you update this PR with the final content?

…xception in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. This PR reverts the [previous patch](apache#23609) on `MicroBatchExecution` and fixes the root cause. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes apache#23660 from dongjoon-hyun/SPARK-26379. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

HeartSaVioR · 2019-01-27T22:13:35Z

@dongjoon-hyun Yes, I've just cherry-picked the FOLLOW-UP commit into PR branch.

dongjoon-hyun · 2019-01-27T22:35:00Z

Thank you, @HeartSaVioR . It's enough since this will be merged as a single commit.

dongjoon-hyun · 2019-01-27T22:36:16Z

@HeartSaVioR . Could you update the PR title and description, too?

SparkQA · 2019-01-28T00:46:02Z

Test build #101734 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-01-28T01:46:04Z

@dongjoon-hyun Sorry for the late. Just updated the title and content of PR.

HeartSaVioR · 2019-01-28T01:46:33Z

Test failure looks not relevant. Retest this, please.

maropu · 2019-01-28T01:47:26Z

yea, that's known flaky tests in branch-2.3..

SparkQA · 2019-01-28T04:05:27Z

Test build #101740 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2019-01-28T04:06:50Z

retest this please

dongjoon-hyun · 2019-01-28T04:10:12Z

Thank you, @HeartSaVioR and @maropu .

dongjoon-hyun

+1, LGTM (pending Jenkins).

SparkQA · 2019-01-28T06:27:00Z

Test build #101743 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-28T06:31:35Z

Retest this please.

SparkQA · 2019-01-28T08:05:02Z

Test build #101747 has finished for PR 23656 at commit 52ac627.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-01-28T08:11:19Z

retest this, please

SparkQA · 2019-01-28T10:44:57Z

Test build #101749 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-01-28T12:31:38Z

Retest this please

maropu · 2019-01-28T12:58:30Z

super flaky....

HeartSaVioR · 2019-01-28T14:48:02Z

Yeah... If the branch should be maintained further, it may need to fix the flaky tests... As it seems to be known flakiness, are we aware about the fix in upper version lines?

SparkQA · 2019-01-28T14:54:36Z

Test build #101758 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-01-28T16:12:20Z

Retest this please.

dongjoon-hyun · 2019-01-28T16:19:48Z

Retest this please.

dongjoon-hyun · 2019-01-28T16:26:35Z

Retest this please.

dongjoon-hyun · 2019-01-28T16:27:07Z

@HeartSaVioR and @maropu . I triggered Jenkins three times intentionally to pass the Jenkins.

SparkQA · 2019-01-28T18:39:13Z

Test build #101765 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-28T18:44:23Z

Test build #101764 has finished for PR 23656 at commit 52ac627.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-28T19:27:42Z

Test build #101763 has finished for PR 23656 at commit 52ac627.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HeartSaVioR · 2019-01-28T20:03:27Z

2 failures, 1 passes. It looks like kind of serious one if Spark community will maintain branch-2.3 forward.

dongjoon-hyun · 2019-01-28T20:12:59Z

@HeartSaVioR . Yes. It's true and branch-2.3 will be EOF in several months.

dongjoon-hyun · 2019-01-28T20:13:08Z

Merged to branch-2.3.

…dException in CurrentBatchTimestamp ## What changes were proposed in this pull request? Spark replaces `CurrentTimestamp` with `CurrentBatchTimestamp`. However, `CurrentBatchTimestamp` is `TimeZoneAwareExpression` while `CurrentTimestamp` isn't. Without TimeZoneId, `CurrentBatchTimestamp` becomes unresolved and raises `UnresolvedException`. Since `CurrentDate` is `TimeZoneAwareExpression`, there is no problem with `CurrentDate`. ## How was this patch tested? Pass the Jenkins with the updated test cases. Closes #23656 from HeartSaVioR/SPARK-26379-branch-2.3. Lead-authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com> Co-authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

maropu · 2019-01-29T00:59:23Z

@HeartSaVioR I'm not familiar with the SS part though, your other pr (#23634) is not related to the failures? It seems the failures are of the outer joins tests...

HeartSaVioR · 2019-01-29T01:03:00Z

If #23634 was the fix for tests they would be flaky for branch-2.4 / master as well. They don't seem to be flaky in master branch.

maropu · 2019-01-29T01:04:44Z

Aha, I see.

HeartSaVioR changed the title ~~[SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query~~ [BRANCH-2.3][SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query Jan 25, 2019

dongjoon-hyun changed the title ~~[BRANCH-2.3][SPARK-26379][SS] Fix issue on adding current_timestamp/current_date to streaming query~~ [SPARK-26379][SS][BACKPORT-2.3] Fix issue on adding current_timestamp/current_date to streaming query Jan 25, 2019

HeartSaVioR changed the title ~~[SPARK-26379][SS][BACKPORT-2.3] Fix issue on adding current_timestamp/current_date to streaming query~~ [SPARK-26379][SS][BRANCH-2.3] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp Jan 28, 2019

dongjoon-hyun approved these changes Jan 28, 2019

View reviewed changes

dongjoon-hyun closed this Jan 28, 2019

HeartSaVioR deleted the SPARK-26379-branch-2.3 branch January 28, 2019 20:29

[SPARK-26379][SS][BRANCH-2.3] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp #23656

[SPARK-26379][SS][BRANCH-2.3] Use dummy TimeZoneId to avoid UnresolvedException in CurrentBatchTimestamp #23656

Conversation

HeartSaVioR commented Jan 25, 2019 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

dongjoon-hyun commented Jan 25, 2019

SparkQA commented Jan 26, 2019

dongjoon-hyun commented Jan 26, 2019

HeartSaVioR commented Jan 26, 2019

dongjoon-hyun commented Jan 27, 2019

HeartSaVioR commented Jan 27, 2019

dongjoon-hyun commented Jan 27, 2019

dongjoon-hyun commented Jan 27, 2019

SparkQA commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019

maropu commented Jan 28, 2019

SparkQA commented Jan 28, 2019

maropu commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

dongjoon-hyun left a comment

Choose a reason for hiding this comment

SparkQA commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

SparkQA commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019

SparkQA commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019

maropu commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019

SparkQA commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

SparkQA commented Jan 28, 2019

SparkQA commented Jan 28, 2019

SparkQA commented Jan 28, 2019

HeartSaVioR commented Jan 28, 2019 • edited Loading

dongjoon-hyun commented Jan 28, 2019

dongjoon-hyun commented Jan 28, 2019

maropu commented Jan 29, 2019

HeartSaVioR commented Jan 29, 2019

maropu commented Jan 29, 2019

HeartSaVioR commented Jan 25, 2019 •

edited

Loading

HeartSaVioR commented Jan 28, 2019 •

edited

Loading