[CELEBORN-1489] Update Flink support with authentication support #2596
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
+CC @otterc, @SteNicholas, @RexXiong |
|
+CC @venkata91 |
mridulm
commented
Jul 1, 2024
| LifecycleManager manager = lifecycleManager; | ||
| if (null != manager) { | ||
| manager.stop(); | ||
| } |
There was a problem hiding this comment.
This is an unrelated minor fix for NPE
mridulm
commented
Jul 1, 2024
| @@ -53,7 +56,7 @@ public TransportClient createClientWithRetry(String remoteHost, int remotePort) | |||
| while (retryCount > 0) { | |||
| try { | |||
| return createClient(remoteHost, remotePort); | |||
| } catch (IOException e) { | |||
| } catch (Exception e) { | |||
There was a problem hiding this comment.
SASL failures are not checked exceptions - we have to retry for those too.
mridulm
commented
Jul 1, 2024
| @@ -565,6 +587,15 @@ public void setDataClientFactory(TransportClientFactory dataClientFactory) { | |||
| @Override | |||
| @VisibleForTesting | |||
| public TransportClientFactory getDataClientFactory() { | |||
| initializeTransportClientFactoryIfRequired(); | |||
There was a problem hiding this comment.
getDataClientFactory gets invoked without setupLifecycleManagerRef primarily for tests.
mridulm
commented
Jul 1, 2024
mridulm
commented
Jul 1, 2024
| } catch (IOException e) { | ||
| throw new RuntimeException("Failed to resolve path " + path, e); | ||
| } | ||
| } |
There was a problem hiding this comment.
This is strictly not required for this PR - but keeps failing in ApiBaseResourceAuthenticationSuite (with maven) without it.
mridulm
commented
Jul 1, 2024
| @@ -95,8 +95,10 @@ class WorkerSource(conf: CelebornConf) extends AbstractSource(conf, MetricsSyste | |||
| def connectionInactive(client: TransportClient): Unit = { | |||
| val applicationIds = appActiveConnections.remove(client.getChannel.id().asLongText()) | |||
| incCounter(ACTIVE_CONNECTION_COUNT, -1) | |||
| applicationIds.asScala.foreach(applicationId => | |||
| incCounter(ACTIVE_CONNECTION_COUNT, -1, Map(applicationLabel -> applicationId))) | |||
| if (null != applicationIds) { | |||
There was a problem hiding this comment.
This was causing NPE in the tests
Thanks for tagging me here. Will review the PR shortly. |
venkata91
suggested changes
Jul 2, 2024
...mmon/src/main/java/org/apache/celeborn/plugin/flink/network/FlinkTransportClientFactory.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
|
Thanks @venkata91, @RexXiong ... have addressed the comments. |
venkata91
approved these changes
Jul 3, 2024
RexXiong
approved these changes
Jul 4, 2024
|
Thanks @venkata91, @RexXiong ! |
|
Let me backport this to 0.5 branch as well @RexXiong |
mridulm
pushed a commit
to mridulm/celeborn
that referenced
this pull request
Jul 4, 2024
Merge conflicts in branch 0.5 Fix authentication support for Apache Flink. Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled. Fixes authentication support for Apache Flink integration This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled. Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster. Closes apache#2596 from mridulm/fix-flink-auth-support. Authored-by: Mridul Muralidharan <mridulatgmail.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
mridulm
pushed a commit
that referenced
this pull request
Jul 4, 2024
…ckport to 0.5 Backport of #2596 to branch-0.5 Conflicts were in: * ApiBaseResourceAuthenticationSuite.scala * ApiBaseResourceSuite.scala Description from #2596: ### What changes were proposed in this pull request? Fix authentication support for Apache Flink. ### Why are the changes needed? Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled. ### Does this PR introduce _any_ user-facing change? Fixes authentication support for Apache Flink integration ### How was this patch tested? This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled. Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster. Closes #2603 from mridulm/fix-flink-auth-support-v0.5. Authored-by: Mridul Muralidharan <mridulatgmail.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix authentication support for Apache Flink.
Why are the changes needed?
Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled.
Does this PR introduce any user-facing change?
Fixes authentication support for Apache Flink integration
How was this patch tested?
This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled.
Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster.