Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log the list of executor nodes #225

Closed
yruslan opened this issue Aug 9, 2023 · 0 comments
Closed

Log the list of executor nodes #225

yruslan opened this issue Aug 9, 2023 · 0 comments
Assignees
Labels
enhancement New feature or request Pramen-Scala

Comments

@yruslan
Copy link
Collaborator

yruslan commented Aug 9, 2023

Background

This helps debugging issues.

Feature

Log the list of executor nodes

Proposed Solution

The sure way of getting this list takes time:

    val sc = spark.sparkContext

    val data = sc.parallelize(1 to 10000000).repartition(sc.defaultParallelism)
    data.mapPartitions { _ => Iterable(java.net.InetAddress.getLocalHost.getHostName).iterator }.collect().distinct.sorted

so it should be configured and turned off by default:

log.executor.nodes = false
@yruslan yruslan added enhancement New feature or request Pramen-Scala labels Aug 9, 2023
@yruslan yruslan self-assigned this Aug 9, 2023
@yruslan yruslan closed this as completed Aug 10, 2023
@yruslan yruslan reopened this Aug 10, 2023
@yruslan yruslan closed this as completed Aug 10, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Pramen-Scala
Projects
None yet
Development

No branches or pull requests

1 participant