Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-7544][CORE] Add Qualification Tool #8484

Merged
merged 5 commits into from
Jan 23, 2025
Merged

Conversation

srinivasst
Copy link
Contributor

@srinivasst srinivasst commented Jan 9, 2025

Adding a qualification tool that can go through Spark event files and output percentage of the application that can be run using Gluten.

Fixes: #7544

@github-actions github-actions bot added CORE works for Gluten Core TOOLS labels Jan 9, 2025
Copy link

github-actions bot commented Jan 9, 2025

#7544

Copy link

github-actions bot commented Jan 9, 2025

Run Gluten Clickhouse CI on x86

1 similar comment
Copy link

github-actions bot commented Jan 9, 2025

Run Gluten Clickhouse CI on x86

@Yohahaha
Copy link
Contributor

Yohahaha commented Jan 9, 2025

could you help confirm which class/rule do the work that decide whether a operator is supported or not?

@srinivasst
Copy link
Contributor Author

@Yohahaha tools/qualification-tool/src/main/java/org/apache/gluten/qt/support/NodeSupportVisitor.java marks nodes as supported / not supported

@FelixYBW
Copy link
Contributor

FelixYBW commented Jan 9, 2025

Good feature! It's something we want to have for long time. Would you add a section and a link to main page (https://github.com/apache/incubator-gluten/blob/main/README.md)

@FelixYBW
Copy link
Contributor

FelixYBW commented Jan 9, 2025

Also can you add an example output file?

<name>Qualification Tool</name>

<properties>
<maven.compiler.source>11</maven.compiler.source>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we rely on the top level pom's java version here?

tools/qualification-tool/pom.xml Show resolved Hide resolved
tools/qualification-tool/pom.xml Show resolved Hide resolved
@github-actions github-actions bot added the DOCS label Jan 17, 2025
Copy link

Run Gluten Clickhouse CI on x86

# Conflicts:
#	README.md
Copy link

Run Gluten Clickhouse CI on x86

@srinivasst
Copy link
Contributor Author

@FelixYBW I have addressed your comments and added sample output.
I also have added an UT that compares output against expected results.

@FelixYBW
Copy link
Contributor

@FelixYBW I have addressed your comments and added sample output. I also have added an UT that compares output against expected results.

Thank you. Let me have a quick try.

Copy link
Contributor

@FelixYBW FelixYBW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thank you to add the big feature!

@FelixYBW FelixYBW merged commit a1e7258 into apache:main Jan 23, 2025
47 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CORE works for Gluten Core DOCS TOOLS
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Core] Qualification Tool for Gluten
4 participants