Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-7544][CORE] Add Qualification Tool #8484

Merged
merged 5 commits into from
Jan 23, 2025
Merged

[GLUTEN-7544][CORE] Add Qualification Tool #8484

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
344b1e4
[GLUTEN-7544][CORE] Add Qualification Tool
srinivasst Jan 9, 2025
957b43a
Fix README.MD
srinivasst Jan 9, 2025
7e531f1
Merge branch 'main' into QT
srinivasst Jan 9, 2025
371cbae
Add UT
srinivasst Jan 17, 2025
88104e2
Merge branch 'refs/heads/main' into QT
srinivasst Jan 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,10 +123,31 @@ The below testing environment: a 8-nodes AWS cluster with 1TB data; Spark-3.1.1

![Performance](./docs/image/clickhouse_decision_support_bench1_22queries_performance.png)

# 8. License
# 8. Qualification Tool

The Qualification Tool is a utility to analyze Spark event log files and assess the compatibility and performance of SQL workloads with Gluten. This tool helps users understand how their workloads can benefit from Gluten.

## Features
- Analyzes Spark SQL execution plans for compatibility with Gluten.
- Supports various types of event log files, including single files, folders, compressed files, and rolling event logs.
- Generates detailed reports highlighting supported and unsupported operations.
- Provides metrics on SQL execution times and operator impact.
- Offers configurable options such as threading, output directory, and date-based filtering.

## Usage

To use the Qualification Tool, follow the instructions in its [README](tools/qualification-tool/README.MD).

## Example Command
```bash
java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f /path/to/eventlog
```
For detailed usage instructions and advanced options, see the Qualification Tool README.

# 9. License
Gluten is licensed under [Apache 2.0 license](https://www.apache.org/licenses/LICENSE-2.0).

# 9. Acknowledgements
# 10. Acknowledgements
Gluten was initiated by Intel and Kyligence in 2022. Several companies are also actively participating in the development, such as BIGO, Meituan, Alibaba Cloud, NetEase, Baidu, Microsoft, IBM, Google, etc.

<a href="https://github.com/apache/incubator-gluten/graphs/contributors">
Expand Down
10 changes: 10 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,16 @@
</properties>
</profile>

<profile>
<id>qualification-tool</id>
<activation>
<activeByDefault>false</activeByDefault>
</activation>
<modules>
<module>tools/qualification-tool</module>
</modules>
</profile>

<profile>
<id>backends-clickhouse</id>
<activation>
Expand Down
1 change: 1 addition & 0 deletions tools/qualification-tool/.gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
target
217 changes: 217 additions & 0 deletions tools/qualification-tool/README.MD
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
# Qualification Tool

The Qualification Tool analyzes Spark event log files to determine the compatibility and performance of SQL workloads with Gluten.

## Build

To compile and package the Qualification Tool, run the following Maven command:

```bash
mvn clean package
```

This will create a jar file in the `target` directory.

## Run

To execute the tool, use the following command:

```bash
java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f <eventFile>
```

### Parameters:
- **`-f <eventFile>`**: Path to the Spark event log file(s). This can be:
- A single event log file.
- A folder containing multiple event log files.
- Deeply nested folders of event log files.
- Compressed event log files.
- Rolling event log files.
- Comma separated files
- **`-k <gcsKey>`**: (Optional) Path to Google Cloud Storage service account keys.
- **`-o <output>`**: (Optional) Path to the directory where output will be written. Defaults to a temporary directory.
- **`-t <threads>`**: (Optional) Number of processing threads. Defaults to 4.
- **`-v`**: (Optional) Enable non verbose output. Omit this flag for verbose mode.
- **`-p <project>`**: (Optional) Project ID for the run.
- **`-d <dateFilter>`**: (Optional) Analyze only files created after this date (format: YYYY-MM-DD). Defaults to the last 90 days.

### Example Usage:
```bash
java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f /path/to/eventlog
```

### Advanced Example:
```bash
java -jar target/qualification-tool-1.3.0-SNAPSHOT-jar-with-dependencies.jar -f /path/to/folder -o /output/path -t 8 -d 2023-01-01 -k /path/to/gcs_keys.json -p my_project
```

## Features

- Analyzes Spark SQL execution plans for compatibility with Gluten.
- Supports single files, folders, deeply nested folders, compressed files, and rolling event logs.
- Provides detailed reports on supported and unsupported operations.
- Generates metrics on SQL execution times and operator impact.
- Configurable verbosity and threading.

## How It Works

The Qualification Tool analyzes a Spark plan to determine the compatibility of its nodes / operators and clusters with Gluten. Here's a step-by-step explanation of the process:

### Example Spark Plan

Consider the following Spark plan:

```
G
/ \
G[2] G[3]
| |
S[2] G[3]
| |
G S
/ \
G[1] G
|
G[1]
|
G
```

- **G**: Represents a plan supported by Gluten.
- **S**: Represents a plan not supported by Gluten.
- **[1], [2], [3]**: Indicates the node belongs to a Whole Stage Code Gen Block (Cluster).

### 1. NodeSupportVisitor

The first step is marking each node as supported (`*`) or not supported (`!`) by Gluten:

```
*G
/ \
*G[2] *G[3]
| |
!S[2] *G[3]
| |
*G !S
/ \
*G[1] *G
|
*G[1]
|
*G
```

- All supported nodes are marked with `*`.
- All unsupported nodes are marked with `!`.

### 2. ClusterSupportVisitor

The second step marks entire clusters as not supported (`!`) if any node in the cluster is unsupported:

```
*G
/ \
!G[2] *G[3]
| |
!S[2] *G[3]
| |
*G !S
/ \
*G[1] *G
|
*G[1]
|
*G
```

#### Reasoning:
Although Gluten supports these operators, breaking Whole Stage Code Gen (WSCG) boundaries introduces row-to-columnar and columnar-to-row conversions, degrading performance. Hence, we pessimistically mark the entire cluster as not supported.

### 3. ChildSupportVisitor

The final step marks nodes and their parents as not supported if their children are unsupported:

```
!G
/ \
!G[2] !G[3]
| |
!S[2] !G[3]
| |
*G !S
/ \
*G[1] *G
|
*G[1]
|
*G
```

#### Reasoning:
If a child node is not supported by Gluten, row-to-columnar and columnar-to-row conversions are added, degrading performance. Therefore, we pessimistically mark such nodes as not supported.

### Sample Output
#### AppsRecommendedForBoost.tsv
| applicationId | applicationName | batchUuid | rddPercentage | unsupportedSqlPercentage | totalTaskTime | supportedTaskTime | supportedSqlPercentage | recommendedForBoost | expectedRuntimeReduction |
|-----------------------------------|-----------------------------------|-----------------------------------------|---------------|--------------------------|---------------|-------------------|-------------------------|---------------------|--------------------------|
| app-20241001064609-0000 | POC - StoreItem | a56bee42 | 0.0% | 5.2% | 3244484900 | 3074633672 | 94.7% | true | 28.4% |
| application_1729530422325_0001 | job-audience-generation-21 | UNKNOWN | 0.3% | 0.6% | 1378942742 | 1365259621 | 99.0% | true | 29.7% |
| application_1729410290443_0001 | job-audience-generation-27 | UNKNOWN | 0.0% | 0.7% | 1212612881 | 1203690936 | 99.2% | true | 29.7% |
| app-20241001080158-0000 | POC - StoreItem | 94088807 | 0.0% | 12.5% | 668601513 | 584812056 | 87.4% | true | 26.2% |
| application_1729991008434_0001 | DailyLoad | UNKNOWN | 0.0% | 12.6% | 17134675 | 14963535 | 87.3% | true | 26.1% |
| application_1730097715348_0003 | Spark shell | UNKNOWN | 0.0% | 0.0% | 4680 | 4680 | 100.0% | true | 30.0% |
| application_1728526688221_0001 | job-audience-generation-27 | UNKNOWN | 0.4% | 59.4% | 805991113 | 323274568 | 40.1% | false | 12.0% |
| application_1726070403340_0450 | driver.py | UNKNOWN | 0.0% | 81.1% | 398992332 | 75187879 | 18.8% | false | 5.6% |
| application_1727842554841_0001 | driver.py | UNKNOWN | 0.0% | 58.3% | 166686890 | 69492539 | 41.6% | false | 12.5% |
| application_1726070403340_0474 | driver.py | UNKNOWN | 0.0% | 81.6% | 325389669 | 59687634 | 18.3% | false | 5.5% |
| application_1729133595704_0001 | job-audience-generation-54 | UNKNOWN | 0.6% | 99.3% | 472621872 | 20205 | 0.0% | false | 0.0% |
| application_1730097715348_0002 | Spark shell | UNKNOWN | 33.8% | 16.6% | 4197 | 2077 | 49.5% | false | 14.8% |
| application_1730097715348_0001 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0011 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0012 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0007 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| app-20241120163343-0000 | Config Test | c2884285 | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0008 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0010 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1730097715348_0004 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1718075857669_0261 | UnregisteredAsset | UNKNOWN | 0.0% | 100.0% | 12205458 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0009 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |
| application_1712155629437_0013 | Spark shell | UNKNOWN | 0.0% | 0.0% | 0 | 0 | 0.0% | false | 0.0% |

#### UnsupportedOperators.tsv
| unsupportedOperator | cumulativeCpuMs | count |
|--------------------------------------------------------|-----------------|---------|
| Scan with format JSON not supported | 1722606716687 | 3105 |
| Parquet with struct | 1722606716687 | 3105 |
| Execute InsertIntoHadoopFsRelationCommand not supported| 37884897400 | 405732 |
| BHJ Not Supported | 20120117622 | 132164 |
| SerializeFromObject not supported | 13856742899 | 16098 |
| MapPartitions not supported | 13856742899 | 16098 |
| FlatMapGroupsInPandas not supported | 257970400 | 224 |
| Sample not supported | 157615298 | 432 |
| DeserializeToObject not supported | 157610379 | 6721 |
| MapElements not supported | 5261628 | 694 |
| Parquet with struct and map | 29020 | 44 |
| Execute CreateViewCommand not supported | 0 | 157 |
| Execute CreateTableCommand not supported | 0 | 42 |
| WriteFiles not supported | 0 | 20 |
| Execute SetCommand not supported | 0 | 12 |
| Execute SaveIntoDataSourceCommand not supported | 0 | 10 |
| CreateTable not supported | 0 | 4 |
| Execute RefreshTableCommand not supported | 0 | 3 |
| Execute DeltaDynamicPartitionOverwriteCommand not supported | 0 | 2 |
| Execute TruncateTableCommand not supported | 0 | 2 |
| Execute AlterTableAddPartitionCommand not supported | 0 | 1 |
| CreateNamespace not supported | 0 | 1 |


### Summary

The tool ensures that the Spark plan optimizes performance by:
1. Identifying individual node compatibility.
2. Accounting for cluster boundaries and WSCG optimizations.
3. Considering child dependencies and their impact on parent nodes.

## Requirements

- **Java**: Ensure you have JDK 11 or later installed.
Loading
Loading