doc: Document CometPlugin to start Comet in cluster mode #836
Conversation
comphead
changed the title
| @@ -37,6 +37,15 @@ Comet will allocate at least `spark.comet.memory.overhead.min` memory. | |||
|
|
|||
| If both `spark.comet.memoryOverhead` and `spark.comet.memory.overhead.factor` are set, the former will be used. | |||
|
|
|||
| ## Memory Tuning in clusters | |||
| In clusters like Kubernetes or YARN to make the resource manager respect correctly Comet memory parameters `spark.comet.memory*` | |||
| it is needed to pass to the starting command line additional Spark configuration parameter `--conf spark.plugins=org.apache.spark.CometPlugin` | |||
There was a problem hiding this comment.
I'm curious why we don't specify the plugin in all cases instead of --conf spark.sql.extensions=org.apache.comet.CometSparkSessionExtensions since the plugin does (or can) register the session extensions?
Other Spark accelerators specify a plugin.
There was a problem hiding this comment.
Thats a great question. Its not needed for local run, but Spark in enterprises get started mostly in clusters. Perhaps we can also update the installation guide and make a remark about the plugin if people run the Comet in cluster mode
There was a problem hiding this comment.
I think it is mostly because we develop the extension at the beginning of Comet project. So all documents and customs are to specify the extension. We develop the Comet plugin after a while when we want to have it to configure memory easily.
| ### Cluster mode | ||
| Running in cluster mode it might be needed to set addtitional `--conf spark.plugins=org.apache.spark.CometPlugin` | ||
| to make the cluster resource managares respect Comet memory parameters. More [Memory Tuning in cluster mode](./tuning.md#memory-tuning-in-cluster-mode) |
There was a problem hiding this comment.
It is not specified for cluster usage, I think.
There was a problem hiding this comment.
It is not specified for cluster usage, I think.
sorry, I dont follow. Does the comment require any fix? 🤔
There was a problem hiding this comment.
Yea, I don't think CometPlugin is bound to cluster mode. We don't need this section probably.
There was a problem hiding this comment.
I feel we need mention it on the installation page, as this is the first page user interacts with and they interested in this topic. Perhaps we can rephrase it somehow
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com>
|
@andygrove @viirya please have a second look? |
* Documenting CometPlugin Co-authored-by: Liang-Chi Hsieh <viirya@gmail.com> (cherry picked from commit 27ab86b)
Which issue does this PR close?
Closes #826
Closes #605
Rationale for this change
Document the way to start Comet in cluster mode so the resource manager respects Comet memory configuration parameters
What changes are included in this PR?
How are these changes tested?