Document DataFusion Threading (and how to separate IO and CPU bound work)

[tustvold]

Is your feature request related to a problem or challenge?

DataFusion performs CPU bound work within async closures. This causes issues if running IO on the same async runtime, as the cooperative nature of such schedulers allows the CPU bound work to starve servicing of IO. This leads to errors such as apache/arrow-rs#5882.

Describe the solution you'd like

I think at the very least this needs to be better documented, I couldn't find any mention of this in the DataFusion documentation following a cursory search.

I also think more holistic approach would be valuable to this, as it stands the use of async within DataFusion acts as a massive footgun that encourages users to intermix IO and CPU work in a way that is at best inefficient, but this can be tracked as a separate follow on task.

Describe alternatives you've considered

No response

Additional context

No response

[alamb]

I recommend two things:

1. Write a blog with background and explanation of why using two threadpools is important with DataFusion and examples of how to do it
2. Add additional documentation (ideally linking to the blog) with a summary and linking to the blog with content.

[ozankabak]

I think it'd be great to have a good documentation on this.

[alamb]

I think it'd be great to have a good documentation on this.

100% agree -- @itsjunetime and @tustvold are working on a bit of it in apache/arrow-rs#6612. I'll try and help with the documentation as well