DataFrame.collect() should be extensible

[andygrove]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Ballista provides its own execution context but uses the DataFusion DataFrame. Calling collect on the DataFrame will run the query in-memory rather than distributed and Ballista users must instead extract the logical plan from the DataFrame and call BallistaContext.collect instead. This is not good UX.

Describe the solution you'd like
As a user, I would just like to call DataFrame.collect() and have it run either in-memory or distributed depending on how I created the context.

I think the way to do this is by making it possible to customize ExecutionContext and override the behavior when a DataFrame is collected.

Describe alternatives you've considered
None

Additional context
None

[jorgecarleitao]

One way to address this:

• Create a trait ExecutionContext with necessary APIs
• make DataFrame hold Box<dyn ExecutionContext>

This allows DataFusions' DataFrame to have different contexts resolved at run time (e.g. whether local or cluster is passed?).

[andygrove]

I am closing this since I found a different solution to this issue. Ballista uses the DataFusion context but provides its own query planner which results in a DistributedQueryExec physical plan, so DataFrame.collect() invokes that.