Add Roadmap to Documentation

[alamb]

Which issue does this PR close?

Closes #1102 suggested by @xudong963

** All Suggestions Welcome **

Rationale for this change

1. Coordinate development work
2. Help Developers new to datafusion to understand where we are going

See also:

• mailing list discussion
• https://news.ycombinator.com/item?id=28290616
• Crowd Sourced Roadmap Jan 2021

What changes are included in this PR?

Add roadmap to docs published on https://arrow.apache.org/datafusion/

Are there any user-facing changes?

Docs

[xudong963]

_the de facto query engine_ typo?

[jimexist]

It means the query engine of choice by fact

[alamb]

I will reword this to less idiomatic English

[xudong963]

Got it. My poor English needs improving (/ω＼)

[jimexist]

it's okay :-) @xudong963 it was not english and indeed a latin phrase to begin with.

@alamb i think de facto is a common phrase, one way to make it smoother is to put it in italic (like in many books containing latin expressions).

i'd make probably hammer it home by including more examples inline, even by using footnote (not sure GitHub markdown supports that).

[alamb]

👍 makes sense. I do think in general it is a good idea to try and minimize 'flowery prose' and idiomatic English is probably the best plan for what have become world wide communities.

[andygrove]

I will contribute some content for the Ballista vision

[xudong963]

2021 is less than three months away. I have a plan to do something for datafusion for the rest of the time.

1. Implement the rest of Set Operators: INTERSECT, EXCEPT, etc #1082
2. Make subqueries more complete: make some of the tests pass https://github.com/apache/arrow-datafusion/issues?q=is%3Aopen+is%3Aissue+milestone%3ATPC-H. Later I'll pull a specific issue and split the task into small pieces.
3. some easy and small fixes

[jimexist]

I think it's good to merge as is, more detailed sections can be subsequent pull requests.

[xudong963]

I think it's good to merge as is, more detailed sections can be subsequent pull requests.

@andygrove said he'll contribute some content for the Ballista vision👀

[alamb]

Thanks @jimexist and @xudong963 . I'll plan to leave this open for the weekend to give anyone who has more time then to contribute and then merge it early next week

[houqp]

thanks @alamb for putting this up.

If I recall correctly, @jorgecarleitao has plans to decouple datafusion into smaller reusable crates too. But if @jorgecarleitao and @andygrove are too busy adding their items right now, we could merge and add those items later as follow up PRs.

@Dandandan do you want to add your tokomak optimizer to the list?

[houqp]

also cc @yjshen in case we missed any item needed from your native spark executor work.

[Dandandan]

@Dandandan do you want to add your tokomak optimizer to the list?
Yes, added!

[yjshen]

also cc @yjshen in case we missed any item needed from your native spark executor work.

Thanks, @houqp. I think what I need most is covered by the Resource Management section. I'm working on prototyping a memory limit version of SortExec currently.

On the Ballista side, I feel Broadcast join is great to add. Besides, we could have a sort-based shuffle writer for memory usage friendly and have a single map output file for each task to avoid creating too many small files when the output partition number is significant.

[rdettai]

Would it be the right place to rate the maturity of each component? I feel that there is a pretty big gap between the DataFusion part of the codebase and Ballista (hard coded parameters, huge functions to be decomposed, un-supported SQL...). This information would be very important for a newcomer, but I am not sure how to formulate it 😄

[alamb]

Would it be the right place to rate the maturity of each component? I feel that there is a pretty big gap between the DataFusion part of the codebase and Ballista (hard coded parameters, huge functions to be decomposed, un-supported SQL...). This information would be very important for a newcomer, but I am not sure how to formulate it 😄

This would be a great thing to describe, but the roadmap is probably not the right place for it (maybe the roadmap could have an entry like "* Mature ballista (see for details)"

How about a document in the userguide https://arrow.apache.org/datafusion/#toc-guide somewhere?

[alamb]

We can add changes as follow on PRs perhaps. Thanks everyone for all the help!

[xudong963]

We can add changes as follow on PRs perhaps. Thanks everyone for all the help!

🎉

[houqp]

Thank you @alamb for making this happen. I can help deploy the doc change later today.

Reminder for @jorgecarleitao and @andygrove to add your entries before the upcoming datafusion 6.0.0 release ;)