support select .. FROM 'parquet.file' in datafusion-cli

[unconsolable]

Which issue does this PR close?

Closes #4580 .

Rationale for this change

See #4580

What changes are included in this PR?

Follow up #4581

• Add DynamicFileCatalog, DynamicFileCatalogProvider, DynamicFileCatalogProvider, which can try to create a ListingTable when table is not found.
• Register DynamicFileCatalog in SessionContext

Are these changes tested?

Manually tested.

$ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.19s
     Running `target/debug/datafusion-cli`
DataFusion CLI v15.0.0
❯ select * from '../parquet-testing/data/alltypes_tiny_pages.parquet' limit 10;
+-----+----------+-------------+--------------+---------+------------+-----------+--------------------+-----------------+------------+-------------------------+------+-------+
| id  | bool_col | tinyint_col | smallint_col | int_col | bigint_col | float_col | double_col         | date_string_col | string_col | timestamp_col           | year | month |
+-----+----------+-------------+--------------+---------+------------+-----------+--------------------+-----------------+------------+-------------------------+------+-------+
| 122 | true     | 2           | 2            | 2       | 20         | 2.2       | 20.2               | 01/13/09        | 2          | 2009-01-13T01:02:05.410 | 2009 | 1     |
| 123 | false    | 3           | 3            | 3       | 30         | 3.3       | 30.299999999999997 | 01/13/09        | 3          | 2009-01-13T01:03:05.430 | 2009 | 1     |
| 124 | true     | 4           | 4            | 4       | 40         | 4.4       | 40.4               | 01/13/09        | 4          | 2009-01-13T01:04:05.460 | 2009 | 1     |
| 125 | false    | 5           | 5            | 5       | 50         | 5.5       | 50.5               | 01/13/09        | 5          | 2009-01-13T01:05:05.500 | 2009 | 1     |
| 126 | true     | 6           | 6            | 6       | 60         | 6.6       | 60.599999999999994 | 01/13/09        | 6          | 2009-01-13T01:06:05.550 | 2009 | 1     |
| 127 | false    | 7           | 7            | 7       | 70         | 7.7       | 70.7               | 01/13/09        | 7          | 2009-01-13T01:07:05.610 | 2009 | 1     |
| 128 | true     | 8           | 8            | 8       | 80         | 8.8       | 80.8               | 01/13/09        | 8          | 2009-01-13T01:08:05.680 | 2009 | 1     |
| 129 | false    | 9           | 9            | 9       | 90         | 9.9       | 90.89999999999999  | 01/13/09        | 9          | 2009-01-13T01:09:05.760 | 2009 | 1     |
| 130 | true     | 0           | 0            | 0       | 0          | 0         | 0                  | 01/14/09        | 0          | 2009-01-14T01:10:05.850 | 2009 | 1     |
| 131 | false    | 1           | 1            | 1       | 10         | 1.1       | 10.1               | 01/14/09        | 1          | 2009-01-14T01:11:05.850 | 2009 | 1     |
+-----+----------+-------------+--------------+---------+------------+-----------+--------------------+-----------------+------------+-------------------------+------+-------+
10 rows in set. Query took 0.156 seconds.

Are there any user-facing changes?

Syntax select .. FROM 'abc.parquet' is supported in datafusion-cli.

[alamb]

Thank you for this PR @unconsolable -- I plan to review this PR tomorrow

[alamb]

Thank you so much @unconsolable ! I tried it out locally and it was soo cool:

I think it is ok to merge without tests because we don't really have a good testing story for datafusion-cli at the moment. However, I think this feature could be useful for other users of datafusion (not just datafusion-cli) so I will file a ticket to add this feature to the core of datafusion as well

(arrow_dev) alamb@MacBook-Pro-8:~/Software/arrow-datafusion2/datafusion-cli$ CARGO_TARGET_DIR=/Users/alamb/Software/target-df2 cargo run --bin datafusion-cli 
    Finished dev [unoptimized + debuginfo] target(s) in 1.33s
     Running `/Users/alamb/Software/target-df2/debug/datafusion-cli`
DataFusion CLI v15.0.0
❯ select * from '/Users/alamb/.influxdb_iox//1/8/1/6/e24b6549-f76c-4fc0-a4f4-152ed60eb4e3.parquet';
+---------+---------------------+------+---------+----------+---------+---------------------+-------+---------+---------+
| blocked | host                | idle | running | sleeping | stopped | time                | total | unknown | zombies |
+---------+---------------------+------+---------+----------+---------+---------------------+-------+---------+---------+
| 0       | MacBook-Pro-8.local | 0    | 2       | 697      | 0       | 2022-07-18T21:05:10 | 700   | 0       | 1       |
| 0       | MacBook-Pro-8.local | 0    | 2       | 696      | 0       | 2022-07-18T21:05:20 | 699   | 0       | 1       |
+---------+---------------------+------+---------+----------+---------+---------------------+-------+---------+---------+
2 rows in set. Query took 0.096 seconds.
❯ 

[ursabot]

Benchmark runs are scheduled for baseline = 3d75bb8 and contender = f9b72f4. f9b72f4 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ec2-t3-xlarge-us-east-2] ec2-t3-xlarge-us-east-2
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on test-mac-arm] test-mac-arm
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-i9-9960x] ursa-i9-9960x
[Skipped ⚠️ Benchmarking of arrow-datafusion-commits is not supported on ursa-thinkcentre-m75q] ursa-thinkcentre-m75q
Buildkite builds:
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

[alamb]

Not sure if others understand the elegance of @unconsolable 's solution here, but it works for other file types as well a directories of files:

❯ select * from './datafusion/core/tests/aggregate_simple.csv';
+---------+----------------+-------+
| c1      | c2             | c3    |
+---------+----------------+-------+
| 0.00001 | 0.000000000001 | true  |
| 0.00002 | 0.000000000002 | false |
| 0.00002 | 0.000000000002 | false |
| 0.00003 | 0.000000000003 | true  |
| 0.00003 | 0.000000000003 | true  |
| 0.00003 | 0.000000000003 | true  |
| 0.00004 | 0.000000000004 | false |
| 0.00004 | 0.000000000004 | false |
| 0.00004 | 0.000000000004 | false |
| 0.00004 | 0.000000000004 | false |
| 0.00005 | 0.000000000005 | true  |
| 0.00005 | 0.000000000005 | true  |
| 0.00005 | 0.000000000005 | true  |
| 0.00005 | 0.000000000005 | true  |
| 0.00005 | 0.000000000005 | true  |
+---------+----------------+-------+
15 rows in set. Query took 0.006 seconds.

I will add some documentation about this feature as it is very cool

[alamb]

proposed docs: #4851