add .parquet files data preview

[RandomFractals]

see finos/perspective#342 for more info

[gbrueckl]

Any time frame when this will be available?
Would like to refer to your extension when it comes to viewing parquet files

[RandomFractals]

I have not looked into it in a while. Last time I checked nodejs parquet lib had some issues and required native modules which I can't support in vscode ...

[gbrueckl]

Looks like there is a new package to work with parquet in TypeScript
https://github.com/kbajalc/parquets

Would this work?

[RandomFractals]

that might work. I'll take it for a spin this weekend.

Thanks for linking me to it.

[gbrueckl]

any update to this?

[RandomFractals]

hoping to look more into this in fall.

[RandomFractals]

@gbrueckl good news. I tried parquets lib you suggested and looks like we can load one of the sample parquet data files now.

I'll see if I can find more parquet data files online and if it works with more examples, I'll package and ship this soon.

[gbrueckl]

awesome!
here are some parquet files that you can test (they are from a sample database)
https://gbpublic.blob.core.windows.net/data/Delta/AdventureWorksDW2012/DimAccount/part-00000-c80d6666-85ea-4b0e-97b2-2d424292b0c1-c000.snappy.parquet
https://gbpublic.blob.core.windows.net/data/Delta/AdventureWorksDW2012/DimDate/part-00000-667f9b57-ecaf-452b-880d-8e1addd46354-c000.snappy.parquet
https://gbpublic.blob.core.windows.net/data/Delta/AdventureWorksDW2012/FactProductInventory/part-00000-9e3e5c91-e2f2-4ee6-aec4-acfd675f9881-c000.snappy.parquet
https://gbpublic.blob.core.windows.net/data/Delta/AdventureWorksDW2012/DimProduct/part-00000-0cb4eaf5-02fd-46ab-8019-dc90d0a9d6ff-c000.snappy.parquet
https://gbpublic.blob.core.windows.net/data/Delta/AdventureWorksDW2012/FactInternetSales_part_big.delta/SalesTerritoryKey%3D4/part-00001-6d55c0d8-ee35-44ec-8d1a-9af7653522c6.c000.snappy.parquet (this one is bigger, 50 MB)

let me know if I can support you further!

regards,
-gerhard

[JacekPliszka]

One small favour - could you add .pq extension for parquet as well?

[RandomFractals]

@gbrueckl thanks for the test files. I hope they get loaded b/c I tried some from https://github.com/apache/parquet-testing and did not have much luck with the data files from that repo due to encoding or shortcomings of the parquets and the underlying parquetjs lib we are using.

@JacekPliszka I'll map .pq extension to parquet data provider. Is that a standard short file extension for parquet files? I have not seen it before.

Let me know if you can get your parquet files to load. This release is super alpha and I am sure there is much more we'll need to tweak to make it work with snappy compression, etc.

[JacekPliszka]

@RandomFractals I think only .parquet is "standard" but you allow .parq which I've never seen while .pq you can find used by different people on the internet. Possibly because this is a standard import in Python:

import pyarrow.parquet as pq

[RandomFractals]

btw, @gbrueckl I tried parquet files you shared. None of them load. I get page dictionary errors. will have to dig into that more later.

[peder1001]

Any updates on whether you will support parquet?

[RandomFractals]

@peder1001 most likely in the new Tabular Data Viewer extension I started working on this year, which does include Tabulator and Perspective viewer. You can try it with CSV data files from marketplace for now:

https://marketplace.visualstudio.com/items?itemName=RandomFractalsInc.tabular-data-viewer