From 664e39c3edd887b77bd977bd30bb2ead99b64431 Mon Sep 17 00:00:00 2001 From: Josh Taylor Date: Thu, 23 Jun 2022 13:18:25 +0800 Subject: [PATCH] Add notes about creating parquet files in the development documentation --- DEVELOPMENT.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index cc9211ef996..89dbcc994ec 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -18,6 +18,32 @@ This runs the tests of the crate without features. To run all features, use cargo test --features full ``` +If you get warnings about parquet files not existing, you can generate the test files by using Python: + +```bash +# Ubuntu: sudo apt install python3-pip python3-venv +# Mac: brew install python3 +# Archlinux: sudo pacman -Syu python + +# Create a virtual environment for python, to keep dependencies contained +python3 -m venv venv + +# Activate the virtual environment +source venv/bin/activate + +# Make sure pip is up to date +pip install pip --upgrade + +# Install pyarrow, version 6 +pip install pyarrow==6 + +# Generate the parquet files (this might take some time, depending on your computer setup) +python parquet_integration/write_parquet.py + +# Get out of venv, back to normal terminal +deactivate +``` + during development of particular parts of the crate, it is usually faster to reduce the feature set - the tests are gated to only the relevant tests of that feature set. For example, if improving JSON, you can use