Setting up and configuring Minio and uploading a Parquet data file

[alberttwong]

This tutorial describes how you can setup and configure Minio.

Prerequisites

For this tutorial you need to:

• Have Docker Desktop or podman container runtime installed
• Download a Snowflake provided Parquet data file.
• Download and configure Minio (an open source object store that you can install on your own workstation) and upload the parquet file into the bucket.

Docker Desktop or podman container runtime installed

This is out of scope for the tutorial. Please see Docker Desktop or podman's website on how to install that software.

Downloading the Sample Data File

To download the sample Parquet data file, click cities.parquet.

The Parquet data file includes sample continent data. The following is a representative example:

{
  "continent": "Europe",
  "country": {
    "city": [
      "Paris",
      "Nice",
      "Marseilles",
      "Cannes"
    ],
    "name": "France"
  }
}

Download and configure Minio (an open source object store that you can install on your own workstation) and upload the parquet file into the bucket

The following commands will download Minio, configure a bucket, create credentials and upload the parquet file into the bucket

Step 1: Run Minio

Going to https://min.io/download#/docker

podman run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"

Step 2: Login to Minio

Login the UI at http://localhost:9001

Step 3: Create S3 compatible bucket

Click on "creating a new bucket". I named it "albertwong".

Issue: There is an outstanding github issue with bucket names with "." in them. Github Issue #22711

Step 4: Create credentials

Then you need credentials. Make sure you save the access_key and secret_key since you'll need that in the next steps.

Step 5: Upload parquet file

Now you'll upload the parquet file. To double check, make sure the file as been uploaded.

[masalinas]

Sorry but in this steps, you don't comment anything about configure parquet format in minio because this functionality is disable by default. I tried to add the environment variable MINIO_API_SELECT_PARQUET set to on in tje operator, but I can't read any parquet file with this error:

botocore.exceptions.ClientError: An error occurred (InternalError) when calling the SelectObjectContent operation (reached max retries: 4): We encountered an internal error, please try again.: cause(parquet format parsing not enabled on server)