Skip to content

Latest commit

 

History

History
54 lines (39 loc) · 2.15 KB

README.md

File metadata and controls

54 lines (39 loc) · 2.15 KB
Raw

Meilisearch

Script used at SneakersAPI.dev to index Meilisearch from PostgreSQL.

Key features:

  • Replicates data from PostgreSQL to Meilisearch.
  • Should support almost any types, the inserted data is in JSON.
  • Time-series data can be synced via cursor to avoid full table scans.

About performance:

  • 180k rows with complex nested JSON: 45s, about ~3.9k rows/s (async enabled)

Configuration

Configuration is done via a YAML file. See config.example.yml for reference.

About enable_async and wait_time:

  • When enable_async is true, batches are sent in a separate thread, it greatly improves performance but the memory usage of the MeiliSearch instance is much higher.
  • When enable_async is false, wait_time is the time in milliseconds to wait before sending the next batch. This is useful on low-end VPS to avoid memory issues as the MeiliSearch instance will have time to free up memory between batches.
  • For low-memory VPS, it's recommended to use enable_async: false and to set a wait_time, depending on the number of rows to sync. You might be interested to use these environment variables on the MeiliSearch instance also:
    • MEILI_EXPERIMENTAL_REDUCE_INDEXING_MEMORY_USAGE=true
    • MEILI_EXPERIMENTAL_MAX_NUMBER_OF_BATCHED_TASKS=5

Running

export MEILISEARCH_DSN=<meilisearch_dsn>
export MEILISEARCH_API_KEY=<meilisearch_api_key>
export DATABASE_URL=<database_url>

go run . [-only=<index_name>] [-drop=[true|false]] [-meta=[true|false]] [-config=<path>]
  • -only=<index_name>: Avoid running all indexes and only process the one specified, by its destination name.
  • -drop=[true|false]: Drop the index and reset cursor, if any.
  • -meta=[true|false]: Update index metadata (searchable, filterable, sortable).
  • -config=<path>: Path to the configuration file. Defaults to config.yml.

Docker

docker build -t meilisearch .
docker run -e MEILISEARCH_DSN=<meilisearch_dsn> \
    -e MEILISEARCH_API_KEY=<meilisearch_api_key> \
    -e DATABASE_URL=<database_url> \
    meilisearch \
    [-only=<index_name>] \
    [-drop=[true|false]] \
    [-meta=[true|false]] \
    [-config=<path>]