Releases: meilisearch/meilisearch
v1.7.5 🐇
After a security flaw has been discovered in the implementation of http2, it’s possible for an attacker to slow down your instance; see this link for more information.
This PR updates our web stack to the latest version containing a fix against this attack.
What's Changed
Full Changelog: v1.7.4...v1.7.5
Contributors
Assets 8
v1.7.4 🐇
v1.7.3 🐇
This new release doesn’t contain any fixes or features.
We make it only because the release-v1.7.2 had an issue and didn’t contain all the required assets (Linux, macOS, and Windows x86 binaries were missing).
What's Changed
- Update version for the next release (v1.7.3) in Cargo.toml by @meili-bot in #4519
Full Changelog: v1.7.2...v1.7.3
Contributors
Assets 8
v1.7.2 🐇
f2f1367
Kerollmops
Clément Renault
What's Changed
- Adds a timeout to the webhook in #4508
Some assets weren’t built correctly, but the docker image is available.
Full Changelog: v1.7.1...v1.7.2
v1.7.1 🐇
Indexing Speed Improvement 🏇
- Skip reindexing when modifying unknown faceted fields by @Kerollmops in #4479
Contributors
Assets 8
v1.7.0 🐇
Meilisearch v1.7.0 focuses on improving v1.6.0 features, indexing speed and hybrid search.
🧰 All official Meilisearch integrations (including SDKs, clients, and other tools) are compatible with this Meilisearch release. Integration deployment happens between 4 to 48 hours after a new version becomes available.
Some SDKs might not include all new features—consult the project repository for detailed information. Is a feature you need missing from your chosen SDK? Create an issue letting us know you need it, or, for open-source karma points, open a PR implementing it (we'll love you for that ❤️).
New features and improvements 🔥
Improved AI-powered search — Experimental
To activate AI-powered search, set vectorStore to true in the /experimental-features route. Consult the Meilisearch documentation for more information.
🗣️ This is an experimental feature and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
New OpenAI embedding models
When configuring OpenAI embedders), you can now specify two new models:
text-embedding-3-smallwith a default dimension of 1536.text-embedding-3-largewith a default dimension of 3072.
These new models are cheaper and improve search result relevancy.
Custom OpenAI model dimensions
You can configure dimensions for sources using the new OpenAI models: text-embedding-3-small and text-embedding-3-large. Dimensions must be bigger than 0 and smaller than the model size:
"embedders": {
"new_model": {
"source": "openAi",
"model": "text-embedding-3-large",
"dimensions": 512 // must be >0, must be <= 3072 for "text-embedding-3-large"
},
"legacy_model": {
"source": "openAi",
"model": "text-embedding-ada-002"
}
}You cannot customize dimensions for older OpenAI models such as text-embedding-ada-002. Setting dimensions to any value except the default size of these models will result in an error.
GPU support when computing Hugging Face embeddings
Activate CUDA to use Nvidia GPUs when computing Hugging Face embeddings. This can significantly improve embedding generation speeds.
To enable GPU support through CUDA for HuggingFace embedding generation:
- Install CUDA dependencies
- Clone and compile Meilisearch with the
cudafeature:cargo build --release --package meilisearch --features cuda - Launch your freshly compiled Meilisearch binary
- Activate vector search
- Add a Hugging Face embedder
Improved indexing speed and reduced memory crashes
- Auto-batch task deletion to reduce indexing time (#4316) @irevoire
- Improved indexing speed for vector store (Hybrid search experimental feature indexing time more than 10 times faster) (#4332) @Kerollmops @irevoire
- Capped the maximum memory of grenad sorters to reduce memory usage (#4388) @Kerollmops
- Added multiple technical and internal indexing improvements (#4350) @ManyTheFish
- Enhance facet incremental indexing (#4433) @ManyTheFish
- Change the threshold triggering incremental indexing (#4462) @ManyTheFish
Stabilized showRankingScoreDetails
The showRankingScoreDetails search parameter, first introduce as an experimental feature in Meilisearch v1.3.0, is now a stable feature.
Use it with the /search endpoint to view detailed scores per ranking rule for each returned document:
curl \
-X POST 'http://localhost:7700/indexes/movies/search' \
-H 'Content-Type: application/json' \
--data-binary '{ "q": "Batman Returns", "showRankingScoreDetails": true }'When showRankingScoreDetails is set to true, returned documents include a _rankingScoreDetails field:
"_rankingScoreDetails": {
"words": {
"order": 0,
"matchingWords": 1,
"maxMatchingWords": 1,
"score": 1.0
},
"typo": {
"order": 1,
"typoCount": 0,
"maxTypoCount": 1,
"score": 1.0
},
"proximity": {
"order": 2,
"score": 1.0
},
"attribute": {
"order": 3,
"attributes_ranking_order": 0.8,
"attributes_query_word_order": 0.6363636363636364,
"score": 0.7272727272727273
},
"exactness": {
"order": 4,
"matchType": "noExactMatch",
"matchingWords": 0,
"maxMatchingWords": 1,
"score": 0.3333333333333333
}
}Improved logging
Log output modified
Log messages now follow a different pattern:
# new format ✅
2024-02-06T14:54:11Z INFO actix_server::builder: 200: starting 10 workers
# old format ❌
[2024-02-06T14:54:11Z INFO actix_server::builder] starting 10 workersLog output format — Experimental
You can now configure Meilisearch to output logs in JSON.
Relaunch your instance passing json to the --experimental-logs-mode command-line option:
./meilisearch --experimental-logs-mode json--experimental-logs-format accepts two values:
human: default human-readable outputjson: JSON structured logs
🗣️ This feature is experimental and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
New /logs/stream and /logs/stderr routes — Experimental
Meilisearch v1.7 introduces 2 new experimental API routes: /logs/stream and /logs/stderr.
Use the /experimental-features route to activate both routes during runtime:
curl \
-X PATCH 'http://localhost:7700/experimental-features/' \
-H 'Content-Type: application/json' \
--data-binary '{
"logsRoute": true
}'🗣️ This feature is experimental, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
/logs/stream
Use the POST endpoint to output logs in a stream. The following example disables actix logging and keeps all other logs at the DEBUG level:
curl \
-X POST http://localhost:7700/logs/stream \
-H 'Content-Type: application/json' \
--data-binary '{
"mode": "human",
"target": "actix=off,debug"
}'This endpoint requires two paramaters:
target: defines the log level and on which part of the engine you want to apply it. Must be a string formatted ascode_part=log_level. Omitcode_part=to set a single log level for the whole strram. Valid values for log level are:trace,debug,info,warn,error, oroffmode: acceptsfmt(basic) orprofile(verbose trace)
Use the DELETE endpoint of /logs/stream to interrupt a stream:
curl -X DELETE http://localhost:7700/logs/streamYou may only have one listener at a time. Meilisearch log streams are not compatible with xh or httpie.
/logs/stderr
Use the POST endpoint to configure the default log output for non-stream logs:
curl \
-X POST http://localhost:7700/logs/stream \
-H 'Content-Type: application/json' \
--data-binary '{
"target": "debug"
}'/logs/stderr accepts one parameter:
target: defines the log level and on which part of the engine you want to apply it. Must be a string formatted ascode_part=log_level. Omitcode_part=to set a single log level for the whole strram. Valid values for log level are:trace,debug,info,warn,error, oroff
Other improvements
- Prometheus experimental feature: add job variable to Grafana dashboard (#4330) @capJavert
- Multiple language support improvements, including expanded Vietnamese normalization (Ð and Đ into d). Now uses Charabia v0.8.7. (#4365) @agourlay, @choznerol, @ngdbao, @timvisee, @xshadowlegendx, and @ManyTheFish
- New experimental feature: change the behavior of Meilisearch in a few ways to run meilisearch in a cluster by externalizing the task queue.
- Add the content type to the webhook (#4450) @irevoire
Fixes 🐞
- Make update file deletion atomic (#4435) @irevoire
- Do not omit vectors when importing a dump (#4446) @dureuill
- Put a bound on OpenAI timeout (#4459) @dureuill
Misc
Contributors
Assets 8
v1.7.0-rc.2 🐇
What's Changed since previous RC
Facet indexing
The facet incremental indexing has been optimized, and the threshold used to choose between bulk and incremental indexing has been changed to fit users' needs:
- Enhance facet incremental by @ManyTheFish in #4433
- Divide threshold by ten by @ManyTheFish in #4462
Semantic search
- Add GPU analytics by @dureuill in #4443
- Put a bound on OpenAI timeout by @dureuill in #4459
- Do not omit vectors when importing a dump by @dureuill in #4446
Benchmarks
- Add subcommand to run benchmarks by @dureuill in #4445
- Replace logging timer by spans by @dureuill in #4458
HA
Fixes
Contributors
Assets 7
v1.7.0-rc.1 🐇
What's Changed since previous RC
- Make several indexing optimizations by @ManyTheFish in #4350
- Update charabia by @ManyTheFish in #4365
- Implement the experimental log mode cli flag and log level updates at runtime by @irevoire in #4410
- Output logs to stderr by @irevoire in #4418
Contributors
Assets 7
v1.6.2 🦊
v1.7.0-rc.0 🐇
Meilisearch v1.7.0 mostly focuses on improving v1.6.0 features, indexing speed and hybrid search. GPU computing is now supported.
New features and improvements 🔥
Improve AI with Meilisearch (experimental feature)
🗣️ AI work is still experimental, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
To use it, you need to enable vectorSearch through the /experimental-features route.
💡 More documentation about AI search with Meilisearch here.
Add new OpenAI embedding models & ability to override their models dimensions
When using OpenAi as source in your embedders index settings (an example here), you can now specify two new models:
text-embedding-3-smallwith a default dimension of 1536.text-embedding-3-largewith a default dimension of 3072.
The new models:
- are cheaper
- produce more relevant results in standardized tests
- allow to set up the dimensions of the embeddings to control the trade-off between accuracy and performance (including storage)
It means that it is now possible to pass the dimensions field when using the OpenAi source. This was previously only available for the userProvided source in the previous releases.
There are some rules, though, which we detail with these examples:
"embedders": {
"large": {
"source": "openAi",
"model": "text-embedding-3-large",
"dimensions": 512 // must be >0, must be <= 3072 for "text-embedding-3-large"
},
"small": {
"source": "openAi",
"model": "text-embedding-3-small",
"dimensions": 1024 // must be >0, must be <= 1536 for "text-embedding-3-small"
},
"legacy": {
"source": "openAi",
"model": "text-embedding-ada-002",
"dimensions": 1536 // must =1536 for "text-embedding-ada-002"
},
"omitted_dimensions": { // uses the default dimension
"source": "openAi",
"model": "text-embedding-ada-002",
}
}Add GPU support to compute embeddings
Enabling the CUDA feature allows using an available GPU to compute embeddings with a huggingFace embedder.
On an AWS Graviton 2, this yields a x3 - x5 improvement on indexing time.
👇 How to enable GPU support through CUDA for HuggingFace embedding generation:
Prerequisites
- Linux distribution with a compatible CUDA version
- NVidia GPU with CUDA support
- A recent Rust compiler to compile Meilisearch from source
Steps
- Follow the guide to install the CUDA dependencies
- Clone Meilisearch:
git clone https://github.com/meilisearch/meilisearch.git - Compile Meilisearch with the
cudafeature:cargo build --release --package meilisearch --features cuda - In the freshly compiled Meilisearch, enable the vector store experimental feature:
❯ curl \
-X PATCH 'http://localhost:7700/experimental-features/' \
-H 'Content-Type: application/json' \
--data-binary '{ "vectorStore": true }'- Add an HuggingFace embedder to the settings:
curl \
-X PATCH 'http://localhost:7700/indexes/your_index/settings/embedders' \
-H 'Content-Type: application/json' --data-binary \
'{ "default": { "source": "huggingFace" } }'Improve indexing speed & reduce memory crashes
- Auto-batch the task deletions to reduce indexing time (#4316) @irevoire
- Improve indexing speed for vector store (makes the Hybrid search experimental feature indexing time more than 10 times faster) (#4332) @Kerollmops @irevoire
- Reduce memory usage, so reduce the memory crashes, by capping the maximum memory of the grenad sorters (#4388) @Kerollmops
Stabilize scoreDetails feature
In v1.3.0, we introduced the experimental feature scoreDetails. We got enough positive feedback on the feature, and we now stabilize it, making this feature enabled by default.
View detailed scores per ranking rule for each document with the showRankingScoreDetails search parameter:
curl \
-X POST 'http://localhost:7700/indexes/movies/search' \
-H 'Content-Type: application/json' \
--data-binary '{ "q": "Batman Returns", "showRankingScoreDetails": true }'When showRankingScoreDetails is set to true, returned documents include a _rankingScoreDetails field. This field contains score values for each ranking rule.
"_rankingScoreDetails": {
"words": {
"order": 0,
"matchingWords": 1,
"maxMatchingWords": 1,
"score": 1.0
},
"typo": {
"order": 1,
"typoCount": 0,
"maxTypoCount": 1,
"score": 1.0
},
"proximity": {
"order": 2,
"score": 1.0
},
"attribute": {
"order": 3,
"attributes_ranking_order": 0.8,
"attributes_query_word_order": 0.6363636363636364,
"score": 0.7272727272727273
},
"exactness": {
"order": 4,
"matchType": "noExactMatch",
"matchingWords": 0,
"maxMatchingWords": 1,
"score": 0.3333333333333333
}
}Logs improvements
We made some changes regarding our logs to help with debugging and bug reporting.
Log format change
The default log format evolved slightly from this:
[2024-02-06T14:54:11Z INFO actix_server::builder] starting 10 workersTo this:
2024-02-06T13:58:14.710803Z INFO actix_server::builder: 200: starting 10 workersExperimental: new routes to manage logs
This new version of Meilisearch introduces 3 new experimental routes
POST /logs/stream: streams the log happening in real-time. Requires two parameters:target: selects what logs you’re interested in. It takes the form ofcode_part=log_level. For example,index_scheduler=infomode: selects in what format of log you want. Two options are available:human(basic logs) orprofile(a way more complex trace)
DELETE /logs/stream: stops the listener from the meilisearch perspective. Does not require any parameters.
💡 More information in the New experimental routes section of this file.
POST /logs/stream route:
- You can have only one listener at a time
- Listening to the route doesn’t seem to work with
xhorhttpiefor the moment - When killing the listener, it may stay installed on Meilisearch for some time, and you will need to call the
DELETE /logs/streamroute to get rid of it.
🗣️ This feature is experimental, and we need your help to improve it! Share your thoughts and feedback on this GitHub discussion.
Other improvements
- Related to the Prometheus experimental feature: add job variable to Grafana dashboard (#4330) @capJavert
Misc
- Dependencies upgrade
- Bump rustls-webpki from 0.101.3 to 0.101.7 (#4263)
- Bump h2 from 0.3.20 to 0.3.24 (#4345)
- Update the dependencies (#4332) @Kerollmops
- CIs and tests
- Documentation
- Add Setting API reminder in issue template (#4325) @ManyTheFish
- Update README (#4319) @codesmith-emmy
- Misc
❤️ Thanks again to our external contributors: