BlockMax WAND

[amourao]

What's being changed:

• New inverted format that:
  • Writes docs ids and term frequencies in blocks
  • Compresses doc ids and frequencies
  • Writes tombstones to a separate area of the file
• Use BlockMax WAND to make search faster

Review checklist

• Documentation has been updated, if necessary. Link to changed documentation:
• Chaos pipeline run or not necessary. Link to pipeline:
• All new code is covered by tests where it is reasonable.
• Performance tests have been run or not necessary.

Cross-functional impact

• This change requires public documentation (weaviate-io) to be updated. Check the box to automatically create a corresponding issue.
  • Score values will change slightly when searching on multiple properties, due to using individual prop statistics instead of global statistics.
• Does it require a change in the client libraries? If yes, please check the boxes for the affected client libraries.
  • Python (weaviate-python-client)
  • JavaScript/TypeScript (typescript-client)
  • Go (weaviate-go-client)
  • Java (java-client)

[amourao]

Quality Gate failed

Failed conditions 14.3% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

The duplication here mostly comes from tests:

• compactor_inverted_integration_test.go shares code with compactor_map_integration_test.go, but making it reusable would make it less readable;
• bm25f_block_test.go shares code with bm25f_test.go. Here it is even harder, as the index setup is the same, but the scores change slightly (by design) across tests.

[sonarcloud]

Quality Gate failed

Failed conditions
14.4% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud