Discover how Elasticsearch 8.11 improved its indexing buffer, resulting in less segment merging and faster ingestion.

How indexing buffer works in Elasticsearch 8.10 and earlier

How Elasticsearch 8.11 improved its indexing buffer

Change 1: flush a single segment at a time

Change 2: flush shards in a round-robin fashion

Conclusion

Elasticsearch indexing buffer: Less merging & fast ingestion in 8.11

Less merging and faster ingestion in Elasticsearch 8.11

Lucene 9.9 brings major speedups to query evaluation. Here are the performance improvements observed in nightly benchmarks & optimization resources.

Key speedup benchmarks in Lucene

Lucene optimization resources

Lucene 9.9: Optimizations, performance & benchmarks insights

Apache Lucene 9.9, the fastest Lucene release ever

Learn about MAXSCORE, block-max MAXSCORE & WAND. Improve the MAXSCORE algorithm to evaluate disjunctive queries more like a conjunctive query.

Introduction to WAND and MAXSCORE

Block-max MAXSCORE & MAXSCORE in Lucene

MAXSCORE usage example

Improving MAXSCORE to intersect terms

MAXSCORE sorts terms by increasing maximum impact score and partitions them into two groups: essential terms and non-essential terms. Non-essential terms are terms with low maximum impact scores whose sum of maximum scores is less than the minimum competitive score. Essential terms are all other terms. Essential terms are used to find candidate matches while non-essential terms are only used to compute the score of a candidate.

How does the MAXSCORE algorithm work?

WAND typically evaluates fewer hits than MAXSCORE but with a higher per-hit overhead. This makes MAXSCORE generally perform better on high k values or with many terms - when skipping hits is hard, and WAND performs better otherwise."

How do block-max MAXSCORE performance characteristics compare to WAND?

MAXSCORE & block-max MAXSCORE: Improving the algorithm

MAXSCORE & block-max MAXSCORE: More skipping with block-max MAXSCORE

In this blog, you'll learn how vector search has been integrated into Elasticsearch and the trade-offs that we made.

Vector search is integrated in Elasticsearch through Apache Lucene

Cons

Merges need to recompute HNSW graphs

Searches need to combine results from multiple segments

RAM needs to scale with the size of the data set to retain optimal performance

Pros

Data sets can scale beyond the total RAM size

Lock-free search

Support for incremental changes

Visibility consistency with other data structures

Incremental snapshots

Filtering and hybrid support

Compatibility with other features

Looking ahead: Separation of indexing and search

The main cons of taking advantage of Apache Lucene for vector search come from the fact that Lucene ties vectors to segments. These cons include:  1) Merges need to recompute HNSW graphs. 2) Searches need to combine results from multiple segments. 3) RAM needs to scale with the size of the data set to retain optimal performance. However, tying vectors to segments is also what enables major features such as efficient pre-filtering, efficient hybrid search, and visibility consistency.

What are the cons of using Apache Lucene for vector search?

The pros of taking advantage of Apache Lucene for vector search is that it enables features such as efficient pre-filtering, efficient hybrid search, visibility consistency, among others.

What are the pros of using Apache Lucene for vector search?

Vector search is integrated into Elasticsearch through Apache Lucene.

How is vector search integrated into Elasticsearch?

Vector search in Elasticsearch: Integration & the design rationale

Vector search in Elasticsearch: The rationale behind the design

2023 has been another big year for Apache Lucene, this blog reviews major milestones of 2023

Author

Adrien Grand

Articles

Less merging and faster ingestion in Elasticsearch 8.11

Apache Lucene 9.9, the fastest Lucene release ever

MAXSCORE & block-max MAXSCORE: More skipping with block-max MAXSCORE

Vector search in Elasticsearch: The rationale behind the design

What happened in Lucene land in 2023?