Learn more about how Elasticsearch autoscales to address ingestion load.

Ingest autoscaling overview

Metrics

Ingestion load

Memory

Scaling the cluster

Conclusion

Elasticsearch ingest autoscaling

Ingest autoscaling in Elasticsearch

We explore how serverless Elasticsearch facilitates searches using data stored in a blob store while maintaining the same visibility semantics as stateful Elasticsearch. We discuss the challenges encountered during implementation and share strategies for balancing costs and complexity.

Refreshes in stateful Elasticsearch

Refreshes in serverless Elasticsearch

Refresh cost optimizations

Batched compound commits

New refresh lifecycle

Considerations and tradeoffs

Conclusions

Acknowledgments

Elasticsearch refresh: How we optimized refreshes in Serverless

How we optimized refresh costs in Elasticsearch Serverless

We discuss the data durability guarantees in stateless including how we fence new writes and deletes with a safety check which prevents stale nodes from acknowledging new writes or deletes

Durability promise and translog

Not knowing is the worst - the problem(s)

Primary terms - an increasing number to the rescue

Coordination term and node-left generation

Object store read after write semantics

The safety check

Fencing writes and deletes from stale nodes

Data safety in a stateless world

Author

Henning Andersen

Articles

Ingest autoscaling in Elasticsearch

How we optimized refresh costs in Elasticsearch Serverless

Data safety in a stateless world