Here's how to set up and use the Elasticsearch Relevance Engine (ESRE) with its Elastic Learned Sparse Encoder capability for log analysis and probe TLS issues.

How does the Elastic Learned Sparse Encoder work?

Setting up ESRE for log analysis and probing TLS issues

Step 0: Create an account on Elastic Cloud

Step 1: Install the Elastic Learned Sparse Encoder model

Step 2: Modify the index template

Step 3: Add a custom ingest pipeline

Step 4: Rollover the index

Step 5: Welcome to semantic search with logs

Step 6: Wiring this up to ChatGPT

Superior relevance out of the box for SREs

When the Elastic Learned Sparse Encoder is applied to a raw text, it generates a data structure. The structure’s keys represent terms and their synonyms found in the raw text. In a process called term expansion, the model adds and removes terms and synonyms based on relevance, creating a scored bag of words (BOW). At query time, the BOW of the query is matched against the BOWs of the raw text. The final rank score of the search results is the result of a formula that takes into account the scores from matching words from the BOWs of the query and the raw log text.

How to set up the Elastic Learned Sparse Encoder for log analysis

Get an account on Elastic Cloud (must be version 8.8) and install the Elastic Agent.

Set up ESRE in this environment and wire up the logs to ESRE.

Leverage Kibana to search the logs using ESRE.

Create an application that can work with Elasticsearch and ESRE.

Setting up ESRE for log analysis: probing TLS issues with ESRE

Logs: Understanding TLS errors with ESRE and generative AI

Learn how to monitor OpenAI API and GPT models using OpenTelemetry and Elastic in this blog post. Explore mothods for monitoring performance, cost, and more.

Leveraging OpenAI APIs

OpenTelemetry to the rescue

Set up Elastic Cloud account (if you already don’t have one)

Monkeying around

Elastic to the rescue

Summary

You can monitor OpenAI API using OpenTelemetry and Elastic. Learn more about the process in this guide.

Author

David Hope

Articles

Logs: Understanding TLS errors with ESRE and generative AI

Monitor OpenAI API and GPT models with OpenTelemetry and Elastic