Learn how to instrument a Python application that uses OpenAI, analyze its performance & cost and integrate large language models (LLMs).

Instrumenting the application with Elastic APM

Analyzing the APM data — Elasticsearch vs. OpenAI performance

Analyzing cost and response time

Latency and failed transaction correlation

ChatGPT prints its output word by word due to limitations in the inference model, which requires considering the last token for computing the next token and limits parallelization. Since every token is processed individually, a token can be sent to the client while the computation for the next token is running.

Why does ChatGPT show the answer one word at a time?

There is a correlation between the number of tokens in a response and the duration of the request: the more tokens returned, the longer the duration. To improve the responsiveness of your application using OpenAI models, consider telling the model to keep the response short.

How to improve the responsiveness of an application that uses OpenAI models?

How to Instrument a Python Application with Elastic APM

Initialize the APM Agent & apply default instrumentation.

Extra work to get more interesting data: add the user’s query to the APM metadata as well as the number of tokens and approximate cost to our APM transaction.

ChatGPT and Elasticsearch: Instrumentation, Performance & Cost

ChatGPT and Elasticsearch: APM instrumentation, performance, and cost analysis

By providing ChatGPT more context and using Elasticsearch's facets & filters, you can refine the search and lower ChatGPT costs. Here's how.

Comparing ChatGPT and Elasticsearch results

Leveraging aggregations, facets, and filtering in Elasticsearch to enhance ChatGPT

Loading multiple results in parallel

Working around the max token limit of ChatGPT: Answering a question from a set of answers

Citations for your ChatGPT results

Searching proprietary data

ChatGPT and Elasticsearch can work together by leveraging Elasticsearch's search capabilities and ChatGPT's contextual understanding. This can help manage proprietary data more effectively.

How can ChatGPT and Elasticsearch work together?

Elasticsearch's faceting and filtering framework allow users to refine their search and reduce the cost of engaging with ChatGPT.

Author

Luca Wintergerst

Articles

ChatGPT and Elasticsearch: APM instrumentation, performance, and cost analysis

ChatGPT and Elasticsearch: enhance user experience with faceting, filtering, and more context