Learn how to manage connectors and use the Connector API to ingest data from MongoDB or other sources into Elasticsearch Serverless.

Prerequisites

Elasticsearch Serverless

Our product catalog

Create a MongoDB connector

Triggering an on-demand sync

Keeping your ES index in sync with source

Sync Rules - Index only what you want

Generate vector embeddings at ingest time

Monitoring your connectors

Tips on automation

Next steps

Appendix

Spin up MongoDB instance with Docker

The Elastic Connector API is used to fully manage Elastic connectors directly from your command-line interface. This is especially useful for specific workflows, as it can automate connector management, monitoring and testing, removing the need to go back and forth between a terminal and the Kibana UI.

What is the Elastic Connector API used for?

Elastic Connector APIs: Ingesting data into Elasticsearch Serverless

How to use the Connector API to ingest data into Elasticsearch Serverless

Learn how to create custom connectors for Elasticsearch to simplify your data ingestion process.

Framework to create custom connectors

What you need to know before you use connectors framework

The framework is written in async-python

Why did we choose async python?

Any other pre-requisites?

Using connectors framework to create a custom connector

Outline of steps

What you need to know before writing your custom connector

What you need to know about connectors protocol

Where custom connectors are hosted

Example: Google Drive connector using connectors framework

How we approached writing the official Elasticsearch Google Drive connector

Start by implementing the methods expected by the BaseDataSource class

The first consideration is: How to “talk” to Google Drive to get data?

The next concern is the connector authentication

Next, let’s implement a simple ping method

Async iterator to return files from google drive for content extraction

Let’s run the connector!

Do you want to create and contribute a new connector?

Acceptance criteria for contributing connectors

Supporting tools to test your connector

Wrap up

Elastic connectors: Creating custom connectors for Elasticsearch

How to create custom connectors for Elasticsearch

Author

Jedr Blaszyk

Articles

How to use the Connector API to ingest data into Elasticsearch Serverless

How to create custom connectors for Elasticsearch