APM Server has been the de facto service for ingesting data from Elastic APM agents and OTel agents. In this blog post, we will walk through our journey of redesigning the APM Server product to scale and evolve into a more generic ingest component for Elastic Observability while also improving the reliability and maintainability compared to the traditional APM Server.

APM Server primer

APM Server as a managed product

Multi Tenant APM Server

Break down the monolith

Ingestion Service

Aggregation Service

Index Service

Glue it back together!

Using Kafka

Observing the system

Ready to level up your observability stack?

Ingesting data with Managed Intake Service

Architecting the next-generation of Managed Intake Service

The Open Crawler is now in beta. This latest version 0.2 update also comes with several new features.

What's new?

Features

Configuration changes

Showcase 1: crawl rules

Crawling only specific endpoints

Crawling everything except a specific endpoint

Showcase 2: extraction rules

Getting the CSS selector

Configuring the extraction rules

Showcase 3: combining it all with Semantic Text

Comparing with Elastic Crawler

Next steps

Open Crawler: Version 0.2 of Elastic's Open Crawler is now in beta

Open Crawler now in beta

Discover the different ways you can ingest data in Elasticsearch. Explore data ingestion tools like Logstash, Client APIs and Elastic Native Connectors + the Elastic Connector Framework.

Data ingestion tools

1. Logstash

Overview

Key features and strengths

Best for

2. Elasticsearch clients

3. Native Connectors and the Connector Framework

Summary

Elasticsearch data ingestion - What's the best tool for the job?

Elasticsearch data ingestion - What's the best data ingestion tool for the job?

The Open Crawler lets users crawl web content and index it into Elasticsearch from wherever they like. Learn about it & how to use it here.

Background

What has changed?

How does the Open Crawler compare to the Elastic Crawler?

Requests (n)

So why is the Open Crawler so much faster?

1. The Open Crawler makes significantly fewer indexing requests.

2. Elastic Crawler also performs many search requests, further slowing down performance

3. Open Crawler is simply doing less with crawl results

How do I use the Open Crawler?

1. Build the Docker image and run it

2. Configure the crawler

3. Validate the domain

4. Crawl!

5. Check the content

Feature comparison breakdown

What's next?

Open Crawler allows users to crawl web content and index it into Elasticsearch from wherever they like.

What is Elastic's Open Crawler?

Open Crawler: Elastic's Open Crawler released for tech-preview

Open Crawler released for tech-preview

Learn how to manage connectors and use the Connector API to ingest data from MongoDB or other sources into Elasticsearch Serverless.

Prerequisites

Elasticsearch Serverless

Our product catalog

Create a MongoDB connector

Triggering an on-demand sync

Keeping your ES index in sync with source

Sync Rules - Index only what you want

Generate vector embeddings at ingest time

Monitoring your connectors

Tips on automation

Appendix

Spin up MongoDB instance with Docker

The Elastic Connector API is used to fully manage Elastic connectors directly from your command-line interface. This is especially useful for specific workflows, as it can automate connector management, monitoring and testing, removing the need to go back and forth between a terminal and the Kibana UI.

What is the Elastic Connector API used for?

Elastic Connector APIs: Ingesting data into Elasticsearch Serverless

How to use the Connector API to ingest data into Elasticsearch Serverless

Learn about full sync and incremental sync for connectors. Discover how incremental sync can boost the performance of Elastic connectors.

Elastic Connectors overview

Connectors content synchronization jobs

1. Full syncs

2. Incremental syncs

Potential optimizations for incremental sync

Fetch fewer documents from 3rd-party systems

Skip download of content of files that haven't changed since previous sync

Skip ingestion of non-modified documents into Elasticsearch

Measuring incremental sync performance

Performance tests

Setup #1 - Google Drive Connector

Setup #2: GitHub connector

Key takeaways

Elastic Connectors are a type of Elastic integrations that sync data from an original data source to an Elasticsearch index. Connectors enable you to create searchable, read-only replicas of your data sources.

What are Elastic Connectors?

Elastic Connectors: Performance impact of incremental syncs

Learn through an example how to leverage the Elastic Web Crawler to crawl a website, ingest its content & prepare the data for search.

What is the Elastic Web Crawler

Website discovery & web crawler creation

A first crawl

Define crawl rules

Enrich the data extracted by the web crawler

Extract data from custom field

Ingest pipeline to transform data

Enable semantic search with ML Inference pipeline

Test search query

Preparing the web crawler for production deployment

Scheduling

Protected website

Conclusion

The Elastic web crawler is an out-of-the-box tool that enables users to crawl a website and ingest its content into Elasticsearch.

What is the Elastic Web Crawler?

The Elastic web crawler begins each crawl by visiting the entry point URLs. From there, the crawler fetches the content from this webpage and extracts it. The HTML document is transformed into a search document and indexed. Then it follows any links to which that page points to, respecting the rules defined by the user. The new URLs are visited using the same methodology until the crawler has indexed the accessible web pages.

How does the Elastic Web Crawler work?

Web crawler: Add search to a website with the Elastic Web Crawler

Elastic web crawler: Add search to your website

Discover the history of extracting text from binary documents at Elastic and learn about our open-code solution: the Data Extraction Service.

Background: Handling binary documents at Elastic

Relevance

Scale

Speed

Introducing the Data Extraction Service

What’s next?

What’s the long-term dream?

It is a dedicated, standalone server that is capable of taking your binary documents as input (via a REST API) and responding with their plain-text contents, as well as some minimal metadata.

What is the Data Extraction Service at Elastic?

Elasticsearch binary document ingestion & Data Extraction Service

Elasticsearch binary document ingestion: Evolution & the Data Extraction Service

Learn how to create custom connectors for Elasticsearch to simplify your data ingestion process.

Framework to create custom connectors

What you need to know before you use connectors framework

The framework is written in async-python

Why did we choose async python?

Any other pre-requisites?

Using connectors framework to create a custom connector

Outline of steps

What you need to know before writing your custom connector

What you need to know about connectors protocol

Where custom connectors are hosted

Example: Google Drive connector using connectors framework

How we approached writing the official Elasticsearch Google Drive connector

Start by implementing the methods expected by the BaseDataSource class

The first consideration is: How to “talk” to Google Drive to get data?

The next concern is the connector authentication

Next, let’s implement a simple ping method

Async iterator to return files from google drive for content extraction

Let’s run the connector!

Do you want to create and contribute a new connector?

Acceptance criteria for contributing connectors

Supporting tools to test your connector

Wrap up

Elastic connectors: Creating custom connectors for Elasticsearch

How to create custom connectors for Elasticsearch

Learn about different ways you can ingest data from AWS S3 into Elastic Cloud

Options Comparison

Elastic Cloud

Elastic Serverless Forwarder

Data Flow

Set up

More Dashboards!

Ingest data from AWS S3 into Elastic Cloud using Elastic Serverless Forwarder

How to ingest data from AWS S3 into Elastic Cloud - Part 1 : Elastic Serverless Forwarder