In the previous article, we covered the core concepts of observability and the open source tools for metrics, logs, and traces. In this article, we will take a closer look at the agents and collectors responsible for this process, including Beats Family, Elastic Agent, Fluent Family, OpenTelemetry Collector, Grafana Alloy, and Grafana Beyla.
Choosing the right tools to collect, process, and export telemetry data can be confusing and costly. The broad range of telemetry you need to process, and the collectors you choose, can vary widely depending on your observability goals. In addition, you face the challenge of addressing the constantly evolving needs of your observability strategy. For example, you may initially need only application observability, but then discover that you must add infrastructure observability. Many organizations manage and configure multiple collectors to address these challenges, thereby introducing additional complexity and potential errors into their observability strategy.
Generally, the Beats family is an open-source lightweight data shippers that you install as agents on your servers to send operational data to Elasticsearch. Beats can send data directly to Elasticsearch or via Logstash, where you can further process and enhance the data (image). The Beats family consists of Filebeat, Metricbeat, Packetbeat, Winlogbeat, Auditbeat, Journalbeat, Heartbeat, and Functionbeat. Each beat ships a different type of information; for example, Winlogbeat ships Windows event logs, Metricbeat ships host metrics, and so forth. Filebeat is designed to ship log files. Filebeat helps keep things simple by offering a lightweight (low-memory footprint) way to forward and centralize logs and files, making SSH unnecessary when you have many servers, virtual machines, and containers generating logs.
Elastic Agent
Elastic Agent is the next-generation agent from Elastic, designed to replace individual Beats by providing a single, unified agent that collects logs, metrics, and security data from your systems. Unlike the Beats family, where you install a separate agent for each data type, Elastic Agent handles all of these through a single installation and is managed centrally via the Fleet application in Kibana, making policy management and integration configuration significantly simpler across your entire infrastructure.
Fluent Bit and Fluentd are both open-source tools developed for log collection and forwarding. Fluent Bit is written in C, has a ~450KB memory footprint, and has zero dependencies, making it the lightweight choice for edge nodes or Kubernetes DaemonSets. Fluentd, written in C and Ruby and with ~40 MB of memory usage, is better suited for complex aggregation and routing scenarios thanks to its 650+ plugin ecosystem. Both support a wide range of backends, including Grafana Loki, Kafka, and Elasticsearch, making them a good fit for vendor-neutral log collection outside the Elastic stack.
Grafana Alloy
Grafana Alloy is an open-source telemetry collector that simplifies the way you gather and send observability data. It’s an OpenTelemetry Collector distribution with built-in Prometheus pipelines and native support for Loki, Pyroscope, and other observability backends. Alloy collects metrics, logs, traces, and profiles in one unified solution. Instead of running separate collectors for each signal type, you configure a single tool that handles all your telemetry needs.
Grafana Beyla
Grafana Beyla is an eBPF-based application auto-instrumentation tool to easily get started with Application Observability. Beyla uses eBPF to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. All data capture occurs without modifying the application code or configuration.
OpenTelemetry Collector
OpenTelemetry Collector, or OTel Collector, is a crucial part of the OpenTelemetry ecosystem, an observability framework for cloud-native software. It's an open-source service that ingests, processes, and exports telemetry data. This telemetry data can include traces, metrics, and logs from different sources.
The primary function of the OpenTelemetry Collector is to receive telemetry data from applications or services, process it, and send it to the backend specified by the user. It's vendor-agnostic, meaning it can receive data from various sources and export it to multiple destinations, including open-source and commercial backends.
Choosing the right agent or collector depends heavily on your existing stack and observability requirements. The comparison table below correlates each tool with its optimal use case.
|
Tool |
Best Use Case |
|
Beats family |
If you are already using the Elastic ecosystem, it is the most straightforward option with native Elasticsearch integration. |
|
Elastic Agent |
To simplify your agent management within the Elastic ecosystem, replace multiple Beats with a single unified agent. |
|
Fluent family |
If you need vendor-neutral log collection with flexibility in output destinations. |
|
Grafana Alloy |
If your stack is built around Grafana, it provides a unified solution for all telemetry data, with native support for Prometheus and Loki. |
|
Grafana Beyla |
If you need application-level observability without touching your code, use eBPF to automatically capture traces and metrics. |
|
OpenTelemetry Collector |
If you need a fully vendor-neutral pipeline that works with any backend and adheres to CNCF standards, it is the most flexible and future-proof choice. |
In the next article, we will move from theory to practice, building on the observability tools, agent, and collector alternatives we covered in this series. We will walk through real-world deployment scenarios to show how they work together in a production observability pipeline.