--- title: Collecting Kubernetes Application Logs with Filebeat and the Elastic Stack (ECK) description: Ship Kubernetes logs with Filebeat into Elasticsearch, and visualize them in Kibana - all managed through the ECK Operator. A hands-on guide with opt-in collection and field filtering --- [Hepapi Blog - hepapi.com](https://hepapi.com/blog) # [Collecting Kubernetes Application Logs with Filebeat and the Elastic Stack (ECK)](https://hepapi.com/blog/collecting-kubernetes-application-logs-with-filebeat-and-the-elastic-stack-eck) Written by [Ersin Sarı](https://hepapi.com/blog/author/ersin-sarı) | Oct 5, 2026, 9:34:12 AM In the previous article in this series, we built a complete log pipeline using Fluent Bit as the collector and Grafana Loki as the storage backend. In this article, we will look at another common approach: shipping Kubernetes logs with Filebeat to Elasticsearch and visualizing them in Kibana, all deployed through the Elastic Cloud on Kubernetes (ECK) Operator. Compared to the Fluent Bit and Loki setup from the previous article, this stack trades Loki's label-based, low-cardinality storage model for Elasticsearch's full-text indexing and richer query capabilities - at the cost of higher resource usage and operational overhead. Which one fits your environment best usually comes down to query patterns, retention requirements, and how much infrastructure you're willing to run. In practice, reach for Elasticsearch and ECK when you need powerful full-text search, complex ad-hoc queries, or correlation across large volumes of log data. If your priority is lightweight, cost-efficient log aggregation with simple label-based filtering, Loki remains the better fit.   #### What We Will Build We will deploy the ECK operator on a local Minikube cluster, then use the eck-stack Helm chart to spin up a quickstart Elasticsearch cluster and Kibana instance. Filebeat will run as a DaemonSet on every node, tailing container logs, enriching them with Kubernetes metadata, and shipping them into Elasticsearch. We will also configure Filebeat to collect logs only from pods matching specific labels, keep unrelated workloads out of our indices, and show how to drop unwanted fields from log records before they are indexed. Finally, we will use Kibana's Discover view to confirm that logs from our sample application are flowing end to end.   #### Demo **Deploy Minikube** ``` minikube start ```   #### Install the ECK Operator The ECK operator is a Kubernetes operator that manages the full lifecycle of Elasticsearch, Kibana, and other Elastic Stack components as native Kubernetes resources. We install it once cluster-wide, and it handles provisioning and reconciling Elastic resources for us. ``` # Add the Elastic Helm Repository helm repo add elastic https://helm.elastic.co && helm repo update ``` ``` # Install the ECK Operator cluster-wide helm upgrade --install elastic-operator elastic/eck-operator -n elastic-system --create-namespace ``` #### Install the ECK Stack (Elasticsearch + Kibana + Filebeat) With the operator running, we can now request an Elasticsearch cluster and a Kibana instance simply by installing the eck-stack chart. This chart wraps the Elasticsearch, Kibana, and Filebeat custom resources that the operator watches and reconciles. **eck-stack-values.yaml** ``` eck-elasticsearch: fullnameOverride: elasticsearch version: 9.3.1 nodeSets: - name: default count: 1 config: node.store.allow_mmap: false volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi eck-kibana: fullnameOverride: kibana version: 9.3.1 spec: count: 1 elasticsearchRef: name: elasticsearch http: tls: selfSignedCertificate: disabled: true service: spec: type: ClusterIP podTemplate: spec: containers: - name: kibana env: - name: NODE_OPTIONS value: "--max-old-space-size=2048" resources: requests: memory: 1Gi cpu: 0.5 limits: memory: 2.5Gi cpu: 2 eck-beats: enabled: true fullnameOverride: filebeat spec: type: filebeat version: 9.3.1 elasticsearchRef: name: elasticsearch kibanaRef: name: kibana config: logging.level: debug logging.selectors: ["elasticsearch"] filebeat.autodiscover.providers: - type: kubernetes node: ${NODE_NAME} hints.enabled: true hints.default_config.enabled: false processors: - add_cloud_metadata: {} - add_host_metadata: {} # - drop_fields: # fields: # - "host.os" # - "host.mac" # - "host.architecture" # - "host.id" # - "kubernetes.node.labels" # - "kubernetes.namespace_labels" # - "kubernetes.namespace_uid" # - "log.file" # - "log.offset" # - "input" # - "ecs" ignore_missing: true daemonSet: podTemplate: spec: serviceAccountName: filebeat automountServiceAccountToken: true terminationGracePeriodSeconds: 30 dnsPolicy: ClusterFirstWithHostNet containers: - name: filebeat securityContext: runAsUser: 0 privileged: false env: - name: NODE_NAME valueFrom: fieldRef: fieldPath: spec.nodeName volumeMounts: - name: varlogcontainers mountPath: /var/log/containers readOnly: true - name: varlogpods mountPath: /var/log/pods readOnly: true - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true volumes: - name: varlogcontainers hostPath: path: /var/log/containers - name: varlogpods hostPath: path: /var/log/pods - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers extraObjects: - | apiVersion: v1 kind: ServiceAccount metadata: name: filebeat namespace: - | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: filebeat- rules: - apiGroups: [""] resources: ["namespaces", "pods", "nodes"] verbs: ["get", "list", "watch"] - apiGroups: ["apps"] resources: ["replicasets"] verbs: ["get", "list", "watch"] - apiGroups: ["batch"] resources: ["jobs"] verbs: ["get", "list", "watch"] - | apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: filebeat- subjects: - kind: ServiceAccount name: filebeat namespace: roleRef: kind: ClusterRole name: filebeat- apiGroup: rbac.authorization.k8s.io ``` This values.yaml deploys the full Elastic Stack on Kubernetes through ECK: a single-node Elasticsearch cluster with persistent storage handles indexing and search, Kibana connects to it over a TLS-free ClusterIP service for easy local access, and Filebeat runs as a DaemonSet on every node to tail container logs and ship them into Elasticsearch. Since all three components reference each other natively through ECK (elasticsearchRef, kibanaRef), authentication, service discovery, and TLS are wired up automatically, letting us focus on what to collect rather than how to connect the pieces together. The filebeat.autodiscover.providers block uses Kubernetes autodiscover to automatically detect pods across the cluster. The key part is the combination of hints.enabled: true and hints.default\_config.enabled: false: this means Filebeat does not collect logs from every pod by default. It only collects from pods that are explicitly annotated with co.elastic.logs/enabled: "true". In other words, instead of "collect everything automatically," the logic here is "only collect from pods that opt in." ``` helm repo add elastic https://helm.elastic.co && helm repo update ``` ``` helm upgrade --install ek-stack elastic/eck-stack -n elastic-stack --create-namespace -f eck-stack-values.yaml ``` ``` kubectl get pods -n elastic-stack -w ``` **Retrieve the Elastic Superuser Password** ECK automatically creates a Kubernetes secret containing credentials for the built-in Elastic superuser. ``` kubectl get secret elasticsearch-es-elastic-user -o go-template='' ```   **Port-Forward the Kibana Service** Navigate to https://localhost:5601 in your browser. ``` kubectl port-forward svc/kibana-kb-http 5601:5601 -n elastic-stack ```   #### Deploy Sample Log Pod To verify the pipeline end-to-end, we deploy the same kind of sample application used in the previous article: a small container that continuously emits JSON-formatted logs alternating between info and warn levels. **log-generator.yaml** ``` apiVersion: v1 kind: Pod metadata: name: log-generator namespace: app labels: app: log-generator annotations: co.elastic.logs/enabled: "true" spec: containers: - name: log-generator image: busybox command: ["sh", "-c"] args: - | while true; do echo "{\"level\":\"info\",\"message\":\"hello from log-generator\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" sleep 3 echo "{\"level\":\"warn\",\"message\":\"something might be wrong\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}" sleep 3 done ``` Note the co.elastic.logs/enabled: "true" annotation on the pod. Since we configured Filebeat's autodiscover with hints.default\_config.enabled: false, this annotation is what tells Filebeat to actually collect logs from this pod. Without it, log-generator would run normally, but its logs would simply be ignored by the autodiscover provider. ``` kubectl create ns app kubectl apply -f log-generator.yaml -n app kubectl logs -f log-generator -n app ``` #### Verify Logs in Kibana Once the pod is running, open Kibana and go to **Stack Management \> Index Management** to confirm that the filebeat-\* datastream has been created and is receiving documents. Then head to **Discover**, create a data view for filebeat-\* if one doesn't already exist, and filter by kubernetes.namespace or kubernetes.pod.name to confirm the log-generator's messages are showing up correctly with their level and message fields parsed out. #### Dropping Unwanted Fields You'll notice the drop\_fields processor in eck-stack-values.yaml is commented out by default. Before applying it, take a look at a log document in Kibana's Discover view and you'll see extra fields like host.os, host.mac, kubernetes.node.labels, log.offset, and ecs cluttering every record - none of which add real value for our use case. Uncomment the drop\_fields block: ``` processors: - add_cloud_metadata: {} - add_host_metadata: {} - drop_fields: fields: - "host.os" - "host.mac" - "host.architecture" - "host.id" - "kubernetes.node.labels" - "kubernetes.namespace_labels" - "kubernetes.namespace_uid" - "log.file" - "log.offset" - "input" - "ecs" ``` Then apply the change: ``` helm upgrade --install ek-stack elastic/eck-stack -n elastic-stack --create-namespace -f eck-stack-values.yaml --force-conflicts ``` Once the Filebeat pods roll out with the updated configuration, go back to Discover and inspect a new log document. The fields listed above will no longer be present, leaving you with cleaner, smaller documents that are easier to read and cheaper to store. #### Conclusion In this article, we deployed the ECK operator to manage Elasticsearch and Kibana declaratively on Kubernetes, installed Filebeat as a DaemonSet to tail container logs, and restricted collection to a single namespace to keep our indices focused and cost-effective. We then verified the pipeline by generating sample logs and inspecting them in Kibana's Discover view. In this demo, Elasticsearch was deployed as a single-node cluster inside the same Kubernetes cluster for simplicity and local testing. This setup is not suitable for production. For production workloads, Elasticsearch should run as a multi-node cluster with dedicated master, data, and ingest roles, sized properly for your log volume and retention needs. [View full post](https://hepapi.com/blog/collecting-kubernetes-application-logs-with-filebeat-and-the-elastic-stack-eck) ```json { "@context" : "http://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Ersin Sarı" }, "dateModified" : "2026-10-05T09:34:12.701Z", "datePublished" : "2026-10-05T09:34:12Z", "headline" : "Collecting Kubernetes Application Logs with Filebeat and the Elastic Stack (ECK)", "image" : { "@type" : "ImageObject", "height" : 1080, "url" : "https://8810897.fs1.hubspotusercontent-na1.net/hubfs/8810897/kubernetes-application-logs-filebeat-elastic-stack-eck.png", "width" : 1920 }, "mainEntityOfPage" : "https://hepapi.com/blog/collecting-kubernetes-application-logs-with-filebeat-and-the-elastic-stack-eck", "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "height" : 60, "url" : "/hs/hsstatic/content_shared_assets/static-1.4092/img/default-amp-logo.png", "width" : 60 }, "name" : "Hepapi Blog" } } ```