---
title: "From Agent to Storage: Collecting Kubernetes Application Logs with Fluent Bit and Loki"
description: Learn how to build a complete Kubernetes log collection pipeline using Fluent Bit and Grafana Loki. This hands-on guide covers GarageFS setup, Loki single binary mode, Fluent Bit DaemonSet configuration, and log visualization in Grafana on a local Minikube cluster.
image: https://hepapi.com/hubfs/kubernetes-application-logs-fluent-bit-loki-1.webp
---

[Skip to content](https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki#main-content)

[![hepapi-logo](https://hepapi.com/hubfs/hepapi-logo.svg)](https://hepapi.com/)

- [About Us](https://hepapi.com/about-us)
  
  Show submenu for About Us 
  
    - [Career](https://hepapi.com/career)
- [Services](https://hepapi.com/services)
  
  Show submenu for Services 
  
    - [Cloud Services](https://hepapi.com/cloud-services)
      
      Show submenu for Cloud Services 
      
          - [AWS Lambda](https://hepapi.com/services/aws-lambda)
          - [Amazon Relational Database Service](https://hepapi.com/services/aws-rds)
          - [AWS Elastic Kubernetes Service (EKS)](https://hepapi.com/aws-elastic-kubernetes-service-eks)
          - [AWS WAF](https://hepapi.com/services/aws-waf)
    - [DevOps Services](https://hepapi.com/services/devops)
    - [Enterprise AI Solutions](https://hepapi.com/services/enterprise-ai-solutions)
      
      Show submenu for Enterprise AI Solutions 
      
          - [Sandbox](https://hepapi.com/services/enterprise-ai-solutions/sandbox)
    - [Software QA Services](https://hepapi.com/services/software-qa)
      
      Show submenu for Software QA Services 
      
          - [TMS](https://hepapi.com/partnerships/tms)
    - [Application Modernization](https://hepapi.com/services/application-modernization)
- [Partnerships](https://hepapi.com/partnerships)
  
  Show submenu for Partnerships 
  
    - [AWS](https://hepapi.com/partnerships/aws)
    - [Suse Rancher](https://hepapi.com/partnerships/suse-rancher)
    - [SonarQube](https://hepapi.com/partnerships/sonarqube)
    - [Dynatrace](https://hepapi.com/partnerships/dynatrace)
    - [Testrail](https://hepapi.com/partnerships/testrail)
- Resources
  
  Show submenu for Resources 
  
    - [Knowledge Hub](https://hepapi.github.io/knowledge-hub/)
    - [Blog](https://hepapi.com/blog)

Open main navigation

Close main navigation

- [About Us](https://hepapi.com/about-us)
  
  Show submenu for About Us 
  
    - [Career](https://hepapi.com/career)
- [Services](https://hepapi.com/services)
  
  Show submenu for Services 
  
    - [Cloud Services](https://hepapi.com/cloud-services)
      
      Show submenu for Cloud Services 
      
          - [AWS Lambda](https://hepapi.com/services/aws-lambda)
          - [Amazon Relational Database Service](https://hepapi.com/services/aws-rds)
          - [AWS Elastic Kubernetes Service (EKS)](https://hepapi.com/aws-elastic-kubernetes-service-eks)
          - [AWS WAF](https://hepapi.com/services/aws-waf)
    - [DevOps Services](https://hepapi.com/services/devops)
    - [Enterprise AI Solutions](https://hepapi.com/services/enterprise-ai-solutions)
      
      Show submenu for Enterprise AI Solutions 
      
          - [Sandbox](https://hepapi.com/services/enterprise-ai-solutions/sandbox)
    - [Software QA Services](https://hepapi.com/services/software-qa)
      
      Show submenu for Software QA Services 
      
          - [TMS](https://hepapi.com/partnerships/tms)
    - [Application Modernization](https://hepapi.com/services/application-modernization)
- [Partnerships](https://hepapi.com/partnerships)
  
  Show submenu for Partnerships 
  
    - [AWS](https://hepapi.com/partnerships/aws)
    - [Suse Rancher](https://hepapi.com/partnerships/suse-rancher)
    - [SonarQube](https://hepapi.com/partnerships/sonarqube)
    - [Dynatrace](https://hepapi.com/partnerships/dynatrace)
    - [Testrail](https://hepapi.com/partnerships/testrail)
- Resources
  
  Show submenu for Resources 
  
    - [Knowledge Hub](https://hepapi.github.io/knowledge-hub/)
    - [Blog](https://hepapi.com/blog)
- [Contact us](https://hepapi.com/contact-us)

[Contact us](https://hepapi.com/contact-us)

[All posts](https://hepapi.com/blog/all)

 October 2, 2026

# From Agent to Storage: Collecting Kubernetes Application Logs with Fluent Bit and Loki

![Picture of Ersin Sarı](https://hepapi.com/hs-fs/hubfs/ersin-sari.jpeg?width=50&name=ersin-sari.jpeg)    Ersin Sarı  ·   5 minute read

In the previous articles in this series, we explored the core observability tools for metrics, logs, and traces, then took a closer look at the agents and collectors that gather telemetry data. Now it is time to put theory into practice. In this article, we will walk through collecting application logs running on Kubernetes using Fluent Bit as the log collector and Grafana Loki as the log storage backend.

 

#### What We Will Build

In this demo, we will set up a complete log collection pipeline on a local Kubernetes cluster using Minikube. Fluent Bit will run as a DaemonSet on each node to collect container logs and forward them to Loki. Loki will store log data in GarageHQ, a lightweight self-hosted S3-compatible object storage, instead of local disk. Finally, Grafana will query and visualize the logs stored in Loki.

 

#### Demo

**Start Minikube**

```
minikube start
```

 

#### GarageFS

Garage is a lightweight geo-distributed data store that implements the Amazon S3 object storage protocol. It enables applications to store large blobs such as pictures, video, images, and documents in a redundant multi-node setting.

```
helm upgrade --install garage-operator oci://ghcr.io/rajsinghtech/charts/garage-operator \
  --namespace hepapi-monitoring \
  --create-namespace --version 0.2.2

kubectl create secret generic garage-admin-token --namespace hepapi-monitoring \
  --from-literal=admin-token=$(openssl rand -hex 32)
```

 

**Install GarageFS Cluster**

```
apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageCluster
metadata:
  name: garage
  namespace: hepapi-monitoring
spec:
  replicas: 1
  zone: us-east-1
  replication:
    factor: 1
  storage:
    data:
      size: 4Gi
  network:
    rpcBindPort: 3901
    service:
      type: ClusterIP
  admin:
    enabled: true
    bindPort: 3903
    adminTokenSecretRef:
      name: garage-admin-token
      key: admin-token
```

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-11-59-02-3645-AM.png?width=1306&height=208&name=undefined-Oct-02-2026-11-59-02-3645-AM.png)

**Create Buckets for Loki Storage**

```
apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageBucket
metadata:
  name: loki-chunks
  namespace: hepapi-monitoring
spec:
  clusterRef:
    name: garage
  quotas:
    maxSize: 2Gi
---
apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageBucket
metadata:
  name: loki-ruler
  namespace: hepapi-monitoring
spec:
  clusterRef:
    name: garage
  quotas:
    maxSize: 1Gi
```

 

**Create Admin-key for All Buckets**

```
apiVersion: garage.rajsingh.info/v1alpha1
kind: GarageKey
metadata:
  name: admin-key
  namespace: hepapi-monitoring
spec:
  clusterRef:
    name: garage
  allBuckets:
    read: true
    write: true
    owner: true
```

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-11-59-15-2369-AM.png?width=1442&height=146&name=undefined-Oct-02-2026-11-59-15-2369-AM.png)

#### Install Loki

In the first article of this series, we covered Loki's deployment modes in detail. For this demo, we will use the single binary (monolithic) mode, which runs all Loki components in a single process and is well suited for local and small-scale setups. If you need a refresher on deployment modes, you can refer back to the [first article](https://hepapi.com/blog/open-source-observability-tools).

```
deploymentMode: SingleBinary

loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: "2025-06-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h
  compactor:
    compaction_interval: 10m
    working_directory: /tmp/compactor
    retention_enabled: true
    retention_delete_delay: 2h
    delete_request_store: s3
  limits_config:
    allow_structured_metadata: true
    retention_period: 48h # 2 days
  storage_config:
    tsdb_shipper:
      active_index_directory: /var/loki/index
      cache_location: /var/loki/index_cache
      cache_ttl: 24h
  storage:
    type: s3
    s3:
      endpoint: http://garage.hepapi-monitoring.svc.cluster.local:3900
      region: garage
      secretAccessKey:  #update me
      accessKeyId:  #update me
      s3ForcePathStyle: true
      insecure: true
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler

singleBinary:
  replicas: 1
  resources:
    requests:
      cpu: "500m"
      memory: "1000Mi"
    limits:
      cpu: "500m"
      memory: "1000Mi"

gateway:
  enabled: true

# Disable other deployment modes
backend:
  replicas: 0
read:
  replicas: 0
write:
  replicas: 0
distributor:
  replicas: 0
ingester:
  replicas: 0
querier:
  replicas: 0
queryFrontend:
  replicas: 0
queryScheduler:
  replicas: 0
ruler:
  replicas: 0
compactor:
  replicas: 0
indexGateway:
  replicas: 0
chunksCache:
  enabled: false
resultsCache:
  enabled: false
lokiCanary:
  enabled: false
test:
  enabled: false
```

Update the accessKeyId and secretAccessKey fields in your values.yaml with the values obtained from the commands below.

```
s3:
  endpoint: http://garage.hepapi-monitoring.svc.cluster.local:3900
  region: garage
  secretAccessKey:
  accessKeyId:
  s3ForcePathStyle: true
  insecure: true
```

```
echo "accessKeyId: $(kubectl get secret admin-key -n hepapi-monitoring -o jsonpath='{.data.access-key-id}' | base64 -d)"
echo "secretAccessKey: $(kubectl get secret admin-key -n hepapi-monitoring -o jsonpath='{.data.secret-access-key}' | base64 -d)"
```

Update the retention\_period field based on your needs. For this demo, we set it to 48h.

```
limits_config:
  allow_structured_metadata: true
  retention_period: 48h
```

```
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install loki grafana/loki --version 6.38.0 -n hepapi-monitoring --create-namespace -f loki-values.yaml
```

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-11-59-30-0910-AM.png?width=1374&height=272&name=undefined-Oct-02-2026-11-59-30-0910-AM.png)

#### Install Grafana

We will install Grafana using Helm and configure Loki as a data source so we can query and explore our application logs directly from the Grafana UI.

```
adminUser: admin
adminPassword: hepapi-monitoring
replicas: 1
persistence:
  enabled: false
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Loki
      type: loki
      uid: loki
      url: http://loki-gateway
      access: proxy
      isDefault: false
```

```
helm repo add grafana https://grafana.github.io/helm-charts
helm upgrade --install grafana grafana/grafana -n hepapi-monitoring --version 10.5.15 -f grafana-values.yaml
```

Once the Grafana pod is ready, you can access the UI by running the following command and navigating to http://localhost:3000 in your browser.

```
kubectl port-forward svc/grafana 3000:80 -n hepapi-monitoring

user: admin
password: hepapi-monitoring
```

 

#### Install Fluent Bit

With Loki up and running, the next step is to deploy Fluent Bit. We will install it using Helm as a DaemonSet, so it runs on every node in the cluster and collects logs from all running containers.

```
kind: DaemonSet
logLevel: info
metricsPort: 2020
flush: 1
config:
  service: |
    [SERVICE]
        Daemon Off
        Flush 
        Log_Level 
        Parsers_File /fluent-bit/etc/parsers.conf
        Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port 
        Health_Check On

  inputs: |
    [INPUT]
        Name tail
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        parser docker_no_time
        Tag kube.*
        Mem_Buf_Limit 5MB
        Skip_Long_Lines On

  filters: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Merge_Log On
        Keep_Log Off
        K8S-Logging.Parser On
        K8S-Logging.Exclude On

    [FILTER]
        Name grep
        Match kube.*
        Regex $kubernetes['namespace_name'] app

  outputs: |
    [OUTPUT]
        name  loki
        match kube.*
        host  loki-gateway
        port  80
        labels  cluster=minikube,service_namespace=$kubernetes['namespace_name'],service_name=$kubernetes['labels']['app'],detected_level=$level

  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
```

```
helm repo add fluent https://fluent.github.io/helm-charts
helm upgrade --install fluent-bit fluent/fluent-bit --version 0.57.6 -f fluentbit.yaml -n hepapi-monitoring
```

In a real cluster, not every log is worth storing. System components, internal controllers, and noisy namespaces can quickly inflate your storage costs and make debugging harder. Fluent Bit's grep filter gives you fine-grained control over which logs are forwarded to Loki.

You can include only the namespaces you care about:

```
[FILTER]
    Name grep
    Match kube.*
    Regex $kubernetes['namespace_name'] ^(app|prod|staging)$
```

Or you can exclude namespaces you want to ignore:

```
[FILTER]
    Name grep
    Match kube.*
    Exclude $kubernetes['namespace_name'] ^(kube-system|calico-system|default)$
```

In this demo, we use the first approach and collect logs only from the app namespace where our sample application is running.

 

#### Deploy Sample Log Pod

To verify that our log pipeline is working end-to-end, we will deploy a simple sample application that continuously outputs JSON-formatted logs at different levels. The application uses the busybox image and alternates between info- and warn-level messages, allowing us to test our Fluent Bit filters and confirm that logs flow into Loki correctly.

```
apiVersion: v1
kind: Pod
metadata:
  name: json-logger
  namespace: prod
  labels:
    app: json-logger
spec:
  containers:
    - name: json-logger
      image: busybox
      command: ["sh", "-c"]
      args:
        - |
          while true; do
            echo "{\"level\":\"info\",\"message\":\"hello from json-logger\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
            sleep 3
            echo "{\"level\":\"warn\",\"message\":\"something might be wrong\",\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"
            sleep 3
          done
```

```
kubectl create ns app
kubectl apply -f sample-log-pod.yaml
```

 

#### Verify Logs in Grafana

Once the application is running and generating logs, navigate to Grafana and click on **Explore** from the left menu. Select **Loki** as the data source, then use the label filters to query your logs. You can filter by cluster, service\_name, or service\_namespace to narrow down the results and confirm that logs are flowing through the pipeline correctly.

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-11-59-51-3114-AM.png?width=1890&height=939&name=undefined-Oct-02-2026-11-59-51-3114-AM.png)

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-11-59-58-7556-AM.png?width=1575&height=687&name=undefined-Oct-02-2026-11-59-58-7556-AM.png)

#### Dropping Unwanted Fields

Even after filtering by namespace, individual log records may still contain fields you do not need, such as stream. Storing these unnecessary fields wastes storage and adds noise when querying logs. You can use the record\_modifier filter to remove specific fields before logs are forwarded to Loki.

```
[FILTER]
    Name record_modifier
    Match kube.*
    Remove_key stream
```

![](https://hepapi.com/hs-fs/hubfs/undefined-Oct-02-2026-12-00-11-5262-PM.png?width=1904&height=823&name=undefined-Oct-02-2026-12-00-11-5262-PM.png)

#### Conclusion

In this article, we walked through a complete log collection pipeline on Kubernetes from the ground up. We deployed GarageFS as a lightweight S3-compatible object storage backend, configured Loki in single binary mode to persist logs in GarageFS buckets, and set up Fluent Bit as a DaemonSet to collect container logs from every node in the cluster. Finally, we connected everything to Grafana for querying and visualization.

It is worth noting that in this demo, GarageFS was deployed inside the same Kubernetes cluster purely for simplicity. In a production environment, this approach is not recommended. If your infrastructure runs on a cloud provider, use a managed object storage service such as AWS S3, Google Cloud Storage, or Azure Blob Storage instead. For on-premises setups, deploy GarageFS on dedicated virtual machines outside the cluster, with multiple nodes for high availability. Either way, keeping your storage layer independent from the cluster protects log data from cluster failures and avoids resource contention.

In the next article, we will explore another popular approach to log collection and storage using Filebeat and Elasticsearch deployed with the Elastic Cloud on Kubernetes (ECK) operator.

Share: [facebook-f icon](http://www.facebook.com/share.php?u=https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki) [twitter icon](https://twitter.com/intent/tweet?url=https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki) [envelope icon](mailto:?body=https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki)

## Related posts

[![](https://hepapi.com/hubfs/Agent%20and%20Collector%20Alternatives%20for%20Observability.jpeg)](https://hepapi.com/blog/agent-and-collector-alternatives-for-observability)

[devops](https://hepapi.com/blog/tag/devops)

#### [Agent and Collector Alternatives for Observability](https://hepapi.com/blog/agent-and-collector-alternatives-for-observability)

[![](https://hepapi.com/hubfs/03.jpeg)](https://hepapi.com/blog/aws-istanbul-local-zone-ile-kvkk-uyumlu-cloud-mimarisi-bölüm-3-nodelar-ve-servisleri)

[devops](https://hepapi.com/blog/tag/devops)

#### [AWS Istanbul Local Zone ile KVKK Uyumlu Cloud Mimarisi - Bölüm 3: Node'lar ve Servisler](https://hepapi.com/blog/aws-istanbul-local-zone-ile-kvkk-uyumlu-cloud-mimarisi-bölüm-3-nodelar-ve-servisleri)

[![](https://hepapi.com/hubfs/Open%20Source%20Observability%20Tools%20(1).png)](https://hepapi.com/blog/open-source-observability-tools)

[devops](https://hepapi.com/blog/tag/devops)

#### [Open Source Observability Tools](https://hepapi.com/blog/open-source-observability-tools)

[![hepapi-logo-light](https://hepapi.com/hubfs/hepapi-logo-light.svg "hepapi-logo-light")](https://hepapi.com/)

[![telcoset-group-company](https://hepapi.com/hs-fs/hubfs/telcoset-group-company.png?width=120&height=36&name=telcoset-group-company.png "telcoset-group-company")](https://telcoset.com.tr/)

Transforming Enterprises with Innovative DevOps, QA, and AI Solutions

[linkedin-in icon](https://www.linkedin.com/company/hepapi) [Follow us on Facebook](mailto:info@hepapi.com)

**Company**

[About](https://hepapi.com/about-us)[Career](https://hepapi.com/career)

[Services](https://hepapi.com/services)

[Partnerships](https://hepapi.com/partnerships)

[Contact Us](https://hepapi.com/contact-us)

[Privacy Policy](https://hepapi.com/privacy-policy)  
[Cookie Policy](https://hepapi.com/cookie-policy)

[Terms & Conditions](https://hepapi.com/terms-and-conditions)

**Resources**

[Blog](https://hepapi.com/blog)

[Knowledge Hub](https://hepapi.github.io/knowledge-hub/)

 

Copyright © 2026, Hepapi Software LTD.

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Ersin Sarı",
    "url" : "https://hepapi.com/blog/author/ersin-sarı"
  },
  "dateModified" : "2026-10-02T12:26:06.836Z",
  "datePublished" : "2026-10-02T12:23:36.000Z",
  "headline" : "From Agent to Storage: Collecting Kubernetes Application Logs with Fluent Bit and Loki",
  "image" : [ "https://hepapi.com/hubfs/kubernetes-application-logs-fluent-bit-loki-1.webp" ],
  "mainEntityOfPage" : {
    "@id" : "https://hepapi.com/blog/from-agent-to-storage-collecting-kubernetes-application-logs-with-fluent-bit-and-loki",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://hepapi.com/hubfs/hepapi-logo.svg"
    },
    "name" : "Hepapi Teknoloji Anonim Sirketi"
  }
}
```