Centralized Logging with Loki for Multi-Namespace Clusters: A DevOps Engineering Guide

ByShubham Mishra

Intermediate · 11 min read

The Problem

In the modern microservices landscape, a single application deployment is rarely a monolith. It is an orchestra of containers spread across multiple namespaces within a Kubernetes cluster. As your organization scales, the volume of logs generated by these pods increases exponentially. Traditionally, developers and SREs have relied on local log files or basic stdout/stderr aggregation. However, this approach hits a hard ceiling when debugging complex, distributed issues.

Consider a real-world scenario: A payment processing service fails intermittently during peak hours. The error manifests in the payment-service namespace, but it relies on a user authentication microservice in the auth-system namespace, which in turn queries a database running in the data-platform namespace. If logs are siloed per namespace or require logging into individual nodes to view container logs, correlating these events becomes a manual, time-consuming nightmare. You might spend hours switching contexts, parsing JSON manually, and guessing at timestamps.

This fragmentation creates “log blind spots.” Without a unified, centralized view, you cannot easily trace a request ID across services, identify cascading failures, or perform historical analysis for compliance. The pain point is not just storage; it is the inability to correlate context across boundaries. This tutorial solves that by implementing Grafana Loki, a log aggregation system designed specifically for Kubernetes, allowing you to query logs from all namespaces using a unified interface.

Core Concepts

To understand how we will solve this problem, we must first define the core components of the Loki architecture. Unlike traditional logging stacks like ELK (Elasticsearch, Logstash, Kibana), Loki does not index the contents of the logs. This is its primary differentiator and the source of its efficiency.

  • Loki Server: The central component responsible for ingesting, storing, and querying logs. It stores logs in compressed format in object storage (like S3, GCS, or MinIO) or local disk. Because it only indexes metadata (labels), it uses significantly less storage than full-text indexing systems.
  • Grafana: The visualization layer. While Loki can be queried via CLI, Grafana provides the UI for exploring logs, creating dashboards, and integrating metrics with logs. In our setup, Grafana acts as the frontend for Loki.
  • Loki Promtail Agent: A lightweight DaemonSet agent deployed on each node. Its job is to tail the log files from containers running on that node, add Kubernetes labels (such as namespace, pod name, and container name) to the log stream, and push them to the Loki server. It handles the heavy lifting of parsing and relabeling locally.
  • LogQL: Loki’s query language. It is inspired by PromQL (Prometheus Query Language). LogQL allows you to filter streams by label matchers and then filter the content within those streams using regex or line filters. This makes it intuitive for users already familiar with Prometheus.
  • Multi-Tenancy: In a multi-tenant or multi-team environment, Loki supports tenant isolation. Each team or namespace can be treated as a separate tenant, ensuring that Team A cannot query Team B’s sensitive logs unless explicitly permitted. For this tutorial, we will focus on a single-tenant setup for simplicity, but the architecture supports scaling to many tenants.

By decoupling indexing from storage, Loki achieves high cardinality support without the performance penalties associated with Elasticsearch. This makes it ideal for clusters where you have thousands of unique pods and dynamic namespaces.

How It Works

The mechanics of centralized logging in Kubernetes rely on a pipeline of collection, enrichment, and storage. Here is the step-by-step flow of data:

  1. Collection: On every Kubernetes node, a Pod running the promtail agent starts. Since it runs as a DaemonSet, there is one instance per node. Promtail mounts the host path where container logs are stored (typically /var/log/pods). It watches these directories for new files.
  2. Enrichment: When Promtail detects a new log file corresponding to a container, it performs two critical actions. First, it parses the filename to extract Kubernetes metadata (namespace, pod_name, container_name, replica_set). Second, it applies configuration rules defined in its YAML config to add additional static labels or drop unwanted logs based on patterns.
  3. Pushing: Promtail batches the enriched log lines and pushes them to the Loki server over HTTP/gRPC. It does not wait for a log entry to finish writing; it pushes in near-real-time batches to reduce network overhead.
  4. Storage: The Loki server receives the batch, assigns a timestamp, and compresses the data. It writes the raw log data to long-term storage (object storage) and updates its index, which contains only the labels and the byte offset in the storage file. This index is kept in memory and on local disk for fast querying.
  5. Querying: When a user queries logs via Grafana, Loki uses the label index to quickly locate relevant files in object storage, fetches only the necessary chunks, decompresses them, and applies any line-level filters before returning the results to the UI.

This architecture ensures that the control plane (Loki Server) remains lightweight, while the compute-intensive task of log parsing is distributed to the edge (Promtail Agents).

Prerequisites

Before proceeding with the implementation, ensure you have the following prerequisites in place. This guide assumes a standard Linux-based Kubernetes cluster.

  • Kubernetes Cluster: A running cluster with at least version 1.20. We recommend having access to kubectl configured with administrative privileges.
  • Helm v3: Helm is the package manager for Kubernetes. Ensure it is installed on your local machine or CI/CD runner. You will use it to deploy Loki and Promtail easily.
  • Grafana Instance: You need a Grafana instance running and accessible. If you do not have one, you can deploy a temporary one for testing, but ideally, you should integrate Loki into your existing Grafana monitoring stack.
  • Object Storage (Optional but Recommended): For production-grade setups, configure access to an S3 bucket, Google Cloud Storage, or Azure Blob Storage. For this tutorial, we will use the default in-memory/local storage mode for Loki to keep the setup simple and portable, but note that this data is volatile.
  • Namespace Awareness: Understand your cluster’s namespace structure. We will demonstrate filtering across default, kube-system, and custom namespaces.

Hands-On Walkthrough

We will now deploy Loki and Promtail using Helm charts. This approach ensures best practices and easy upgrades.

Step 1: Add the Grafana Helm Repository

First, add the official Grafana Helm repository to your local Helm client.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Step 2: Deploy Loki Server

Create a file named loki-values.yaml. We will configure Loki to use minimal resources for this tutorial but enable multi-tenancy headers if needed later.

# loki-values.yaml
ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_block_size: 262144
  chunk_retain_period: 30s

singleBinary:
  replicas: 1

monolithic:
  auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096

common:
  path_prefix: /tmp/loki
  storage:
    filesystem:
      directory: /tmp/loki/chunks

limits_config:
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h

schema_config:
  configs:
    - from: 2020-10-24
      store: tsdb
      object_store: filesystem
      schema: v13
      period: 24h

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Deploy Loki to the monitoring namespace. If this namespace doesn’t exist, create it:

kubectl create namespace monitoring
helm install loki grafana/loki -n monitoring -f loki-values.yaml

Step 3: Configure and Deploy Promtail

Promtail needs to know where to send logs and how to label them. Create a file named promtail-values.yaml.

# promtail-values.yaml
config:
  clients:
    - url: http://loki.monitoring.svc.cluster.local:3100/loki/api/v1/push
  scrape_configs:
    - job_name: kubernetes-pods
      pipeline_stages:
        - docker: {}
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_namespace]
          target_label: namespace
        - source_labels: [__meta_kubernetes_pod_name]
          target_label: pod
        - source_labels: [__meta_kubernetes_pod_container_name]
          target_label: container
        - action: replace
          source_labels: [__meta_kubernetes_pod_node_name]
          target_label: node_name
        - action: replace
          source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - action: replace
          source_labels: [__meta_kubernetes_pod_name]
          target_label: kubernetes_pod_name
        - action: replace
          replacement: kube-container-$(kubernetes_pod_name)-$(container)
          target_label: __path__

Deploy Promtail. Note that we do not specify a namespace here because DaemonSets typically run across all namespaces, but the chart installs into the current context or a specified one. We will install it in monitoring as well for consistency.

helm install promtail grafana/promtail -n monitoring -f promtail-values.yaml

Step 4: Connect Loki to Grafana

If you are using an existing Grafana instance, you need to add Loki as a datasource. If you deployed a fresh Grafana, you can do this automatically by adding a ConfigMap or updating the values. For simplicity, let’s assume you are connecting to an external Grafana. Open your Grafana UI, go to Configuration > Data Sources > Add data source, select Loki, and set the URL to http://loki.monitoring.svc.cluster.local:3100. Save and Test.

Verifying It Works

Now that the infrastructure is up, we need to verify that logs from multiple namespaces are flowing correctly into Loki.

Step 1: Generate Test Logs

Create a dummy pod in the default namespace that outputs logs:

kubectl run test-logger --image=alpine -- sh -c 'while true; do echo "INFO $(date) Message from default namespace"; sleep 5; done'

Create another pod in the kube-system namespace (you may need admin rights to write directly there, or create a test namespace called dev-team):

kubectl create namespace dev-team
kubectl run dev-logger --namespace=dev-team --image=alpine -- sh -c 'while true; do echo "WARN $(date) Error from dev team app"; sleep 5; done'

Step 2: Check Promtail Logs

Ensure Promtail is running and has started scraping:

kubectl logs -n monitoring -l app=promtail -f | head -n 20

You should see lines indicating successful pushes to the Loki endpoint. Look for level=info msg="Pushed batch successfully".

Step 3: Query in Grafana

Open Grafana and navigate to the Explore tab. Select the Loki datasource. Enter the following LogQL query:

{namespace="default"}

You should see the logs from the test-logger pod. Now try querying the other namespace:

{namespace="dev-team"}

If both queries return results, your multi-namespace centralized logging is working. You can combine them:

{namespace=~"default|dev-team"} |= "ERROR"

This demonstrates the power of LogQL: filtering by multiple namespaces and specific text content simultaneously.

Common Mistakes

Even with a straightforward setup, several pitfalls can arise.

  1. High Cardinality Labels: A common mistake is labeling logs with highly variable fields, such as request_id or user_email, as top-level labels in Loki. Loki’s index grows linearly with the number of unique label combinations. If you label every unique user email, your Loki server will consume massive amounts of RAM and eventually crash. Instead, put high-cardinality data in the log line itself and filter by regex in LogQL.
  2. Network Connectivity Issues: Promtail runs on nodes. If your cluster uses strict network policies, ensure that the nodes can reach the Loki service DNS. A silent failure occurs when Promtail cannot connect, and logs are dropped silently. Always check Promtail logs for connection errors.
  3. Time Skew: Loki relies on accurate timestamps. If your cluster nodes have significant time drift, log ordering will be incorrect. Use NTP (Network Time Protocol) synchronization across all nodes. Loki will warn you about out-of-order samples, which can degrade query performance.
  4. Ignoring Retention Policies: By default, Loki does not delete old logs unless configured to do so. In a high-volume cluster, your disk space (or object storage costs) will grow indefinitely. Always configure retention_period in the table_manager or limits config to manage storage lifecycle.

Going Further

Once the basic setup is verified, consider these production-grade enhancements.

Scaling Loki

The single-binary mode used above is suitable for small to medium clusters. For large-scale deployments, split the Loki components into Microservices Mode: Distributor, Ingester, Querier, and Gateway. This allows you to scale each component independently based on load. The Distributor handles ingestion, the Ingester manages storage chunks, and the Querier handles API requests.

Security and Multi-Tenancy

In multi-tenant environments, enable auth_enabled: true in the Loki config. This requires clients (Promtail) to send an X-Scope-OrgID header identifying the tenant. Promtail can be configured to inject this header based on the namespace, ensuring that the finance namespace only sends logs to the finance tenant bucket. Additionally, use mTLS between Promtail and Loki to encrypt data in transit.

Performance Tuning

Adjust the chunk_idle_period and chunk_block_size in Loki values. Smaller blocks increase query latency but reduce storage overhead. Larger blocks improve compression but make queries slower for recent logs. Tune these based on your average log size and query patterns. Also, monitor the loki_distributor_received_ingest_bytes_total metric to detect bottlenecks in ingestion.

Alternative Approaches

While Loki is excellent for Kubernetes-native logging, consider alternatives like Vector or Fluent Bit if you need complex transformation pipelines before sending logs to Loki. These agents can preprocess logs (e.g., redact PII, parse JSON into structured fields) before pushing to Loki, reducing the load on Loki and improving query performance.

Cleanup

To remove the resources created during this tutorial, execute the following commands:

helm uninstall loki -n monitoring
helm uninstall promtail -n monitoring
kubectl delete namespace monitoring
kubectl delete namespace dev-team
kubectl delete pod test-logger

Note that if you used persistent storage or object storage, ensure you clean up any buckets or volumes associated with the Loki data path to avoid unexpected cloud costs.

References

This article was drafted with AI assistance and reviewed by the DevOpsEngine editors before publication.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *