Key Takeaways
- Kubernetes observability rests on three pillars: logs, metrics, and traces.
- Critical Kubernetes metrics cover health (restart counts, pod phase), performance (CPU and memory usage), and state (etcd leader, failed jobs).
- Designing observability in from the start, with standardized log formats and consistent labels, beats adding it later.
Feature flags let teams shift traffic away from a failing dependency in production without a redeployment. Kubernetes observability is the systematic collection of data on the health, state, and performance of various cluster components. It gives teams the visibility to pinpoint what is wrong and where. Pairing observability data with feature flags and release management takes it further. Teams can isolate bad changes and roll them back without a full redeployment.
This article discusses the 3 main pillars of observability and the critical metrics to monitor for Kubernetes. It also covers automatic alerts on metrics, key challenges in Kubernetes observability, and practical recommendations for running Kubernetes observability in production.
The following table summarizes the key concepts covered in this article and serves as a quick reference for the ideas discussed throughout.
Summary of key Kubernetes observability concepts
Concept | Description |
|---|---|
3 pillars of Kubernetes observability | Logs, metrics, and traces. Each illustrates a different perspective on the state, health, and performance of Kubernetes. |
Critical Kubernetes metrics | CPU/memory usage pod health restart counts request latency |
Alerts and notifications | Alerts enable timely responses to incidents by notifying engineers and developers of anomalies and SLO violations. |
Observability by design | Modern Kubernetes applications treat observability as a first-class citizen. Modern engineering teams design, plan, and maintain observability alongside the application workload rather than adding it later. |
Why does Kubernetes observability matter?
When a critical production incident occurs, the business impact can be immediate, even for small changes in metrics. From that moment, every minute matters. The objective is not only to restore service quickly but also to identify the root cause before the operational and financial impact escalates.
However, the source of the problem is rarely obvious. The issue could stem from a slow database query, a misconfigured network policy disrupting communication between services, an unhandled exception introduced in a recent deployment, or an external dependency experiencing latency or failure.
Without a structured approach to investigation, engineering teams can spend valuable time moving among dashboards, logs, and infrastructure to find the underlying cause.
Rapid incident resolution depends on observability. It allows teams to systematically narrow the scope of investigation, correlate signals across the technology stack, and pinpoint the precise component responsible for the service degradation.
1. Three pillars of Kubernetes observability
The three pillars of observability are logs, metrics, and traces. The following table summarizes these three pillars, their definitions, and their roles in understanding system behavior.
Name | Description | Role in observability |
|---|---|---|
Logs | Timestamped records of discrete events. The most common debugging tool in software applications. | Capture detailed event information, including metadata. |
Metrics | Numeric measurements recorded at regular intervals | Capture trends and patterns over time. Also used as a foundation for raising alerts |
Traces | End-to-end record of the request journey across different components and services. | End-to-end view of a specific request |
All three pillars (logs, metrics, and traces) work together to provide a unified picture of the system's state, health, and performance.
Logs
Logs are generally formatted text strings written by developers to understand system behavior during development and in production. They are good at showing detailed information about a particular event, including timestamps to understand the system's sequence of events.
The text-based simplicity of logs means that they can be used in any Kubernetes environment without any additional observability-specific tools.
Metrics
Unlike logs, metrics only capture high-level information, such as the number of event occurrences or specific values that change over time. They are time-series data points at regular intervals and must be considered cumulatively over time.
Common examples of metrics include CPU load, RAM utilization, and disk usage, collected at a configured scrape interval.
Traces
Traces allow teams to review the entire lifecycle of a request in a single data point. Each trace is composed of multiple spans. A span represents the time a request spends in a particular component. As the request moves from one component to another, new spans are generated. Each new span is linked to the previous span, enabling the development team to identify and prioritize bottlenecks in the request path. Further, it also enables the team to explore which part of the system is failing to process the request.
Observability tooling
In practice, each pillar relies on dedicated agents and engines to collect, aggregate, and process data at scale.
Logs are collected using tools like Fluent Bit and Grafana Alloy. This logging data is then sent to centralized log systems such as Loki or Elasticsearch.
When it comes to metrics, Prometheus is a most widely used tool, though commercial platforms like Datadog, Grafana Labs, and New Relic offer managed alternatives. Prometheus plays a dual role, collecting metrics by scraping targets and ingesting and storing time-series data locally.
Traces work a bit differently: you instrument the application itself so it can emit spans, which are then pushed to systems like Tempo, Jaeger, or Zipkin. Datadog and New Relic also natively support distributed tracing.
OpenTelemetry has become the common instrumentation layer for observability stacks. It provides a vendor-neutral SDK and collector that ships logs, metrics, and traces to any compatible backend. Grafana Labs, Datadog, and New Relic all natively support OpenTelemetry.
Advances in Kubernetes observability
Increasingly, there has been more focus on building tools that can ingest logs, metrics, and traces in a single deployment. This enables engineering teams to help lower operational complexity and reduce costs. It also lets teams build stronger correlation across the three pillars.
LaunchDarkly brings these signals together with feature management. It gives teams a single place to correlate observability data with deployment events. The following diagram shows how all three observability signals travel from your application to the LaunchDarkly platform.
Figure 1: Kubernetes observability data flow
2. Critical Kubernetes metrics
Kubernetes exposes hundreds of metrics across the API server, kubelet, and cAdvisor. Together, they capture Kubernetes internal state, workload health, and performance. The table below highlights the most important ones, along with the example queries they help answer.
Category | Metric | Why does it matter? |
|---|---|---|
Health | kube_pod_container_status_restarts_total | Is a container crash looping? A rising restart count may indicate the CrashLoopBackOff condition. |
Health | kube_pod_status_phase | Are pods in a healthy running state? Or are they stuck in a Pending or Failed state? |
Health | kube_node_status_condition | Is every node ready? Conditions like MemoryPressure and DiskPressure may make the node unschedulable for additional workload. |
Performance | container_cpu_usage_seconds_total | How much CPU is a container consuming over time? Comparing it with CPU requests and limits helps identify throttling and over-provisioning. |
Performance | container_memory_working_set_bytes | How close is the container to its memory limits? It helps detect memory pressure and the risk of OOM kills. |
Performance | node_cpu_seconds_total | How much CPU is being utilized? This is instrumental in cluster-wide CPU capacity planning. |
Performance | kube_horizontalpodautoscaler_status_current_replicas | Is the HPA scaling up and down as expected? Comparing the current and desired replica counts reveals a lag in scaling. |
State | etcd_server_has_leader | Does the Kubernetes etcd server have a leader? A value of 0 indicates a cluster-wide emergency, since the control plane cannot store Kubernetes state writes. |
State | kube_job_status_failed | Has any job failed? This is useful in detecting silent failures in cron jobs. |
State | kube_persistentvolumeclaim_status_phase | Are all persistent volume claims bound? Unbound PVCs block pod scheduling for stateful workloads. |
3. Alerts and notifications
Kubernetes environments generate a constant stream of metrics. This telemetry data is processed by an alert pipeline that converts raw data into alert notifications.
Figure 2: Alert pipeline
As the diagram shows, Prometheus sits at the middle of this pipeline. It evaluates alerting rules at regular intervals, checking whether the condition defined in each alert rule is satisfied. A simple alerting rule looks like the following:
This rule fires when a container in a pod restarts within a 5-minute rolling window. When the alert expression crosses the threshold defined in the alerting rule, the alert state changes from Inactive to Pending. The for clause plays a critical role in alert stability. Without it, the alert expression that oscillates around the threshold will cause alert flapping. The alert will rapidly cycle between Inactive and Firing.
The Pending state acts as a buffer, ensuring that an alert fires only when the condition is sustained, rather than transient. The alert remains Pending for the duration specified in the alerting rule's for clause. Once this duration has elapsed, the alert is moved to the Firing state and is handed off to Alertmanager.
Figure 3: Alert state lifecycle
Within Alertmanager, related alerts are grouped to reduce noise. Each alert group is routed to a notification handler based on its labels. For example, a critical severity alert can be sent to email, while a warning can be routed to a Slack channel. Alertmanager also supports silences, allowing teams to suppress known alerts during planned maintenance. Supported notification handlers include Slack, Jira, email, and webhook.
4. Responding to an incident
An alert is triggered when a metric exceeds its defined threshold. Alertmanager routes the notification to the on-call engineer, identifying which metric has breached its threshold. For example, if a metric is associated with the checkout service, the issue is immediately narrowed to the checkout workflow. While this significantly reduces the scope of the investigation, it does not identify where in the workflow the failure is occurring.
The next step is to examine the distributed traces of the checkout workflow and identify patterns. For example, if errors are concentrated around the integration with the external payment service provider, the investigation can be refined from a broad issue within the checkout flow to a specific failure point in the payment integration.
With the issue localized, the final step is to examine associated logs. They reveal more details, like repeated 429 Too Many Requests responses from the external payment provider, indicating that requests are being rate-limited.
At this stage, operational context becomes as important as the telemetry itself. For instance, if the payment provider was recently onboarded as part of a pilot program, with a capped request quota defined under the pilot agreement, the integration may exceed the allocated quota, resulting in the observed failures.
This illustrates the value of distributed tracing. Rather than spending hours isolating the source of a problem, engineering teams can quickly identify the precise component introducing errors or latency.
Incident resolution with LaunchDarkly
If using LaunchDarkly for feature flags and traffic management, the fix can be completed without code changes or deployments. All the engineering team needs to do is adjust the traffic split in the LaunchDarkly dashboard. For example, instead of the previous 50%, only 10% of the traffic is sent to the new payment service provider. The change takes effect in production without code changes or CI/CD deployment.
Without this type of incident resolution, the team's MTTR can be capped by the pipeline's speed.
5. Observability by design
In the past, observability was not planned during the software design phase. Adding observability on a need basis often led to inconsistent labels and limited code coverage.
The modern approach recommends establishing observability during the design and development phases. Treat it as a first-class requirement alongside functionality, performance, and automated testing.
Emit logs in a standardized format and ensure they are enriched with context metadata such as service name, environment, trace ID, and request ID. This makes them easily searchable and correlated with other signals.
Ensure services expose custom metrics covering business indicators, such as order processing rates and payment success ratios, as well as technical metrics such as memory and CPU utilization. Both business and technical metrics are necessary to give teams visibility into what the system is doing, not just how the underlying resources are used. Business metrics like payment_success_ratio do double duty: they inform incident response, and they provide the signal a guarded release or experiment needs to detect a regression automatically.
Keep metadata and labels consistent across logs, metrics, and traces to support cross-signal correlation during an incident. A field called service should mean the same thing everywhere, and the checkout label in metrics should correspond to the same component as the checkout tag in a trace.
Write and refine alert rules in parallel with code development, rather than leaving it to a separate team post-deployment. Engineers who understand the code are the best people to define meaningful thresholds and the appropriate "for" clauses, resulting in a richer context and more accurate signals.
6. Kubernetes observability recommendations
Recommendation | Action items |
|---|---|
Centralized observability with correlation | Deploy a unified backend (e.g., the LGTM stack or LaunchDarkly) such that logs, metrics, and traces exist in a single system. Enable cross-signal correlation by linking trace IDs to logs. Define and use a consistent set of labels (e.g., service, env, namespace) across all three pillars of observability. |
Signal over volume | Drop high-volume, low-value logs, such as nginx access logs and debug-level logs. Avoid scraping high-cardinality labels, such as user IDs. Apply tail-based sampling to prioritize the ingestion of error traces over successful ones. |
Observability as a first-class citizen | Set requests and limits on all observability components to avoid saturation Alert on ingestion errors, scrape failures, and observability components restart Define explicit retention and capacity policies to avoid unbounded data growth |
RED metrics with disciplined alerting | Instrument each service with request rate, error rate, and duration metrics Use a for clause 3-5 times the metric scrape interval to avoid alert flapping. Periodically retire alerts that fire but do not lead to action. |
Conclusion
Logs, metrics, and traces are the three pillars of observability, giving a different perspective. You need all three to understand Kubernetes behavior.
It is not something you add later but needs to be part of the design from the start. The sooner you build it, the easier it is to run things reliably.
More data does not always mean better visibility. Collecting large volumes of logs, metrics, and traces can be expensive with limited gains. Being selective about what you collect is usually more effective.
Integrating observability data with feature flags lets you run experiments with your infrastructure and respond to degradations more gracefully. Further, you can update system behavior in production without risking new deployments.
FAQs
What is the difference between Kubernetes monitoring and observability? Monitoring tells you that something is wrong; observability helps you work out why. Monitoring watches predefined metrics and thresholds, such as CPU usage or pod restarts. Observability combines logs, metrics, and traces so you can investigate failures you did not anticipate and trace them to a specific component.
Do you need OpenTelemetry for Kubernetes observability? No, but it is the practical default. OpenTelemetry provides a vendor-neutral SDK and collector that ships logs, metrics, and traces to any compatible backend, so you can change vendors without reinstrumenting your applications. Grafana Labs, Datadog, and New Relic all support it natively.
How do you reduce Kubernetes observability costs? Collect less, not more. Drop high-volume, low-value data such as nginx access logs and debug-level logs, avoid scraping high-cardinality labels like user IDs, and apply tail-based sampling so error traces are ingested ahead of successful ones. Explicit retention and capacity policies stop data growing without bound.
How do you stop Kubernetes alerts from becoming noise? Group related alerts and route them by label so each one reaches the right team. Retire any alert that fires repeatedly but never leads to action.
