Observability

After this lesson, you will be able to: Read Pod logs effectively, install Prometheus + Grafana for metrics, and understand the role of distributed tracing.

Observability is what makes K8s operable. Without metrics + logs + traces, you can't tell why an app is slow at 3am.

Prerequisites:Managed Kubernetes

Logs: kubectl, then aggregation

kubectl logs gets you started; production needs aggregation.

html

# Single Pod logs
kubectl logs <pod>
kubectl logs <pod> -c <container>   # multi-container Pod
kubectl logs <pod> --previous       # logs from prior crash
kubectl logs -f <pod>               # follow

# Across many Pods
kubectl logs -l app=myapp --tail=200
kubectl logs -l app=myapp -f --max-log-requests=20

# stern (the better multi-Pod log tool)
brew install stern
stern myapp                          # follow logs across all Pods with 'myapp' in name
stern myapp -n production --since 5m

# For production: ship logs to Loki / CloudWatch / Datadog / Elastic.
# Apps should log JSON to stdout; a log shipper (Promtail, Fluent Bit) does the rest.

Metrics: Prometheus is the K8s default

Prometheus scrapes metrics from your Pods (via /metrics endpoints) and stores them. kube-state-metrics + node-exporter provide cluster + node metrics. Grafana visualizes them. The kube-prometheus-stack Helm chart bundles all of this. Apps expose metrics in Prometheus format: counters (requests_total), gauges (memory_usage_bytes), histograms (request_duration_seconds_bucket).

Install kube-prometheus-stack

One command. Adds Prometheus + Grafana + AlertManager.

tsx

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

# Access Grafana
kubectl port-forward -n monitoring svc/kube-prometheus-grafana 3000:80
# Browse http://localhost:3000   (default admin/prom-operator)

# Default dashboards: cluster overview, namespaces, nodes, workloads.
# Add your own metrics by exposing /metrics on your app and creating a ServiceMonitor.

Tracing: when logs and metrics aren't enough

Distributed tracing connects 'this request' across many services. Each service adds a span; the result is a flame graph showing where time was spent. OpenTelemetry is the modern standard (instrumentation SDK, backend-agnostic). Backends: Jaeger, Tempo, Honeycomb, Datadog. When to add it: when one user request spans 3+ services and 'where's the latency?' is the daily question. Don't bother before that.

Common mistakes only experienced engineers avoid

Logging non-JSON to stdout. Aggregators struggle to parse free-form text. Skipping metric labels (`app=myapp, env=prod`). Without them, dashboards mix data. Cardinality explosions (high-cardinality labels like user_id in metrics). Crashes Prometheus. Alerting on every WARN log. Drowns the on-call. Skipping runbooks. An alert without 'here's what to do' is half useful.

Quick Check

Your service is slow but logs look normal. What signal do you reach for next?

Pick the most useful diagnostic.

←Managed Kubernetes (EKS / GKE / AKS)

Back to Kubernetes

Passion Project: Multi-Service Helm Chart→