After this lesson, you will be able to: Apply the three pillars of observability (logs, metrics, traces), use structured logging and the four golden signals, instrument with OpenTelemetry, and set up error tracking and sane on-call alerting.
You cannot operate what you cannot see. This lesson covers the three pillars of observability, structured logging, the four golden signals and Prometheus/Grafana, distributed tracing with OpenTelemetry, application performance monitoring with Sentry, and the human side: on-call, good alerts, runbooks, and blameless post-mortems.
Logs are timestamped records of discrete events (what happened). Metrics are numeric measurements over time (how much, how fast, aggregated and cheap to store). Traces follow a single request across services (where the time went). Each answers a different question: logs for detail, metrics for trends and alerting, traces for latency and cross-service debugging. Mature systems use all three; relying on logs alone leaves you blind to trends and distributed latency.
Log JSON, not free-form strings. A structured log ({ level, msg, requestId, userId, durationMs }) is queryable, filterable, and aggregatable; a string log is not. Use log levels deliberately (DEBUG, INFO, WARN, ERROR, FATAL) and attach a correlation ID (request ID) that flows through every log line for a request, so you can reconstruct one request's journey across services. Never log secrets or full PII.
Google's four golden signals are the metrics that matter most for any service: latency (how long requests take), traffic (how many requests), errors (how many fail), and saturation (how full the system is, e.g. CPU/memory/queue depth). Prometheus scrapes and stores time-series metrics; Grafana visualizes them in dashboards and drives alerts. Instrument these four first; they catch most problems before users complain.
A trace represents one request; spans are the timed sub-operations within it (a DB query, an API call). In a microservices system, a trace shows exactly which service and which span made a request slow. OpenTelemetry is the vendor-neutral standard for instrumenting code to emit traces (and metrics and logs); backends like Jaeger or Grafana Tempo store and visualize them. Instrument once with OpenTelemetry and you can switch backends without re-instrumenting.
Application Performance Monitoring tools like Sentry (generous free tier, drop-in SDK) capture exceptions with stack traces, breadcrumbs, and release context, so you learn about errors before users report them; Datadog and New Relic are heavier all-in-one platforms. On the human side: route alerts through PagerDuty or Opsgenie, but write good alerts (actionable and rare, not noisy), keep runbooks so the on-call engineer knows what to do, and run blameless post-mortems that fix systems rather than blame people. Alert fatigue from noisy alerts is itself an outage risk.
Pick one.
String logs you cannot query. No correlation ID, so you cannot follow a request. Alerting on causes (CPU 80%) instead of symptoms (latency/errors users feel), creating noise. Logging secrets or PII. Adding tracing to nothing, or instrumenting per-vendor instead of via OpenTelemetry. Post-mortems that blame a person instead of fixing the system. Treating dashboards no one looks at as 'observability.'
Sign in and purchase access to unlock this lesson.