The black box problem
A Go service handles requests perfectly in development. You deploy it to production and watch the dashboard. CPU usage stays flat. Memory allocation looks normal. The logs just repeat request completed every few seconds. Then the latency spikes. Users complain. You have no idea if the delay lives in the database, a third-party API, or a goroutine stuck waiting on a channel. You are staring at a black box. The only way to find the problem is to guess, add print statements, rebuild, and redeploy. That guessing game costs time and burns through deployment quotas.
Observability turns the black box into a glass box. It gives you a way to ask questions about a running system without rewriting it. You do not need to predict every possible failure mode ahead of time. You just need to emit the right signals so you can reconstruct what happened after the fact. Go makes this straightforward because the language exposes runtime statistics directly and enforces a clean concurrency model. You do not need heavy frameworks to start. You just need to structure your data and let the runtime do the heavy lifting.
What observability actually means
Observability is the ability to measure a system's internal state by analyzing its external outputs. Monitoring tells you when something breaks. Observability tells you why it broke. Think of a car dashboard versus a mechanic's diagnostic computer. The dashboard shows speed and fuel level. The diagnostic computer reads engine temperature, ignition timing, and oxygen sensor voltages. You correlate those signals to find a misfiring cylinder. In software, you collect external signals: logs, metrics, and traces. You analyze them to reconstruct internal state.
Go services generate these signals naturally. Every HTTP request arrives with headers and a body. Every database call returns a result or an error. Every goroutine either finishes or blocks. The language gives you direct access to the scheduler, the garbage collector, and the memory allocator. You just need to route that data to a place where you can query it. Structured data is the foundation. Plain text logs are hard to parse. JSON or key-value pairs let you filter, aggregate, and join events across services.
Observability is plumbing. Run it through every long-lived call site.
The three pillars in Go
Logs, metrics, and traces cover different questions. You need all three because they complement each other.
Logs are discrete events. They answer what happened. In Go, structured logging is the standard. You log request IDs, user IDs, and error chains. The log/slog package in the standard library handles JSON formatting automatically. You attach context values to the logger so every line carries the same correlation identifiers. Logs are expensive to store at scale, so you log at the right level. Info for normal flow. Warn for recoverable issues. Error for failures that require attention. Debug for development only.
Metrics are numerical measurements over time. They answer how much or how fast. Request rates, latency percentiles, goroutine counts, and memory allocations fall here. Metrics are cheap to store because you aggregate them into time series. Go's runtime package exposes many of these for free. You can query runtime.NumGoroutine() or runtime.MemStats without adding overhead. Metrics drive alerts. You set thresholds and let the system notify you when behavior drifts.
Traces follow a single request across services. They answer where time was spent. A trace breaks a request into spans. Each span has a start time, end time, and parent-child relationship. Traces reveal bottlenecks that metrics hide. A metric might show p99 latency at 200 milliseconds. A trace shows that 180 milliseconds live in a single database query. Go's context propagation makes tracing easy. You attach a trace ID to the context and pass it downstream. Every service that reads the context continues the same trace.
A minimal working example
Here is the simplest way to start emitting observable data from a Go program. The example uses the standard library logger and demonstrates context propagation for correlation.
package main
import (
"context"
"log/slog"
"net/http"
"time"
)
// HandleRequest attaches a request ID and logs structured events
func HandleRequest(w http.ResponseWriter, r *http.Request) {
start := time.Now()
ctx := r.Context()
// Generate a unique identifier to correlate logs across handlers
reqID := "req-" + start.Format("20060102150405")
ctx = context.WithValue(ctx, "request_id", reqID)
// Create a logger bound to this request's context
logger := slog.With("request_id", reqID)
// Simulate downstream work that respects cancellation
process(ctx, logger)
// Record duration and method for metric aggregation later
logger.Info("request finished",
"method", r.Method,
"duration_ms", time.Since(start).Milliseconds(),
)
w.WriteHeader(http.StatusOK)
}
// process performs work and checks context for early termination
func process(ctx context.Context, logger *slog.Logger) {
// Simulate I/O wait
time.Sleep(50 * time.Millisecond)
// Check if the client disconnected during the wait
if err := ctx.Err(); err != nil {
logger.Warn("request cancelled", "error", err)
return
}
}
The code shows three observable patterns. A unique request ID flows through the context. The logger attaches that ID to every line. The handler records duration for metrics. The downstream function checks ctx.Err() to avoid wasting work. This structure scales from a single binary to a distributed system.
How the runtime handles it
Go's scheduler moves goroutines across OS threads automatically. You do not manage threads manually. The runtime tracks goroutine creation, blocking, and completion. When you call runtime.NumGoroutine(), you get a snapshot of active goroutines. When you query runtime.MemStats, you get allocation counts, heap size, and GC pause times. These numbers update continuously. You can expose them via an HTTP endpoint or push them to a metrics collector.
Context propagation is the glue that holds observability together. context.Context always goes as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines. When a request arrives, you create a context. You pass it to database calls, HTTP clients, and goroutines. If the client disconnects, the context cancels. Every downstream call checks ctx.Err() and stops early. This prevents goroutine leaks and wasted CPU. The worst goroutine bug is the one that never logs.
The garbage collector runs concurrently with your program. It pauses execution for milliseconds, not seconds. You can observe GC behavior through runtime.MemStats.PauseTotalNs and runtime.MemStats.NumGC. High allocation rates trigger more frequent collections. Tracing allocation sites helps you find hot paths. Go's race detector and profiler integrate with these runtime stats. You do not need to guess where memory goes. You just read what the runtime already tracks.
Trust the scheduler. Measure the outcomes.
Realistic service setup
Production services need more than a single handler. They need database calls, external API requests, and background workers. Here is how you attach observability to a realistic data fetch. The example shows a custom tracer interface and context-aware error handling.
package main
import (
"context"
"database/sql"
"log/slog"
"time"
)
// QueryDB executes a statement and logs timing and errors
func QueryDB(ctx context.Context, db *sql.DB, logger *slog.Logger, query string) ([]byte, error) {
start := time.Now()
// Execute with context to respect deadlines and cancellation
rows, err := db.QueryContext(ctx, query)
if err != nil {
// Log the failure with structured fields for downstream correlation
logger.Error("query failed", "query", query, "error", err)
return nil, err
}
defer rows.Close()
// Collect results into a byte slice for simplicity
var result []byte
for rows.Next() {
var val []byte
if err := rows.Scan(&val); err != nil {
logger.Warn("scan failed", "error", err)
continue
}
result = append(result, val...)
}
// Record success metrics for aggregation
logger.Info("query completed",
"query", query,
"duration_ms", time.Since(start).Milliseconds(),
"rows", len(result),
)
return result, nil
}
The function follows Go conventions. The receiver name pattern does not apply here because it is a package-level function, but the signature respects the ctx first rule. Error handling uses the standard if err != nil pattern. The community accepts the boilerplate because it makes the unhappy path visible. The logger attaches timing and row counts. You can pipe these logs into a metrics pipeline or a trace backend. The context ensures that long-running queries cancel when the client drops the connection.
Common pitfalls and runtime surprises
Observability breaks when you emit the wrong data or block the hot path. Logging every request body at info level fills storage in hours. You filter by sampling or log only on errors. Blocking on a metrics pusher stalls the request. You batch metrics and send them asynchronously. Missing request IDs makes correlation impossible. You attach the ID at the ingress point and pass it through every boundary.
The compiler catches some mistakes early. If you pass a non-comparable type to context.WithValue, the compiler rejects the program with context.WithValue expects a key type that is comparable. If you forget to use a returned error, you get declared and not used. If you try to pass a *string to a function expecting a value, the compiler complains with cannot use x (type *string) as string value in argument. Strings are already cheap to pass by value. Pointers add indirection without saving memory.
Goroutine leaks happen when a goroutine waits on a channel that never gets closed. Always have a cancellation path. Use select with ctx.Done() to break out of blocking operations. The _ (underscore) discards a value intentionally. result, _ := ... says you considered the second return value and chose to drop it. Use it sparingly with errors. Dropping errors silently hides failures that observability should surface.
Public names start with a capital letter. Private names start lowercase. No keywords like public or private. Interfaces are accepted, structs are returned. "Accept interfaces, return structs" keeps your code flexible. You can swap implementations without changing callers. Observability tools follow this pattern. You implement a tracer interface and pass it to your HTTP router. The router calls the interface. You never depend on a specific vendor.
gofmt is mandatory. Don't argue about indentation. Let the tool decide. Most editors run it on save. Consistent formatting makes code reviews faster. You argue logic, not whitespace.
When to reach for what
Use structured logging when you need discrete event records for debugging and audit trails. Use metrics when you need to track trends, set alerts, and measure system health over time. Use distributed tracing when you need to follow a single request across multiple services and identify latency bottlenecks. Use runtime statistics when you need to monitor Go-specific resource usage like goroutine counts and GC pauses. Use plain print statements when you are prototyping locally and will delete them before shipping.