The log stream is your contract
You deploy a new Go microservice. It works. Requests come in, data flows out. Then the on-call engineer pings you at 2 AM. "I can't find the error in Kibana. Your logs are just blobs of text." You look at your code. You used fmt.Println everywhere. The logs hit stdout, the container runtime grabs them, and the log aggregator sees a stream of unstructured noise. Structured logging isn't a luxury. It's the bridge between your code and the tools that keep the system alive.
Structured logging and the JSON bridge
Go's standard library has log, but it writes plain text. Text logs are hard to query. A search for "Error 500" matches "Error 5001". JSON logs have keys and values. {"level": "error", "code": 500}. The aggregator indexes the keys. You can filter by level=error and code=500. The pattern is simple: your Go app writes structured JSON to stdout. A log collector reads stdout and ships it to ELK, Datadog, or Loki. You never connect your Go app directly to the log backend. That coupling is a trap.
Think of logging like packing a shipment. Your code is the factory. The log aggregator is the warehouse. If you throw items in a box with no labels, the warehouse can't sort them. You need a standard format (JSON) and a reliable courier (agent or sidecar). The format ensures the warehouse can read the data. The courier ensures the data arrives. Your job is to pack the box correctly. The infrastructure handles the rest.
Minimal JSON logger with zap
Here's the simplest way to get structured JSON out of Go using zap, the industry-standard logger for performance. zap is fast because it avoids reflection and allocates memory efficiently.
package main
import (
"os"
"time"
"go.uber.org/zap"
"go.uber.org/zap/zapcore"
)
func main() {
// Production encoder config adds timestamps and levels automatically.
encoderConfig := zap.NewProductionEncoderConfig()
encoderConfig.EncodeTime = zapcore.ISO8601TimeEncoder // Human-readable timestamps help debugging.
// JSON encoder turns log entries into structured JSON bytes.
encoder := zapcore.NewJSONEncoder(encoderConfig)
// Core connects encoder, output destination, and log level.
// os.Stdout ensures logs go to the container's standard output.
core := zapcore.NewCore(encoder, zapcore.AddSync(os.Stdout), zapcore.InfoLevel)
// Build the logger.
logger := zap.New(core)
defer logger.Sync() // Flushes buffer to avoid losing logs on exit.
// Fields are added per-log, not globally.
logger.Info("request completed",
zap.String("method", "GET"),
zap.Int("status", 200),
zap.Duration("latency", 12*time.Millisecond),
)
}
zap buffers writes for performance. The defer logger.Sync() call ensures the buffer empties when the program exits. If you skip Sync, the process might crash or terminate before the logs flush. The output is a single JSON line per log entry. Keys are consistent. Values are typed. The aggregator can parse this reliably.
JSON to stdout. Let the infrastructure do the shipping.
The lifecycle of a log entry
The logger creates an encoder. The encoder formats fields into JSON. The core writes to stdout. The container runtime captures stdout. The log collector reads the stream. The collector parses JSON and sends it to the API of the backend. Your code only knows about stdout. If you switch from ELK to Loki, you change the collector config, not your Go code. This decoupling is the key.
Your application is stateless regarding logs. It doesn't store logs. It doesn't manage connections to the backend. It writes to a stream. This follows the 12-factor app methodology. The log collector handles retries, batching, and authentication. If the backend is down, the collector buffers locally. Your app keeps running. You never block your request handler waiting for a log to ship.
Realistic handler with context and fields
Real apps need request IDs, context propagation, and error details. Here's a middleware that attaches a logger to the context and a handler that uses it.
// Middleware attaches a logger to the context with request metadata.
func loggingMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// Clone logger with request fields. Cloning is cheap and thread-safe.
logger := baseLogger.With(
zap.String("request_id", r.Header.Get("X-Request-ID")),
zap.String("method", r.Method),
)
// Store logger in context using a typed key to avoid collisions.
ctx := context.WithValue(r.Context(), LoggerContextKey{}, logger)
next.ServeHTTP(w, r.WithContext(ctx))
})
}
The middleware clones the base logger and adds request-specific fields. Cloning is safe for concurrent requests. Each request gets its own logger instance with its own fields. The logger is stored in the context using a typed key. Using a custom type for the key prevents collisions with other packages that might use string keys.
// handleData retrieves the logger and records the operation result.
func handleData(w http.ResponseWriter, r *http.Request) {
// Assert type safety when retrieving from context.
logger := r.Context().Value(LoggerContextKey{}).(*zap.Logger)
start := time.Now()
processData(r.Context())
// Log outcome with duration and business metrics.
logger.Info("data processed",
zap.Duration("duration", time.Since(start)),
zap.Int("rows", 42),
)
}
The handler retrieves the logger from the context. It asserts the type. If the middleware didn't run, the assertion panics. In production code, use a type assertion with comma-ok to handle missing values gracefully. The handler logs the result with duration and business metrics. These fields become queryable in the aggregator.
Context carries the logger. Fields travel with the request.
Schema consistency and sensitive data
Structured logging fails if your keys are inconsistent. One handler logs user_id, another logs userId. The aggregator sees two different fields. You can't query across them. Define a naming convention. Use snake_case or camelCase consistently. Document the fields. Treat your log schema like a database schema. Changes to fields should be reviewed.
Sensitive data leaks are a common risk. Developers log request bodies. Passwords, tokens, and PII end up in logs. Logs are often less secure than the application database. Mask sensitive fields before logging. Use a custom encoder or a helper function to scrub data. zap allows custom field creators. Wrap sensitive values. Never log raw credentials. The cost of logging also matters. High-volume services generate terabytes of logs. Logging every request at debug level burns money. Use log levels wisely. Sample logs in high-throughput paths. zap supports sampling. Configure sampling to drop logs after a threshold. Protect your wallet.
Shipping logs without coupling
Your Go code never talks to ELK or Datadog. It writes to stdout. The container runtime captures stdout. A log collector agent reads the stream. The agent parses JSON and ships it. This pattern follows the 12-factor app methodology. Your app is stateless regarding logs. If you switch backends, you reconfigure the agent, not the app. In Kubernetes, the sidecar pattern or daemonset agent handles this. In Docker, the logging driver does the work. Focus on JSON output. Let the infrastructure handle transport.
Convention aside: gofmt is mandatory. Don't argue about indentation; let the tool decide. Most editors run it on save. Your log code should follow the same formatting rules as the rest of the codebase. Consistency reduces cognitive load.
Pitfalls and runtime traps
Forgetting to flush the logger causes lost logs. zap buffers writes for performance. If the process crashes, the buffer empties. Call logger.Sync() before exit. The compiler won't stop you, but the logs will vanish.
Passing the logger by value vs pointer. zap.Logger is safe to copy. You can pass *zap.Logger or zap.Logger. The convention is to pass the pointer or clone. Cloning is preferred for adding fields.
Context value retrieval. If you forget to set the value, the assertion panics. Use type assertion with comma-ok: logger, ok := ctx.Value(key).(*zap.Logger). If ok is false, handle the error.
If you pass a non-serializable type to zap, the encoder panics. zap expects types it knows how to encode. Custom structs need a MarshalLogObject method. The compiler won't catch this. The runtime panics with panic: runtime error: invalid memory address or nil pointer dereference inside the encoder. Define MarshalLogObject for custom types to return a zapcore.ObjectEncoder. The compiler rejects the code with cannot use myStruct (type MyStruct) as zapcore.ObjectMarshaler in argument if you try to pass it directly without the method.
Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Always have a cancellation path. If you spawn a goroutine to log asynchronously, ensure it respects context cancellation. The worst goroutine bug is the one that never logs.
Flush on exit. Mask secrets. Sample at scale.
Choosing a logger
Use zap when you need maximum throughput and low latency in high-scale services.
Use slog when you want a standard library solution with zero dependencies and acceptable performance for most workloads.
Use zerolog when you are building CLI tools or embedded systems where binary logging size matters more than human readability.
Use logrus when you are maintaining legacy codebases that already depend on it and migration cost outweighs performance gains.
Use plain log when you are writing a simple script or tool where structured fields add no value.
Pick the logger that matches your scale. JSON is the format, not the library.