The ticket that connects the dots
You have two services. Service A accepts a user request and calls Service B to check inventory. Service B fails. You see an error log in Service A and a panic log in Service B. The timestamps are close, but you have no proof they are the same request. You are staring at two separate log files, guessing which error belongs to which user.
This is the distributed tracing problem. The solution is a trace context: a unique identifier that travels with the request across service boundaries. Think of it like a ticket number at a government office. You get a number when you arrive. You hand that number to every clerk you visit. When you leave, the office can replay your entire journey by looking up the number.
In Go, the ticket lives in context.Context. The context is a tree structure that flows down through your call stack. When a request arrives, the context carries the trace ID. When you make an outgoing call, you must copy that trace ID into the request headers so the next service can continue the chain.
Go does not do this automatically. The standard library keeps context.Context generic. It handles deadlines and cancellation, but it knows nothing about tracing. You need a library to pack the trace data into headers and unpack it on the other side. OpenTelemetry is the industry standard for this work. The otelhttp package provides wrappers that handle the header dance for HTTP services.
How trace context moves
The W3C Trace Context standard defines the format for these headers. The primary header is traceparent. It contains the trace ID, the current span ID, and flags. It looks like a hex string: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01.
When Service A receives a request, it reads traceparent from the headers. It creates a new span for its work and stores that span in the request context. When Service A calls Service B, it writes the updated traceparent into the outgoing headers. Service B reads the header, extracts the trace ID, and creates a child span.
The result is a single trace that spans multiple services. You can see the full path of a request in your observability UI. You can measure latency between services. You can find the exact error that broke the chain.
The mechanism relies on context.Context. Go functions that participate in a trace must accept a context as their first parameter. The convention is to name it ctx. The context carries the active span. If you pass ctx to a database driver or an HTTP client, the instrumentation can read the span and attach the trace ID to the operation.
Minimal server setup
Here is the simplest way to wrap an HTTP handler. The otelhttp.NewHandler function extracts trace headers from incoming requests and creates a span. It injects the span into the request context so your handler can access it.
package main
import (
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
// processOrder handles the business logic for an order.
// It receives a context that already contains the active trace span.
func processOrder(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// ctx holds the span. Any child goroutines or DB calls
// should pass this ctx to continue the trace.
w.WriteHeader(http.StatusOK)
}
func main() {
// Wrap the handler to extract trace headers and start a span.
// The handler name "processOrder" appears in the trace UI.
wrappedHandler := otelhttp.NewHandler(http.HandlerFunc(processOrder), "processOrder")
http.ListenAndServe(":8080", wrappedHandler)
}
The wrapper does three things. It reads the traceparent header from the request. It creates a new span and links it to the parent trace. It stores the span in r.Context(). Your handler code does not change. You just call r.Context() to get the context with the span.
Minimal client setup
Outgoing calls need the reverse operation. The client must read the active span from the context and write it into the request headers. The otelhttp.NewTransport wrapper handles this injection.
package main
import (
"context"
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
// callInventory checks stock levels in a separate service.
// It passes the request context to propagate the trace.
func callInventory(ctx context.Context) error {
// Wrap the transport to inject trace context headers.
// This copies the current span ID and trace ID into the request headers.
transport := otelhttp.NewTransport(http.DefaultTransport)
client := &http.Client{Transport: transport}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://inventory/api", nil)
if err != nil {
return err
}
// The transport reads the span from ctx and adds headers.
// The downstream service will extract these headers to continue the trace.
_, err = client.Do(req)
return err
}
The transport reads the span from ctx. It formats the traceparent header and attaches it to the request. If there is no active span in the context, the transport does nothing. The downstream service will start a new trace. This is correct behavior. A new trace starts when there is no parent.
Realistic service chain
Real code involves structs, error handling, and multiple calls. The pattern remains the same. Pass ctx everywhere. Wrap handlers and transports. The trace flows automatically.
package main
import (
"context"
"errors"
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)
// InventoryClient wraps the HTTP client for inventory calls.
// It holds the transport with trace injection enabled.
type InventoryClient struct {
client *http.Client
}
// NewInventoryClient creates a client ready for trace propagation.
// The receiver name "c" follows Go convention for short names.
func NewInventoryClient() *InventoryClient {
transport := otelhttp.NewTransport(http.DefaultTransport)
return &InventoryClient{
client: &http.Client{Transport: transport},
}
}
// CheckStock queries the inventory service for an item.
// It returns an error if the item is unavailable.
func (c *InventoryClient) CheckStock(ctx context.Context, itemID string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://inventory/check/"+itemID, nil)
if err != nil {
return err
}
resp, err := c.client.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return errors.New("inventory check failed")
}
return nil
}
// handleCheckout processes a payment and checks inventory.
// It demonstrates propagating context through a chain of calls.
func handleCheckout(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
// ctx contains the span for this HTTP request.
// All downstream calls must receive this ctx.
client := NewInventoryClient()
err := client.CheckStock(ctx, "item-123")
if err != nil {
// Log the error. The logger should automatically include
// the trace ID if configured with OpenTelemetry.
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
func main() {
handler := otelhttp.NewHandler(http.HandlerFunc(handleCheckout), "handleCheckout")
http.ListenAndServe(":8080", handler)
}
The InventoryClient struct holds the client. The receiver name is c, matching the type initial. This is a Go convention. Receiver names should be short and consistent. The CheckStock method takes ctx as the first parameter. It creates a request with http.NewRequestWithContext. The transport injects the trace headers. The downstream service receives the context.
Error handling uses the standard if err != nil pattern. Go makes the unhappy path visible. The verbosity is a feature. You see every error site. If your logger is configured with OpenTelemetry, it reads the trace ID from ctx and attaches it to the log entry. You can search logs by trace ID.
Pitfalls and silent breaks
Tracing breaks when the context stops flowing. The most common bug is creating a new context inside a handler. If you call context.Background() instead of using r.Context(), you sever the trace. The span is lost. The downstream calls appear as orphans in the trace UI.
Another trap is goroutines. If you spawn a goroutine, you must pass ctx to it. If you capture ctx in a closure, you are safe. If you start a goroutine without ctx, the goroutine has no trace. Any work done in that goroutine is invisible to the trace.
Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Always have a cancellation path. Use ctx.Done() to detect cancellation. If the parent request is cancelled, the child goroutine should exit. The worst goroutine bug is the one that never logs.
Compiler errors catch some mistakes. If you forget to import otelhttp, the compiler rejects the program with undefined: otelhttp. If you pass the wrong type to a function, you get a type mismatch error like cannot use ctx (variable of type context.Context) as string value in argument. These errors stop you from running broken code.
Runtime panics are harder. If you pass a nil context to a function that expects a valid context, the function might panic when it tries to read values. Always check for nil contexts at boundaries. The convention is that ctx is never nil. If a function requires a context, the caller must provide one.
Decision matrix
Use otelhttp.NewHandler when you need automatic extraction on incoming HTTP requests. Use otelhttp.NewTransport when you need automatic injection on outgoing HTTP calls. Use manual propagator.Inject when you are using a custom protocol like gRPC or AMQP that does not have an official instrumentation package. Use plain http.Handler without wrapping when the service is internal-only and you do not care about distributed traces. Use context.WithValue sparingly to attach non-tracing data to the context; prefer function parameters for business data.
Where to go next
- How to Implement Distributed Tracing in Go with OpenTelemetry
- How to Use logrus for Structured Logging in Go
- How to Implement Health Checks for Microservices in Go
Context is plumbing. Run it through every long-lived call site. Tracing is free until you forget the context.