When the runtime changes its mind
You deploy a Go service. It runs smoothly for months. You update the Go version in your CI pipeline. The tests pass. You deploy the new binary. Suddenly, your HTTP client starts timing out on upstream calls, or a nil pointer panic that used to be caught is now crashing the process with a cryptic message. The code didn't change. The runtime did.
Go occasionally changes internal behavior to improve performance or fix bugs. Sometimes those changes break edge cases in your application. You need a way to tell the runtime, "Please behave like the old version," or "Please show me exactly where this is going wrong." Go provides two tools for this: the GODEBUG environment variable and the //go:debug directive. These let you toggle runtime flags, opt out of breaking changes, and enable internal tracing without rewriting your code.
Runtime knobs, not app config
GODEBUG and //go:debug control the Go runtime, not your application logic. They are switches for internal mechanisms like the garbage collector, the scheduler, the HTTP client, and panic handling. You use them to debug runtime issues, mitigate breaking changes, or gather performance data.
Think of these as diagnostic ports on the engine. You don't use them to steer the car. You use them to check the oil pressure or disable a turbocharger that's acting up. Your application configuration handles business rules and feature flags. GODEBUG handles the machinery underneath.
The environment variable GODEBUG applies globally when you start the process. You set it in the shell or your deployment manifest. The //go:debug directive bakes settings directly into the binary during compilation. You add it to your source files. The directive is useful when you want a specific release to always run with a certain flag, regardless of the environment.
Minimal example: panic tracing
Here's the simplest way to use a debug flag. The panicnil flag controls how the runtime reports nil pointer panics. When enabled, it prints a stack trace for the panic, which makes it much easier to find the source of the bug.
//go:debug panicnil=1
package main
import "fmt"
func main() {
// panicnil=1 enables stack traces for nil pointer panics.
// This helps locate the exact line causing a nil dereference.
// The directive is baked into the binary, so this flag is always active.
var p *int
// This panics. With panicnil=1, the runtime prints a full stack trace.
// Without it, you might only see "runtime error: invalid memory address".
fmt.Println(*p)
}
The //go:debug directive must appear before the package clause. If you place it after the package declaration, the compiler rejects the file with //go:debug must appear before the package clause. The compiler processes these directives during the build, so the settings are fixed in the resulting binary.
How the flags work
When you run a Go program, the runtime checks for GODEBUG at startup. It parses the value as a comma-separated list of key=value pairs. Each key corresponds to an internal flag. The runtime applies the values and continues. If you set a flag that doesn't exist, the runtime ignores it. You can verify active flags by printing the value of GODEBUG inside your program, though the runtime doesn't expose a list of supported flags.
The //go:debug directive works differently. The compiler reads the directive and embeds the flag into the binary. When the binary runs, those flags are active as if they were set via GODEBUG. You can still override //go:debug settings by setting GODEBUG in the environment. The environment variable takes precedence.
This precedence is intentional. It lets you bake in safe defaults with directives while retaining the ability to toggle behavior at launch for debugging.
Realistic example: HTTP/2 issues
HTTP/2 can cause subtle issues with some servers or proxies. Multiplexing, header compression, and stream priority can interact poorly with buggy upstream services. If your HTTP client starts failing after a Go upgrade, you might want to disable HTTP/2 to isolate the problem.
//go:debug http2client=0
package main
import (
"fmt"
"net/http"
)
func main() {
// http2client=0 disables HTTP/2 for the HTTP client.
// This forces all outgoing requests to use HTTP/1.1.
// Use this to debug issues caused by HTTP/2 multiplexing or server bugs.
resp, err := http.Get("https://example.com")
if err != nil {
fmt.Println("Request failed:", err)
return
}
fmt.Println("Status:", resp.Status)
}
In a deployment script, you might prefer GODEBUG so you can toggle the flag without rebuilding.
# Disable HTTP/2 for the client and enable GC tracing.
# This helps correlate GC pauses with network latency spikes.
export GODEBUG=http2client=0,gctrace=1
# Run the service with the debug flags active.
# The runtime applies these settings before main() starts.
./my-service
The gctrace=1 flag prints garbage collection statistics to stderr. Each line shows the GC cycle number, the amount of memory allocated, the pause time, and other metrics. This output is invaluable when you suspect GC pauses are causing latency spikes. You can pipe the output to a log file and analyze the pause times alongside your application metrics.
Common flags and what they do
The set of available flags grows with each Go release. Some flags are stable across versions. Others are temporary workarounds for specific issues. Here are the most useful ones for debugging and performance analysis.
The gctrace flag enables GC tracing. It prints detailed stats for every GC cycle. Use it when you need to understand GC behavior. The output includes the cycle number, allocation rate, pause time, and heap size. High pause times indicate the GC is struggling. You might need to tune GOGC or reduce allocation pressure.
The schedtrace flag prints scheduler state at regular intervals. You set the interval in milliseconds. For example, schedtrace=1000 prints state every second. The output shows the number of goroutines, running goroutines, waiting goroutines, and blocked syscalls. Use this when goroutines seem stuck or the scheduler is underutilized.
The asyncpreemptoff flag disables asynchronous preemption. In Go 1.14 and later, the scheduler can preempt goroutines asynchronously to improve latency. This can cause deadlocks in code that assumes preemption only happens at safe points. Setting asyncpreemptoff=1 reverts to synchronous preemption. Use this to debug deadlocks that appear after upgrading Go.
The http2client and http2server flags control HTTP/2 support. http2client=0 disables HTTP/2 for the client. http2server=0 disables HTTP/2 for the server. Use these to isolate HTTP/2 issues or force HTTP/1.1 for compatibility.
The tls13ticket flag controls TLS 1.3 ticket resumption. tls13ticket=0 disables ticket resumption. Use this if you encounter TLS handshake issues with certain servers.
The panicnil flag controls nil pointer panic reporting. panicnil=1 enables stack traces. Use this to debug nil pointer panics.
The gocover flag controls coverage instrumentation. gocover=0 disables coverage. Use this to speed up builds when you don't need coverage data.
The runtime documentation lists all available flags. The list changes over time. Check the release notes for new flags and deprecations.
Pitfalls and runtime errors
Debug flags are powerful, but they come with risks. Some flags disable optimizations or safety checks. Running with the wrong flags can mask bugs or degrade performance. Never use debug flags in production unless you have a specific reason. Treat them as temporary tools.
The //go:debug directive must appear at the top of the file. The compiler enforces strict ordering. If you place the directive after the package clause or after imports, the compiler rejects the file with //go:debug must appear before the package clause. This error is common when you copy a directive from the middle of a file. Always put directives before the package declaration.
//go:debug panicnil=1
package main
import "fmt"
func main() {
// Directives must be before the package clause.
// This file compiles successfully.
fmt.Println("OK")
}
Environment variables can be a security risk if you expose internal state. Flags like gctrace and schedtrace print sensitive runtime information to stderr. If your logging system captures stderr and sends it to a public dashboard, you might leak internal metrics. Ensure your debug output goes to secure logs.
Some flags have performance costs. gctrace and schedtrace generate output on every cycle or interval. This I/O can slow down the program. Disable tracing when you're done debugging.
The asyncpreemptoff flag can increase latency. Synchronous preemption waits for a safe point before switching goroutines. This can delay context switches. Only use this flag to debug deadlocks. Re-enable async preemption once you fix the issue.
Convention aside: Go environment variables are uppercase by convention. The GODEBUG variable follows this rule. The flags inside the value are lowercase key-value pairs. This distinction helps you recognize runtime variables at a glance.
Decision: when to use each tool
Use GODEBUG when you need to toggle runtime behavior at launch without recompiling. Use GODEBUG for debugging sessions, performance profiling, or temporary workarounds in staging environments. The environment variable lets you change flags on the fly.
Use //go:debug when you want to bake a debug setting into a specific binary for a release. Use //go:debug when you need a flag to be active by default, such as disabling HTTP/2 for a known-buggy upstream service. The directive ensures the flag is always present, even if the environment is misconfigured.
Use application-level configuration when you need to control business logic or feature flags. Use config files, flags, or environment variables for your app settings. Never use GODEBUG to toggle application behavior. The runtime flags are for the runtime, not your code.
Use standard profiling tools like pprof when you need detailed performance analysis. GODEBUG provides high-level tracing. pprof gives you CPU profiles, memory profiles, and block profiles. Combine GODEBUG with pprof for a complete picture.
GODEBUG is a scalpel, not a hammer. Directives bake in behavior; env vars toggle it. Trust the runtime flags for runtime problems, and keep your app config separate.