Fix: Tests Pass Locally but Fail in CI

The "Works on My Machine" Trap

You run go test ./... in your terminal. Every test passes. You push the commit. The CI runner spins up, executes the exact same test suite, and fails with a panic or an assertion error. The code hasn't changed. The logic is identical. The difference lives in the invisible layer between your machine and the build server. This mismatch is the most common source of flaky tests in Go.

Go programs are compiled binaries, which suggests they should behave identically everywhere. That's mostly true. The Go runtime and compiler, however, read environment variables to adjust behavior. GODEBUG controls runtime internals like garbage collection and network timeouts. CGO_ENABLED decides whether to link C code. If your local machine has GODEBUG=asyncpreemptoff=1 and CI doesn't, the scheduler behaves differently. If CI has a different version of glibc or lacks gcc, C-dependent code breaks. The binary is the same, but the knobs turned during build and run differ.

Environment variables are silent configuration. Make them loud.

How environment variables steer Go

Go uses a set of environment variables to configure the toolchain and runtime. These variables act like hidden switches. Most are off by default. Flipping one changes how the engine handles memory, networks, or scheduling. The two most dangerous variables for test consistency are GODEBUG and CGO_ENABLED.

GODEBUG accepts a comma-separated list of key=value pairs. The runtime parses this string at startup and toggles internal flags. Common flags include:

asyncpreemptoff=1: Disables asynchronous preemption. Goroutines are only preempted at function calls. This changes scheduling timing and can expose or hide race conditions.
nethttpmuxgo121=1: Enables the stricter HTTP path matching introduced in Go 1.21. Trailing slashes are handled differently.
panicnil=1: Changes panic(nil) to panic with a non-nil error. This affects error handling logic.
sigpipe=1: Restores the pre-Go 1.17 behavior of sending SIGPIPE on broken network writes.

CGO_ENABLED controls whether the compiler allows C code. When set to 1, the compiler invokes the C toolchain. When set to 0, cgo is disabled. Many CI pipelines set CGO_ENABLED=0 to speed up builds and produce statically linked binaries. If your code depends on a package that uses cgo, the build fails in CI but succeeds locally.

GOOS and GOARCH determine the target platform. If CI runs on Linux and you develop on macOS, the binary executes different assembly routines. File path separators, case sensitivity, and system calls differ. Tests that assume Windows paths fail on Linux. Tests that assume case-insensitive file systems fail on case-sensitive ones.

Think of GODEBUG as a debug menu in a video game. You can toggle physics, gravity, or frame rate. The game is the same, but the experience changes. If you play with different settings, your high score might not transfer.

Probing the environment

The first step in diagnosing a mismatch is to compare the environment. Run go env in both your local terminal and the CI logs. Look for differences in GODEBUG, CGO_ENABLED, GOOS, GOARCH, and GOFLAGS.

You can also write a test that logs these values. This makes the environment visible in the test output.

package main

import (
	"os"
	"runtime"
	"testing"
)

// TestEnvProbe logs environment variables that affect build and runtime behavior.
// Run this test to compare output between local and CI environments.
func TestEnvProbe(t *testing.T) {
	// runtime.Version() reveals the Go version used to build the binary.
	// Mismatches here often cause subtle behavior changes.
	t.Logf("Go version: %s", runtime.Version())

	// GODEBUG can alter runtime behavior significantly.
	// A local setting like GODEBUG=nethttpmuxgo121=1 changes HTTP routing.
	godebug := os.Getenv("GODEBUG")
	t.Logf("GODEBUG: %q", godebug)

	// CGO_ENABLED affects compilation.
	// If this is 0, any code using cgo will fail to compile.
	cgo := os.Getenv("CGO_ENABLED")
	t.Logf("CGO_ENABLED: %q", cgo)

	// GOOS and GOARCH determine the target platform.
	// Path separators and file system behavior depend on these values.
	t.Logf("GOOS: %s, GOARCH: %s", runtime.GOOS, runtime.GOARCH)
}

This test doesn't fix the problem. It reveals the difference. Once you see that CI has CGO_ENABLED=0 and local has CGO_ENABLED=1, you know where to look.

Comparison is the cure for mystery. Run go env everywhere.

How the compiler and runtime diverge

When you run go test, the toolchain compiles your test binary. During compilation, the compiler checks CGO_ENABLED. If the value is 0, the compiler strips out all C dependencies. If your code imports a package that uses cgo, the build fails with an error like # command-line-arguments: relocation target C.printf not defined. This error means the compiler tried to link a C symbol but couldn't find it because cgo was disabled.

If CGO_ENABLED is 1, the compiler invokes the C compiler. The result depends on system libraries like gcc and glibc. If CI lacks these libraries, the build fails with exec: "gcc": executable file not found in $PATH. The compiler needs the C toolchain to compile cgo code.

At runtime, the Go runtime parses GODEBUG. This variable accepts key=value pairs that toggle internal debugging flags. For example, GODEBUG=asyncpreemptoff=1 disables asynchronous preemption in the scheduler. This changes how goroutines are interrupted. A test that relies on precise timing or goroutine scheduling might pass with async preemption on but fail when it's off. The runtime also reads GOOS and GOARCH to select platform-specific code paths. If CI runs on a different architecture, the binary executes different assembly routines.

The compiler and runtime are sensitive to their environment. Treat environment variables as part of your codebase. Lock them down.

The HTTP mux mismatch

A realistic example involves the net/http package. Go 1.21 introduced a stricter path matching algorithm for the default ServeMux. The new algorithm rejects trailing slashes by default. If you have a handler for /api/users, a request to /api/users/ returns 404.

Go provides a GODEBUG flag to opt out of the new behavior. GODEBUG=nethttpmuxgo121=0 restores the old routing. If your local machine has this flag set, tests that send requests with trailing slashes pass. If CI doesn't have the flag, the tests fail.

package main

import (
	"net/http"
	"net/http/httptest"
	"testing"
)

// TestHTTPMuxBehavior demonstrates how GODEBUG affects HTTP routing.
// Go 1.21 introduced a stricter path matching algorithm.
func TestHTTPMuxBehavior(t *testing.T) {
	mux := http.NewServeMux()

	// Register a handler for /api/users.
	mux.HandleFunc("/api/users", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	// Create a test request with a trailing slash.
	req := httptest.NewRequest(http.MethodGet, "/api/users/", nil)

	rec := httptest.NewRecorder()
	mux.ServeHTTP(rec, req)

	// In Go 1.21+, the new mux rejects the trailing slash by default.
	// If GODEBUG=nethttpmuxgo121=0 is set locally, this passes.
	// In CI without that flag, this fails with a 404.
	if rec.Code != http.StatusOK {
		t.Errorf("expected 200, got %d", rec.Code)
	}
}

The fix is to lock the GODEBUG setting in go.mod. The godebug directive ensures the runtime uses the same flags regardless of the environment.

godebug (
	nethttpmuxgo121=1
)

This directive goes in the go.mod file. It applies to all builds and runs. CI and local machines now use the same routing behavior. The test passes everywhere.

Lock the runtime with godebug. Don't let CI surprise you.

Pitfalls and compiler signals

Several pitfalls cause tests to pass locally but fail in CI. The compiler and runtime provide signals to help you diagnose them.

The race detector changes memory layout and timing. Running go test -race adds instrumentation to every memory access. This slows down the program and changes scheduling. Tests that rely on timing might pass without the race detector but fail with it. The race detector also catches data races. If a test has a race, it might pass locally by chance but fail in CI due to different timing. Always run tests with -race in CI. If a test fails with -race, fix the race. Don't disable the detector.

Path differences cause failures on Windows versus Linux. Windows uses backslashes and is case-insensitive. Linux uses forward slashes and is case-sensitive. A test that reads config.json might fail if the file is named Config.json. The compiler rejects undefined variables with undefined: Config. The runtime returns open Config.json: no such file or directory if the file doesn't exist. Use filepath.Join to build paths. It handles separators automatically. Use strings.ToLower or strings.ToUpper to normalize case when needed.

C dependencies require the C toolchain. If CI sets CGO_ENABLED=0, cgo code fails to compile. The compiler rejects the build with # command-line-arguments: relocation target C.printf not defined. This error means the compiler tried to link a C symbol but couldn't find it. Fix this by setting CGO_ENABLED=1 in CI or installing the required system libraries. If you don't need cgo, remove the dependency. cgo adds complexity and reduces portability.

The compiler complains with imported and not used if you import a package but don't use it. This error is strict. It prevents dead code. If a test imports a package for side effects, use a blank identifier to suppress the error. import _ "package" tells the compiler you intentionally imported the package.

Convention aside: t.Setenv is preferred over os.Setenv in tests. t.Setenv returns a cleanup function that restores the original value after the test. This prevents environment variables from leaking to other tests. Always call the cleanup function or use the return value. t.Setenv keeps tests isolated.

The worst test bug is the one that depends on timing. Fix the race, don't add a sleep.

When to apply which fix

Use go env to compare GODEBUG, CGO_ENABLED, and GOOS between local and CI when tests fail only in one environment.

Use a godebug directive in go.mod when you need to enforce consistent runtime flags like nethttpmuxgo121 across all build environments.

Use CGO_ENABLED=1 and install system dependencies like gcc in CI when your code imports packages that use cgo.

Use t.Setenv in test functions when a test requires a specific environment variable that should not leak to other tests.

Use go test -race in CI when you want to catch data races and ensure tests are safe under instrumentation.

Use filepath.Join when building file paths to handle separator differences between operating systems.

Consistency is the goal. Lock the environment, not the test.

Where to go next

Your code behaves differently because the computer running your tests (CI) has different settings than your own computer. Think of it like a recipe that works in your kitchen but fails in a restaurant because the oven temperature or ingredients are slightly different. You need to explicitly tell the CI system to use the exact same settings you use locally.