Shell magic vs Go plumbing
You write a shell script that chains ls -l | grep test | sort. It works instantly. The shell creates three processes, builds two invisible pipes, connects the output of one to the input of the next, starts everything, and waits. You get a filtered, sorted list.
You try to do the same in Go and realize there is no | operator for processes. Go does not hide the plumbing. You must create the pipe, wire the ends, start the processes, and manage the lifecycle. Go gives you the tools to build the pipeline, but you hold the wrench. This explicit control prevents silent failures and gives you precise error handling, but it requires understanding how data flows between processes.
The io.Pipe tube
Shell piping relies on OS pipes. Go provides io.Pipe for in-memory piping between io.Reader and io.Writer interfaces. io.Pipe returns a coupled pair: a *io.PipeReader and a *io.PipeWriter. Data written to the writer becomes available to the reader.
The critical detail is synchronization. io.Pipe is synchronous. Writing to the pipe blocks if the internal buffer is full and no one is reading. Reading blocks if the buffer is empty and no one is writing. This behavior provides natural backpressure. If the consumer is slow, the producer slows down automatically. You never need a massive buffer to prevent memory exhaustion. The pipe regulates the flow.
io.Pipe works with exec.Cmd because exec.Cmd exposes Stdout as an io.Writer and Stdin as an io.Reader. You assign the pipe writer to the first command's Stdout and the pipe reader to the second command's Stdin. The data flows from the first process, through the Go pipe, into the second process.
io.Pipe blocks the writer. Backpressure is automatic.
Minimal pipeline
Here's the skeleton: create two commands, make a pipe, wire the ends, start both, wait.
package main
import (
"io"
"os/exec"
)
func main() {
// cmd1 generates data; this process writes to stdout
cmd1 := exec.Command("ls", "-l")
// cmd2 consumes data; this process reads from stdin
cmd2 := exec.Command("grep", "test")
// io.Pipe creates a synchronous in-memory pipe; writer blocks if buffer fills and reader isn't active
reader, writer := io.Pipe()
// cmd1.Stdout is an io.Writer; assigning the pipe writer routes all output into the tube
cmd1.Stdout = writer
// cmd2.Stdin is an io.Reader; assigning the pipe reader feeds data from the tube into the process
cmd2.Stdin = reader
// Start launches the OS process and returns immediately; both must run concurrently to avoid deadlock
cmd1.Start()
cmd2.Start()
// Wait blocks until cmd1 exits; this ensures all data is written before closing the pipe
cmd1.Wait()
// Closing the writer sends EOF to the reader; cmd2 must see EOF to know the stream is done
writer.Close()
// Wait collects the exit status of cmd2; check this for processing errors
cmd2.Wait()
}
Execution trace
The program runs in a specific sequence. cmd1.Start() forks a new OS process running ls. The Go runtime redirects the child's standard output to the pipe writer. cmd2.Start() forks another process running grep. The runtime redirects its standard input to the pipe reader. Both processes run concurrently.
ls writes directory entries to its stdout. The data enters the pipe buffer. grep reads from its stdin. The data leaves the pipe buffer. grep filters lines. ls finishes and exits. cmd1.Wait() returns in the Go program. You call writer.Close(). This closes the write end of the pipe. grep attempts to read more data, hits the closed end, and receives an EOF. grep finishes processing and exits. cmd2.Wait() returns.
If you use cmd1.Run() instead of Start(), the program deadlocks. Run() starts the process and waits for it to finish. ls writes to the pipe. The pipe buffer fills. ls blocks waiting for the buffer to drain. cmd2 never starts because cmd1.Run() hasn't returned. The program hangs forever. The compiler cannot detect this logic error. The error is silence.
Start both processes. Wait for both. Close the pipe.
Robust pipeline with errors
Real code handles failures. Commands can fail to start, exit with errors, or hang. You need to wrap errors, clean up processes, and return the first failure. The convention is to check errors immediately and return. Verbose error handling makes the unhappy path visible.
// PipeCommands connects two commands via io.Pipe and handles errors.
// This function ensures the pipe closes even if cmd1 fails.
func PipeCommands(cmd1, cmd2 *exec.Cmd) error {
// io.Pipe creates the coupled reader and writer
pr, pw := io.Pipe()
cmd1.Stdout = pw
cmd2.Stdin = pr
// Start cmd1; fail fast if the process cannot launch
if err := cmd1.Start(); err != nil {
return fmt.Errorf("start cmd1: %w", err)
}
// Start cmd2; kill cmd1 if cmd2 fails to launch to prevent orphans
if err := cmd2.Start(); err != nil {
cmd1.Process.Kill()
return fmt.Errorf("start cmd2: %w", err)
}
// Wait for cmd1 to finish writing data
err1 := cmd1.Wait()
// Close writer to signal EOF; cmd2 needs this to exit
pw.Close()
// Wait for cmd2 to finish processing
err2 := cmd2.Wait()
// Return the first error encountered
if err1 != nil {
return err1
}
return err2
}
The pw.Close() call is mandatory. If cmd1 crashes, cmd1.Wait() returns an error, but cmd2 is still running and reading from the pipe. If you don't close the writer, cmd2 waits for data that never comes. The process leaks. The worst pipe bug is the one that hangs without a stack trace.
Convention aside: if err != nil { return err } is verbose by design. The community accepts the boilerplate because it makes error paths explicit. Wrapping with fmt.Errorf("...: %w", err) preserves the error chain for debugging.
Pitfalls and silent failures
Piping commands introduces several failure modes. The compiler helps with type errors but cannot catch runtime logic issues.
If you assign the wrong type to Stdout, the compiler rejects the program with cannot use writer (type *os.File) as io.Writer value in assignment. This is a compile-time safety net. Runtime errors are harder.
Forgetting to close the pipe writer causes the reader to block indefinitely. The downstream process never sees EOF. It waits for more input. The goroutine or process leaks. Always close the writer after the producer finishes, even if the producer fails.
Piping Stderr is rarely what you want. cmd.Stderr defaults to inheriting from the Go program, which usually sends errors to the terminal. If you pipe Stderr, error messages mix with data. The consumer might treat an error message as valid input. Keep Stderr separate unless you have a specific reason to merge streams.
Long-running pipelines need cancellation. exec.CommandContext creates a command tied to a context.Context. If the context cancels, the process receives a signal and terminates. HTTP handlers and background workers should always use CommandContext. Context is plumbing. Run it through every long-lived call site.
The worst goroutine bug is the one that never logs. The same applies to leaked processes. Monitor exit statuses and close pipes explicitly.
Decision matrix
Use io.Pipe when you need to connect the output of one process to the input of another in memory.
Use cmd.Stdout = os.Stdout when you want the command output to appear directly in the terminal without piping.
Use exec.CommandContext when the pipeline might run for a long time and needs a cancellation deadline.
Use a single exec.Command with Run when you don't need to chain processes and just want to execute a tool.
Use os.Pipe when you need file descriptors for advanced OS-level control, though io.Pipe covers 99% of use cases.
Trust the pipe. Connect the hoses.