The tube between goroutines
You are building a tool that needs to compress a stream of data and send it over the network, or maybe you are launching a subprocess and need to feed it input while reading its output. You do not want to write everything to a temporary file. You want the output of one stage to flow directly into the input of the next, just like cat file | gzip > out.gz in the shell. In Go, you get that same streaming behavior with os.Pipe.
A pipe is a one-way street in memory. One goroutine writes to the street, another reads from it. The operating system handles the buffering. If the writer is faster, the reader catches up. If the reader is faster, the writer blocks until data arrives. It is a synchronization primitive disguised as I/O.
os.Pipe creates a kernel-backed pipe and returns two file descriptors: one for reading, one for writing. The read end implements io.Reader. The write end implements io.Writer. You can pass these to any function that expects those interfaces, including io.Copy, os/exec commands, or compression writers.
Minimal pipe example
Here is the simplest pipe: create it, spawn a writer, read until EOF.
package main
import (
"fmt"
"io"
"os"
)
func main() {
// os.Pipe creates a connected pipe in the kernel.
// It returns a reader file and a writer file.
reader, writer, err := os.Pipe()
if err != nil {
panic(err)
}
// Run the writer in a goroutine to avoid deadlock.
// The main goroutine will read, so the writer needs its own thread of execution.
go func() {
// Defer close ensures the pipe closes even if the write panics.
// Closing the writer signals EOF to the reader.
defer writer.Close()
writer.Write([]byte("Hello from the pipe"))
}()
// ReadAll blocks until the writer closes the pipe or returns an error.
// It consumes the entire stream into memory.
data, err := io.ReadAll(reader)
if err != nil {
panic(err)
}
fmt.Println(string(data))
// Close the reader to release the file descriptor.
reader.Close()
}
The writer runs in a goroutine because Write blocks if the pipe buffer is full, and ReadAll blocks if the buffer is empty. If both ran in the same goroutine, the program would deadlock immediately. The defer writer.Close() is critical. io.ReadAll waits for EOF. EOF happens when the writer closes the pipe. Without that close, the reader hangs forever.
Pipes block until data flows. Always close the writer to unblock the reader.
How the kernel handles the flow
When you call os.Pipe, the OS allocates a circular buffer in kernel space. The size is usually 64KB on Linux, but you should not rely on the exact size. The buffer decouples the writer and reader slightly. The writer can push data into the buffer and return, even if the reader has not started yet. Once the buffer fills, the writer blocks until the reader drains some bytes.
If the reader reads faster than the writer, the reader blocks when the buffer is empty. This blocking is efficient. The OS puts the goroutine to sleep and wakes it when data arrives. No busy-waiting.
Error handling follows the standard Go convention. The community accepts the boilerplate of if err != nil because it makes the unhappy path visible. Pipes fail, and you need to know when they do. If you forget to capture the error from os.Pipe, the compiler rejects the program with error returned but not handled. If you pass the wrong type to a function expecting an io.Reader, the compiler complains with cannot use writer (variable of type *os.File) as io.Reader value in argument.
Trust the pipe buffer, but respect the close.
Streaming with io.Copy
Real code rarely reads everything into memory. Use io.Copy to stream data from the pipe to a destination without buffering the whole payload. io.Copy reads from the source, writes to the destination, and reuses a small internal buffer. It handles chunking automatically.
func streamChunks() {
// Create a pipe for streaming data between stages.
pr, pw, err := os.Pipe()
if err != nil {
panic(err)
}
// Writer goroutine simulates a data source producing chunks.
go func() {
defer pw.Close()
// Write in small chunks to demonstrate streaming behavior.
// The pipe buffers data in the kernel until the reader consumes it.
for i := 0; i < 5; i++ {
pw.Write([]byte(fmt.Sprintf("Chunk %d\n", i)))
}
}()
// io.Copy reads from pr and writes to os.Stdout until EOF.
// It handles buffering and chunking automatically, making it efficient for large streams.
if _, err := io.Copy(os.Stdout, pr); err != nil {
panic(err)
}
pr.Close()
}
io.Copy is the workhorse for moving data between readers and writers. It returns the number of bytes copied and an error. If the writer closes the pipe, io.Copy returns nil error and the byte count. If the reader closes early, io.Copy returns an error. The writer goroutine will see a write |0: broken pipe error on the next write attempt. This happens when the consumer stops listening.
The receiver name convention applies here too. If you wrap this logic in a struct method, name the receiver one or two letters matching the type: (s *Streamer) Stream(...), not (this *Streamer). Keep it idiomatic.
User-space pipes with io.Pipe
Sometimes you do not need the kernel. io.Pipe creates a synchronous in-memory pipe that implements io.Reader and io.Writer interfaces. It runs entirely in user space, managed by goroutines. Use io.Pipe when you need to chain pure Go streams or mock I/O for testing.
func main() {
// io.Pipe creates a synchronous in-memory pipe.
// It returns io.Reader and io.Writer interfaces, not file descriptors.
pr, pw := io.Pipe()
// The writer must run in a goroutine.
// io.Pipe blocks until both the reader and writer are active.
go func() {
defer pw.Close()
pw.Write([]byte("Data from io.Pipe"))
}()
// ReadAll consumes the stream.
// io.Pipe handles synchronization entirely in user space.
data, err := io.ReadAll(pr)
if err != nil {
panic(err)
}
fmt.Println(string(data))
}
io.Pipe returns interfaces, not *os.File. This means you cannot pass the ends to system calls or subprocesses. It is lighter weight for in-process communication. The writer blocks until the reader starts reading, and vice versa. io.Pipe is useful when you want to inject a stream into a function that accepts io.Reader without creating a kernel object.
os.Pipe talks to the kernel. io.Pipe stays in Go. Pick the tool that matches your boundary.
Pitfalls and leaks
Pipes introduce concurrency hazards. The most common bug is a goroutine leak. If you spawn a writer goroutine and the reader returns early without closing the pipe, the writer might block forever on Write. The goroutine never exits, and the program holds resources indefinitely. Always ensure the reader closes the pipe or the writer has a cancellation path.
Context is plumbing. If your pipe writer runs for a long time, pass a context.Context as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines. If the context cancels, the writer should stop writing and close the pipe.
Another pitfall is ignoring the error from Write. If the reader closes the pipe, Write returns an error. If you ignore it, you might continue writing to a dead pipe, wasting CPU cycles. Check the error and break the loop.
The worst goroutine bug is the one that never logs. Add logging to your pipe writer so you can detect when it blocks or fails. If a goroutine hangs, logs help you find the dead end.
When to use pipes
Use os.Pipe when you need a kernel-managed pipe for subprocess I/O or when you require file descriptors that can be passed to system calls.
Use io.Pipe when you need a user-space pipe that implements io.Reader and io.Writer for chaining pure Go streams without kernel overhead.
Use os/exec with StdinPipe when you are launching an external command and need to write data to its standard input while reading its output.
Use a channel when you are passing typed Go values between goroutines and want the compiler to enforce type safety and synchronization.
Goroutines are cheap. Pipes are not magic.