The pipe that connects goroutines
You have a function that expects an io.Reader. It reads data, processes it, and moves on. You have data in memory, or data coming from a network socket, or data being generated on the fly. You don't want to write that data to a temporary file just to satisfy the interface. You want to connect the writer to the reader directly, in memory, while both sides run at the same time.
That is where io.Pipe lives. It creates a connected pair of io.Reader and io.Writer that share a buffer. One goroutine writes data, another goroutine reads it. The pipe handles the synchronization. If the writer is faster than the reader, the writer waits. If the reader is faster, the reader waits. No channels needed. No manual locking. The pipe enforces flow control automatically.
How the pipe works
Think of io.Pipe like a physical pipe between two workers. One worker pours water into one end. The other worker drinks from the other end. The pipe has a limited capacity. If the drinker is slow, the pipe fills up and the pourer has to stop until space opens. If the pourer stops, the drinker eventually gets nothing and knows the stream is finished.
In Go, io.Pipe returns a *io.PipeReader and a *io.PipeWriter. They are linked internally. The buffer is small, typically 512 bytes. This size is deliberate. It forces the writer to yield control frequently, which keeps memory usage low and ensures the reader gets data in small chunks. The pipe uses a mutex internally to protect the buffer. When you call Write, the method checks if there is space. If yes, it copies bytes and returns. If no, the goroutine blocks until the reader consumes some data. When you call Read, the method checks for data. If yes, it copies bytes and returns. If no, the goroutine blocks until the writer sends more or closes the pipe.
This blocking behavior is the synchronization mechanism. You don't need to coordinate the goroutines with channels or wait groups. The pipe coordinates them for you. The writer blocks when the buffer is full. The reader blocks when the buffer is empty. The system self-regulates.
Minimal example
Here's the simplest pipe: spawn a writer, send a message, read it back.
package main
import (
"fmt"
"io"
)
func main() {
// Create the connected reader and writer pair.
pr, pw := io.Pipe()
// Writer runs in a separate goroutine to avoid deadlock.
go func() {
// Close the writer when done to signal EOF to the reader.
defer pw.Close()
// Write data into the pipe buffer.
pw.Write([]byte("hello"))
}()
// Read from the pipe until the writer closes.
buf := make([]byte, 10)
n, _ := pr.Read(buf)
fmt.Println(string(buf[:n]))
}
The writer runs in a goroutine because Write and Read block. If both ran in the same goroutine, the writer would block waiting for the reader, and the reader would block waiting for the writer. The program would hang. Separating them into goroutines lets the pipe mediate the exchange.
The defer pw.Close() is essential. The reader waits for data until the writer closes. When Close is called, the reader gets io.EOF on the next Read. Without the close, the reader waits forever. This is a common source of goroutine leaks. The reader goroutine hangs, waiting for data that never comes.
Convention aside: io.Pipe returns interfaces. The function signature is func Pipe() (*PipeReader, *PipeWriter), but the values satisfy io.Reader and io.Writer. This follows the Go mantra: accept interfaces, return structs. You can pass pr to any function that takes an io.Reader. The caller doesn't need to know it's a pipe. It just knows it can read.
Pipes block until data flows. Close the writer to let the reader finish.
Walkthrough of the mechanics
When io.Pipe is called, it allocates a shared structure containing the buffer, a mutex, and condition variables for signaling. The PipeReader and PipeWriter hold pointers to this structure. They are two handles to the same resource.
When the writer calls Write, it locks the mutex. It checks the buffer length. If the buffer has space, it copies the bytes, updates the length, unlocks, and returns. If the buffer is full, it waits on a condition variable. The reader wakes the writer when it consumes data.
When the reader calls Read, it locks the mutex. It checks the buffer length. If data exists, it copies bytes, shrinks the buffer, unlocks, and returns. If the buffer is empty, it checks if the writer has closed. If closed, it returns io.EOF. If not closed, it waits on a condition variable. The writer wakes the reader when it sends data or closes.
This mechanism ensures that data never gets lost. Bytes written are either in the buffer or being read. The mutex prevents race conditions. The condition variables prevent busy-waiting. The goroutines sleep when they can't make progress and wake when they can.
The buffer size is fixed. You cannot change it. If you need a larger buffer, you must use a different approach, like a channel with a larger capacity or a bytes.Buffer with manual synchronization. The small buffer in io.Pipe is a feature. It forces back-pressure. If the writer tries to dump megabytes of data at once, it blocks after 512 bytes. The reader must keep up. This protects the system from memory exhaustion.
Trust the pipe to synchronize. Don't add extra channels to coordinate what the pipe already handles.
Realistic example
Here's a realistic pattern: a generator function feeds a pipe while the main goroutine consumes the stream.
package main
import (
"fmt"
"io"
)
// GenerateData writes lines to the pipe and closes when done.
func GenerateData(pw *io.PipeWriter) {
defer pw.Close()
for i := 0; i < 5; i++ {
// Write blocks if the internal buffer is full.
pw.Write([]byte(fmt.Sprintf("line %d\n", i)))
}
}
func main() {
pr, pw := io.Pipe()
// Start the generator in the background.
go GenerateData(pw)
// Read blocks until data is available or the pipe closes.
buf := make([]byte, 1024)
for {
n, err := pr.Read(buf)
if n > 0 {
fmt.Print(string(buf[:n]))
}
if err != nil {
break
}
}
}
The GenerateData function takes the PipeWriter. It writes lines in a loop. Each Write call sends data to the buffer. If the buffer fills, the function pauses. This pauses the generation, which is exactly what you want. The generator doesn't outpace the consumer.
The main function reads in a loop. It checks n to see how many bytes were read. It checks err to detect the end. When GenerateData finishes, it closes the pipe. The reader gets io.EOF and breaks the loop.
This pattern separates concerns. The generator focuses on producing data. The consumer focuses on processing it. The pipe connects them. You can swap the generator for a network fetcher or a file reader without changing the consumer. You can swap the consumer for a compressor or a logger without changing the generator.
Convention aside: Error handling on the write side matters. If pw.Write returns an error, it usually means the reader has closed the pipe or the pipe is broken. The writer should stop and close itself. Ignoring write errors can lead to silent data loss. The if err != nil check is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible.
The generator yields control to the consumer. The pipe enforces the pace.
Pitfalls and errors
Pipes are simple, but they have traps. The most common issue is deadlock. If the writer and reader run in the same goroutine, and the buffer fills, the program hangs. The writer waits for the reader. The reader waits for the writer. The solution is always separate goroutines. One writes. One reads. Never mix them in the same flow unless you are certain the buffer won't fill.
Another trap is forgetting to close the writer. The reader waits for io.EOF. If the writer doesn't close, the reader waits forever. This leaks the reader goroutine. The goroutine sits in the runtime, consuming resources, doing nothing. The leak is silent. No panic. No log. Just a process that never exits or a worker that never finishes.
Convention aside: Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Pipes work the same way. Always have a cancellation path. Close the writer to signal the end. If the writer might run indefinitely, use a context.Context to cancel it. Pass the context to the writer goroutine. When the context cancels, the writer stops and closes the pipe.
Writing to a closed pipe returns an error. The runtime returns io.ErrClosedPipe. This happens if you call Write after Close. The compiler won't stop you. The error surfaces at runtime. Check the error if you care about write failures.
Reading from a closed pipe returns io.EOF. This is the normal way to detect the end of the stream. If you read and get io.EOF, the writer has finished. If you read and get a different error, something went wrong. Handle io.EOF explicitly. Don't treat it as a failure. It's the signal that the stream is done.
The buffer size is small. If you write large chunks, the writer blocks frequently. This is normal. It doesn't mean the pipe is slow. It means the pipe is protecting memory. If you need to transfer large amounts of data quickly, ensure the reader keeps up. If the reader is slow, the writer will block, and the transfer will be slow. The pipe matches the speed of the slowest side.
The worst goroutine bug is the one that never logs. Close the pipe to avoid silent hangs.
When to use io.Pipe
Use io.Pipe when you need to connect a writer to a reader across goroutines with back-pressure. Use bytes.Buffer when you can collect all data in memory before reading it. Use a channel of bytes when you need explicit control over message boundaries. Use os.Pipe when you need a file descriptor for a subprocess. Use plain sequential code when the data fits in memory and you don't need concurrency.
io.Pipe shines when you have a streaming source and a streaming sink, and you want them to run concurrently without unbounded memory growth. It is the bridge between the io.Reader and io.Writer worlds. It turns a writer into a reader. It turns a reader into a writer. It makes streaming data easy.
Don't fight the type system. Wrap the value or change the design.