The universal adapter for bytes
You are building a tool that processes data. The data might arrive from a local file, an HTTP request, a database stream, or standard input. Each source exposes a different API. Each destination expects data in its own format. Writing a separate processing function for every combination is tedious and error prone. You want one function that handles the transformation logic, regardless of where the bytes come from or where they go.
The io package solves this by defining two universal contracts: Reader and Writer. Think of them as standardized plumbing fittings. A Reader promises to fill a byte slice you provide. A Writer promises to take a byte slice you give it. The underlying source could be a disk drive, a TCP socket, or a slice of memory in RAM. The contract stays the same. This is how Go achieves composition without inheritance. You write your logic once against the interface, and it works everywhere.
The io package is plumbing. Treat it as such.
How the contract works
Every type that implements io.Reader must provide a single method: Read(p []byte) (n int, err error). Every type that implements io.Writer must provide Write(p []byte) (n int, err error). The signatures look simple, but they carry strict expectations.
When you call Read, you pass a buffer. The implementation fills as much of that buffer as it can, returns the number of bytes written to n, and returns an error if something went wrong. The method is allowed to return fewer bytes than the buffer size. It is also allowed to return io.EOF alongside a partial read. The Write method works in reverse: you hand it data, it returns how many bytes it accepted, and an error if the destination rejected the data.
This design forces callers to handle partial operations. Network sockets rarely deliver exactly the number of bytes you ask for. Disk caches might only have a fragment ready. The contract acknowledges reality instead of pretending streams are perfectly aligned.
Interfaces are accepted, structs are returned. Build your pipelines against io.Reader and io.Writer, not concrete file or socket types.
A minimal custom reader
Here is a simple type that implements io.Reader. It yields the digits of a counter until it reaches a limit.
package main
import (
"fmt"
"io"
"strconv"
)
// CounterReader yields sequential digits until it reaches max.
type CounterReader struct {
current int
max int
}
// Read fills p with the next digit and returns the count.
func (c *CounterReader) Read(p []byte) (n int, err error) {
if c.current >= c.max {
return 0, io.EOF // signal the stream is exhausted
}
digit := strconv.Itoa(c.current)
copy(p, digit) // write the digit string into the caller's buffer
c.current++ // advance the internal state
return len(digit), nil // return bytes written, no error
}
func main() {
r := &CounterReader{max: 5}
buf := make([]byte, 10)
n, err := r.Read(buf)
fmt.Printf("read %d bytes: %q\n", n, buf[:n])
}
The Read method respects the contract. It fills the provided slice, returns the exact byte count, and signals completion with io.EOF. The caller owns the buffer and decides how to slice it. This pattern scales from tiny counters to megabyte streams.
What happens under the hood
When you call io.Copy(dst, src), nothing magical happens. The function allocates a 32 kilobyte buffer by default. It enters a loop, calls src.Read(buf), checks the error, calls dst.Write(buf[:n]), and accumulates the total byte count. When Read returns io.EOF, the loop breaks and returns the total.
This design matters because it decouples your logic from buffer management. You do not need to track offsets, handle partial reads, or guess optimal chunk sizes. The standard library handles the busy work. You only provide the source and destination.
io.ReadAll follows a different path. It repeatedly calls Read and appends to a growing byte slice until io.EOF arrives. It is convenient for small payloads, but it loads the entire stream into memory. A 500 megabyte file will consume 500 megabytes of RAM. The function does not stream. It materializes.
Goroutines are cheap. Channels are not magic. Buffers are finite.
Realistic pipeline with safety limits
Real applications rarely trust unbounded streams. A malicious client can send a terabyte payload to exhaust your server. The io package provides wrappers that enforce boundaries without changing the interface. io.LimitReader caps the number of bytes a reader will return. io.TeeReader splits a stream so you can log or hash it while passing it along.
Here is a realistic handler that reads an upload, enforces a size limit, copies it to disk, and logs the byte count.
package main
import (
"fmt"
"io"
"os"
)
// ProcessUpload reads from src, caps it at maxBytes, and writes to dst.
func ProcessUpload(src io.Reader, dst io.Writer, maxBytes int64) (int64, error) {
// wrap src to prevent memory exhaustion from oversized payloads
limited := io.LimitReader(src, maxBytes)
// copy data through the limit wrapper into the destination
n, err := io.Copy(dst, limited)
if err != nil {
return n, fmt.Errorf("copy failed: %w", err)
}
// check if the stream was truncated by the limit
if n == maxBytes {
return n, fmt.Errorf("upload exceeded %d byte limit", maxBytes)
}
return n, nil
}
func main() {
file, err := os.Create("upload.bin")
if err != nil {
panic(err)
}
defer file.Close()
n, err := ProcessUpload(os.Stdin, file, 1024)
fmt.Printf("wrote %d bytes\n", n)
}
The wrapper sits between the raw input and the copy operation. It intercepts every Read call and stops returning data once the cap is reached. The rest of the pipeline never knows the difference. It still sees an io.Reader. This is the power of composition. You chain behaviors without rewriting core logic.
Context is plumbing. Run it through every long-lived call site.
Common pitfalls and compiler signals
Manual loops over Read are a frequent source of subtle bugs. The contract allows partial reads. If you call Read once and assume the buffer is full, you will silently drop data. If you ignore io.EOF, your loop will spin forever or treat a clean shutdown as a failure. The compiler will not save you here. The interface is satisfied, but the logic is wrong.
Interface mismatches are caught at compile time. If your Read method returns the wrong types or omits the error return value, the compiler rejects the program with cannot use myReader (variable of type MyReader) as io.Reader value in argument: MyReader does not implement io.Reader (wrong type for method Read). Fix the signature to match Read(p []byte) (n int, err error) exactly. Go does not allow method name mangling or optional parameters.
Another common mistake is passing io.EOF as a regular error up the call stack. io.EOF is a sentinel value, not a failure. Functions like io.Copy and io.ReadAll expect it and handle it gracefully. If you wrap it in fmt.Errorf without checking, you break the contract for callers that rely on errors.Is(err, io.EOF).
The worst goroutine bug is the one that never logs. The worst I/O bug is the one that silently drops partial reads.
Choosing the right tool
Use io.Copy when you need to move data between a reader and a writer with automatic buffering and partial-read handling. Use io.ReadAll when the payload is small, bounded, and you need the entire contents in memory at once. Use io.LimitReader when you must protect against oversized streams or enforce quota boundaries. Use a manual Read loop when you need to inspect or transform bytes before they reach the destination, such as parsing a custom binary protocol. Use io.Pipe when you want to connect a reader and writer that run in separate goroutines, creating a synchronous in-memory stream. Use plain sequential code when you don't need streaming: read the file, process it, write it out. The simplest thing that works is usually the right thing.
Don't fight the type system. Wrap the value or change the design.