The firehose problem
You are writing a tool to download a massive CSV report from an API. You call http.Get, grab the response, and dump the body into a string. Your laptop fans scream. Memory usage spikes to 90 percent. The program crashes with an out-of-memory error. The file was two gigabytes. You only needed to parse it row by row. Loading everything into RAM was the mistake.
Streaming the response body solves this. You read small chunks, process them, and discard them. Memory usage stays flat regardless of file size. Go makes this easy because the HTTP client is built on streams from the ground up.
The io.Reader contract
Go treats the HTTP response body as a stream. The Response struct has a Body field of type io.ReadCloser. This type implements the io.Reader interface, which is the foundation of almost all I/O in Go. Files, network connections, buffers, and HTTP bodies all share this interface.
The interface defines a single method:
type Reader interface {
Read(p []byte) (n int, err error)
}
You pass a byte slice to Read. The method fills that slice with data and returns the number of bytes written, plus an error. The contract is strict. Read blocks until it has data, the stream ends, or an error occurs. It can return fewer bytes than the slice size. It can return both bytes and an error at the same time.
The stream ends when Read returns io.EOF. This sentinel error tells you there is no more data. You process the bytes, call Read again, and repeat until io.EOF appears. The key invariant is that you never hold the entire payload in memory unless you explicitly copy it all.
The stream is a contract. You ask for bytes, the network delivers, or it fails.
Minimal read loop
Here's the raw loop pattern. You allocate a buffer, call Read in a loop, and handle the end of stream. This gives you full control over chunk size and processing logic.
package main
import (
"fmt"
"io"
"log"
"net/http"
)
func main() {
resp, err := http.Get("https://example.com/large-file")
// Check network errors immediately.
if err != nil {
log.Fatal(err)
}
// Close the body to release the connection back to the pool.
defer resp.Body.Close()
// Buffer size determines how much data you fetch per network round-trip.
// 4KB is a common starting point for small chunks.
buf := make([]byte, 4096)
for {
n, err := resp.Body.Read(buf)
// Read returns the bytes read and an error.
// n can be less than len(buf) if data is limited.
if err != nil {
// io.EOF means the stream ended cleanly.
if err == io.EOF {
break
}
// Any other error indicates a network or protocol failure.
log.Fatal(err)
}
// Process only the bytes that were actually read.
fmt.Printf("Read %d bytes\n", n)
}
}
Read returns what it can. Check the count. Respect the error.
What happens at runtime
When you call http.Get, the standard library opens a TCP connection and sends the HTTP request. The Response object arrives, but the body data hasn't been fully downloaded yet. The kernel buffers incoming packets.
When you call resp.Body.Read, Go pulls bytes from that kernel buffer into your slice. If the buffer is empty, Read blocks the goroutine until more data arrives or the server closes the connection. This blocking behavior is efficient. The goroutine sleeps without consuming CPU.
The defer resp.Body.Close() is essential. Closing the body signals to the HTTP client that you are done. This allows the underlying connection to be reused for future requests. The http.Client maintains a pool of idle connections. If you skip the close, the connection leaks. The client might eventually time out and drop it, but you waste resources and risk hitting connection limits.
Close the body. Return the connection. Keep the pool healthy.
Copying to a writer
Most of the time, you don't write the loop manually. You pipe the reader to a writer. io.Copy handles the buffering and looping for you. It reads from the io.Reader and writes to an io.Writer, streaming data in chunks.
Here's how you download a file to disk without buffering the whole payload.
package main
import (
"fmt"
"io"
"net/http"
"os"
)
// DownloadFile streams an HTTP response to a local file.
func DownloadFile(url, dest string) error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// Validate status before processing the body.
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("status %d", resp.StatusCode)
}
out, err := os.Create(dest)
if err != nil {
return err
}
defer out.Close()
// io.Copy handles the read loop and buffering automatically.
// It streams data directly from the network to the disk.
_, err = io.Copy(out, resp.Body)
return err
}
io.Copy uses a default buffer size of 32KB. This is efficient for most workloads. If you need to reuse a buffer across multiple copies to save allocations, use io.CopyBuffer and pass your own slice.
io.Copy is the workhorse. Use it to move data, not to parse it.
Streaming with cancellation
http.Get is a convenience function. It doesn't support cancellation. In production, you use http.NewRequestWithContext. This lets you attach a deadline. If the download takes too long, the context cancels, and the read returns an error. This prevents goroutine leaks where a goroutine waits forever on a slow server.
context.Context always goes as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines.
package main
import (
"context"
"fmt"
"io"
"net/http"
"time"
)
// StreamWithTimeout demonstrates attaching a context to control the request lifecycle.
func StreamWithTimeout(url string) error {
// Create a context that cancels after 5 seconds.
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return err
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer resp.Body.Close()
buf := make([]byte, 1024)
for {
n, err := resp.Body.Read(buf)
if err != nil {
// Context cancellation surfaces as an error here.
return err
}
// Process buf[:n]...
_ = n
}
}
The body read will fail if the context is cancelled. This ensures the goroutine doesn't block indefinitely. The worst goroutine bug is the one that never logs. Cancel the context or close the body.
Buffering for text streams
Raw Read calls can be slow if you read one byte at a time. The network stack has overhead. bufio.Reader adds a buffer on top of the io.Reader. It fetches larger chunks and serves small reads from memory. Use bufio.NewReader when you need methods like ReadLine or ReadByte. It reduces system calls.
package main
import (
"bufio"
"fmt"
"io"
"net/http"
)
// BufferedStream shows how bufio reduces system calls for small reads.
func BufferedStream(url string) error {
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
// bufio wraps the body with a buffer.
// Subsequent reads pull from memory, not the network.
reader := bufio.NewReader(resp.Body)
line, err := reader.ReadString('\n')
if err != nil {
return err
}
fmt.Println(line)
return nil
}
Pitfalls and errors
Forgetting to close the body is the most common leak. The compiler won't catch this. The program runs, but connections pile up. Eventually, the client runs out of file descriptors or hits the connection limit. The error manifests as dial tcp: lookup ... no such host or timeouts on subsequent requests. Always pair resp.Body with defer resp.Body.Close().
Reading n bytes but processing len(buf) is a silent corruption bug. Read can return fewer bytes than the buffer size. If you process buf instead of buf[:n], you read garbage or stale data. The compiler won't stop you. You just get corrupted output.
io.EOF can arrive with data. Read can return both bytes and io.EOF in the same call. The stream ends, but you still have bytes to process. Check n > 0 before breaking on EOF. If you break immediately, you drop the last chunk.
If you try to read after the body is closed, you get read tcp ...: use of closed network connection. This usually happens if you pass the body to a goroutine and close it before the goroutine finishes.
The if err != nil check is verbose by design. The community accepts this boilerplate because it forces you to handle the unhappy path. You cannot accidentally swallow an error. Trust the verbosity. It saves you at 3 AM.
Limiting the stream
Sometimes you don't trust the Content-Length header. A malicious server might send more data than expected. io.LimitReader wraps the body and stops reading after a limit. This protects against memory bombs.
package main
import (
"io"
"net/http"
)
// SafeStream limits the amount of data read to prevent memory exhaustion.
func SafeStream(url string, limit int64) ([]byte, error) {
resp, err := http.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
// LimitReader stops reading after limit bytes.
// It returns io.EOF when the limit is reached, even if more data exists.
limited := io.LimitReader(resp.Body, limit)
data, err := io.ReadAll(limited)
return data, err
}
io.ReadAll reads until io.EOF. With LimitReader, it stops at the limit. This is safe for small payloads where you need the whole thing in memory.
Decision matrix
Use io.Copy when you need to transfer the entire stream to a writer, such as saving to a file or sending to another HTTP response. Use a manual Read loop when you need to process chunks with custom logic, like parsing a binary protocol or updating a progress bar. Use io.ReadAll when the payload is small and you need the complete data in memory as a byte slice. Use bufio.Scanner when the stream contains text lines and you want to iterate over them without managing buffers. Use io.LimitReader when you must enforce a hard cap on data size to protect against abuse. Use http.NewRequestWithContext when the request might hang and you need a cancellation path.
Pick the tool that matches the flow. Copy for transfer, loop for logic.