The bucket brigade approach to uploads
You are building a tool where users drop a file onto your server. Maybe it is a photo uploader for a portfolio site, or a CSV importer for a dashboard. You write the handler, test it with curl, and it works. Then a user uploads a 500MB video, and your server memory spikes to 4GB before the process gets killed. Or worse, the file saves as an empty blob because you did not parse the form correctly. File uploads look simple until the network gets slow, the files get big, or the client sends something unexpected.
HTTP file uploads usually use multipart/form-data. The browser splits the request into parts separated by a boundary string. Each part contains metadata like the filename, and the raw bytes of the file. Go's net/http handles the heavy lifting of parsing this boundary-separated mess. The key insight is streaming. You do not load the whole file into memory. You read chunks from the request and write chunks to disk. io.Copy is the workhorse here. It reads from a source and writes to a destination in a loop, using a small buffer. This keeps memory usage flat regardless of file size.
Think of io.Copy as a bucket brigade. You do not haul the entire river into your house. You pass buckets from the source to the destination. The bucket is a small buffer, usually 32KB. io.Copy grabs a bucket, fills it from the request, runs it to the file, dumps it, and repeats. The bucket is the same one every time. Memory usage stays flat.
Minimal streaming handler
Here is the bare minimum handler. It reads a field named "file", creates a destination, and streams the bytes. The code is split to show the parsing phase and the write phase separately.
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// Check method first to skip parsing if the request is wrong.
if r.Method != http.MethodPost {
http.Error(w, "Method not allowed", http.StatusMethodNotAllowed)
return
}
// ParseMultipartForm reads headers into memory up to the limit.
// 10MB is a safe default to prevent memory exhaustion attacks.
err := r.ParseMultipartForm(10 << 20)
if err != nil {
http.Error(w, "Bad request", http.StatusBadRequest)
return
}
// FormFile extracts the file stream and metadata.
// The second return value contains the filename and size.
file, _, err := r.FormFile("file")
if err != nil {
http.Error(w, "Missing file field", http.StatusBadRequest)
return
}
defer file.Close()
}
Now handle the disk write and response.
// Create the destination file with read/write for owner only.
out, err := os.Create("upload.bin")
if err != nil {
http.Error(w, "Server error", http.StatusInternalServerError)
return
}
defer out.Close()
// io.Copy reads and writes in 32KB chunks by default.
// This streams the file without buffering the whole payload.
_, err = io.Copy(out, file)
if err != nil {
http.Error(w, "Write failed", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
How the runtime handles this
When the request arrives, ParseMultipartForm reads the headers. If the form data exceeds the memory limit, it fails fast. FormFile returns an io.ReadCloser. This is just a stream. You can read from it. io.Copy creates a 32KB buffer. It reads from the request into the buffer, writes the buffer to the file, repeats. The buffer is reused. Memory stays at roughly 32KB plus the form limit. The defer calls ensure files close even if an error happens halfway through. defer schedules the function call to run when the surrounding function returns. It runs in LIFO order, so out.Close() runs before file.Close().
Stream everything. Memory is finite; disks are cheap.
Real-world safety: atomic writes and validation
Real services need unique filenames, size checks, and cleanup. Using the client-provided filename is dangerous. A malicious client can send ../../etc/passwd to overwrite system files. You also need to handle partial writes. If the server crashes while writing, you do not want a half-finished file appearing in your directory. The solution is atomic writes. Write to a temporary file first, then rename it to the final name. os.Rename is atomic on most filesystems. The file appears in the final directory only when the write is complete.
Here is a handler that generates a safe filename and validates the size before writing.
// Header.Size comes from the Content-Disposition header.
// Clients can lie, but it is a quick check before allocating disk.
if header.Size > 50<<20 {
http.Error(w, "File too large", http.StatusRequestEntityTooLarge)
return
}
// Generate a random name to prevent path traversal attacks.
// Never trust the filename sent by the client.
name := make([]byte, 16)
_, _ = rand.Read(name)
safeName := fmt.Sprintf("%x.bin", name)
// Create a temp file in the target directory.
// This ensures the file lands on the same filesystem for the rename.
tmp, err := os.CreateTemp("uploads", safeName)
if err != nil {
http.Error(w, "Server error", http.StatusInternalServerError)
return
}
Now copy the data and finalize the file.
// Copy the stream to the temp file.
// If this fails, the defer on os.Remove cleans up the partial file.
_, err = io.Copy(tmp, file)
if err != nil {
tmp.Close()
http.Error(w, "Write failed", http.StatusInternalServerError)
return
}
// Close the temp file before renaming.
// Windows requires handles to be closed before moving files.
tmp.Close()
// Rename is atomic on most filesystems.
// The file appears in the final directory only when the write is complete.
finalPath := filepath.Join("uploads", safeName)
err = os.Rename(tmp.Name(), finalPath)
if err != nil {
http.Error(w, "Move failed", http.StatusInternalServerError)
return
}
Convention asides
The if err != nil pattern is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible. You cannot accidentally swallow an error behind a silent catch block. Every error must be handled explicitly.
The underscore _ discards a value intentionally. _, _ = rand.Read(name) says "I considered the return values and chose to drop them". Use it sparingly with errors. Dropping an error from rand.Read is acceptable here because a crypto failure is rare, but in production code you should check it.
Public names start with a capital letter. Private names start lowercase. There are no keywords like public or private. The capitalization determines visibility across packages.
Atomic writes save you from debugging half-written files at 3 AM.
Pitfalls and compiler checks
File upload code has specific traps. The most common is memory exhaustion. If you forget to set a limit on ParseMultipartForm, Go defaults to 32MB. A large upload fills memory. The server returns a 500 error with http: multipart: memory limit exceeded if the form data exceeds the limit. Always set a limit that matches your form header expectations.
Another trap is reading the body twice. FormFile consumes the request body to parse the multipart data. If you try to read r.Body after calling FormFile, you get nothing. The runtime complains with http: request body already read. Structure your handler so parsing happens once, and you work with the returned streams.
Path traversal is a security risk. If you use the client filename directly, a request with filename ../../../var/www/config.yml can overwrite files outside your upload directory. Always generate your own filename or sanitize aggressively. Generating a random name is the safest approach.
The compiler helps with imports. Forget to use an import and you get imported and not used from the compiler. This prevents dead code and keeps binaries small. It also forces you to acknowledge every dependency.
Trust gofmt. Argue logic, not formatting. The community uses gofmt to standardize code style. Most editors run it on save. Do not waste time debating indentation.
Timeouts and context
Long uploads can hang. A client might start a large upload and then disconnect, leaving your server waiting for data that will never arrive. Or a slow network might cause the upload to take hours. You need timeouts. Go uses context.Context to manage request lifecycles.
Wrap the request body with a timeout or pass a context to downstream calls. The convention is that context.Context always goes as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines.
import "time"
// uploadWithTimeout adds a deadline to prevent hanging uploads.
func uploadWithTimeout(w http.ResponseWriter, r *http.Request) {
// Set a 5-minute deadline for the entire request.
// This cancels the context if the upload takes too long.
ctx, cancel := context.WithTimeout(r.Context(), 5*time.Minute)
defer cancel()
// Use the context-aware body reader.
// This ensures the read stops if the context is cancelled.
r.Body = http.MaxBytesReader(w, r.Body, 100<<20)
// Pass ctx to any downstream processing functions.
// They should check ctx.Err() periodically.
processUpload(ctx, w, r)
}
http.MaxBytesReader enforces a hard limit on the total request body size at the transport layer. It returns an error if the client sends more bytes than allowed. This protects against clients that ignore headers and keep sending data.
Context is plumbing. Run it through every long-lived call site.
Decision matrix
Use io.Copy when you need to stream data between an io.Reader and an io.Writer without managing buffers manually. Use r.ParseMultipartForm with a limit when handling file uploads to cap memory usage for form headers. Use os.CreateTemp followed by os.Rename when you need atomic file writes to prevent partial files from appearing in your directory. Use a random generated filename when you must prevent path traversal attacks and filename collisions. Use http.MaxBytesReader when you want to enforce a hard limit on the total request body size at the transport layer. Use plain r.Body reading when the payload is not multipart, such as a raw JSON or binary stream.