The hose, the bucket, and the stream
You need to fetch a file from a remote server and save it to disk. Maybe your CLI tool downloads a configuration blob. Maybe a scraper grabs a PDF. The goal is simple: get bytes from a URL and write them to a file. Go handles this with a streaming model that keeps memory usage low, even for gigabyte-sized downloads.
HTTP responses are streams. The net/http package gives you a Response object where the body is an io.ReadCloser. Think of it as a hose. Water flows through it. You don't get the whole lake dumped into your RAM at once. You read chunks as they arrive. io.Copy is the bucket brigade that moves water from the hose to your storage tank without overflowing.
The minimal download pattern
Here's the baseline pattern. Fetch the resource, create a local file, copy the stream, and close everything.
package main
import (
"io"
"net/http"
"os"
)
// main demonstrates the simplest file download pattern.
func main() {
// http.Get returns a Response with a streaming Body.
resp, err := http.Get("https://example.com/file.txt")
if err != nil {
panic(err)
}
// defer ensures the body closes even if io.Copy panics later.
defer resp.Body.Close()
// Create truncates the file if it exists, or creates it new.
out, err := os.Create("file.txt")
if err != nil {
panic(err)
}
// defer closes the file handle when main returns.
defer out.Close()
// Copy reads from resp.Body and writes to out in chunks.
io.Copy(out, resp.Body)
}
Goroutines are cheap. Channels are not magic. In this synchronous pattern, the main goroutine blocks until the download finishes. That's usually fine for a simple script.
How the plumbing works
http.Get dials the server. It handles DNS resolution, TCP handshake, and TLS negotiation. It waits for the HTTP headers to arrive. Once the headers are read, it returns the Response object. The body is still open. The connection is still alive.
io.Copy takes over. It allocates a 32KB buffer on the stack. It loops: read a chunk from resp.Body, write that chunk to out, repeat. It stops when the read returns io.EOF. This means memory usage stays flat. You never hold the whole file in RAM.
The defer statements run when main returns. resp.Body.Close() is crucial. It signals to the HTTP client that you're done with the body. This allows the underlying TCP connection to be reused for future requests. If you skip this, the connection leaks. Eventually, your program runs out of file descriptors and crashes.
The Go community accepts verbose error handling. if err != nil { return err } makes the unhappy path visible. You see exactly where things can fail. Trust the verbosity. It saves debugging time later.
Production-ready download function
The minimal example panics on errors and ignores status codes. Real code checks the status, handles errors gracefully, and uses a client with timeouts. http.Get uses a default client with no timeout. A slow server can block your program forever.
Here's a robust function that wraps the download logic.
package main
import (
"fmt"
"io"
"net/http"
"os"
"time"
)
// DownloadFile fetches a URL and writes it to a local file.
func DownloadFile(url, filename string) error {
// Client allows configuring timeouts and transport settings.
client := &http.Client{
Timeout: 30 * time.Second,
}
// Get returns the response or a network error.
resp, err := client.Get(url)
if err != nil {
return fmt.Errorf("request failed: %w", err)
}
// defer ensures the body closes to reuse the connection.
defer resp.Body.Close()
// Check status code to reject 404 or 500 errors.
if resp.StatusCode != http.StatusOK {
return fmt.Errorf("server returned status: %d", resp.StatusCode)
}
// Create truncates the file if it exists.
out, err := os.Create(filename)
if err != nil {
return fmt.Errorf("file creation failed: %w", err)
}
// defer closes the file handle when the function returns.
defer out.Close()
// Copy streams the body to the file.
_, err = io.Copy(out, resp.Body)
if err != nil {
return fmt.Errorf("copy failed: %w", err)
}
return nil
}
Don't fight the type system. Wrap the error or change the design. Using fmt.Errorf with %w preserves the error chain. Callers can unwrap it later to check the root cause.
Pitfalls and compiler errors
The memory bomb
A common mistake is using io.ReadAll instead of io.Copy. io.ReadAll reads the entire body into a byte slice in memory. If you download a 1GB file, your process allocates 1GB of RAM. On a constrained server, the OS kills your process.
The compiler won't stop you. io.ReadAll returns []byte and error. It compiles fine. The failure happens at runtime. Use io.Copy to stream to a file. Use io.ReadAll only for small payloads like JSON responses or short text files.
The connection leak
Forgetting resp.Body.Close() leaks the connection. The HTTP client keeps the connection open, waiting for you to finish. After enough requests, you exhaust the file descriptor limit. The program hangs or crashes with too many open files.
Always close the body. defer resp.Body.Close() right after checking the error from Get is the standard pattern.
The loop variable capture
If you download files concurrently in a loop, you might run into the loop variable capture issue.
urls := []string{"https://a.com", "https://b.com"}
for _, u := range urls {
// BAD: u is the same variable reused in each iteration.
go func() {
DownloadFile(u, "out.txt") // u might change before goroutine runs.
}()
}
In older Go versions, this caused subtle bugs where goroutines read the final value of u. In Go 1.22+, the compiler rejects this with loop variable u captured by func literal. You must create a new variable inside the loop.
for _, u := range urls {
u := u // Create a new variable scoped to this iteration.
go func() {
DownloadFile(u, "out.txt")
}()
}
Trust gofmt. Argue logic, not formatting. The loop variable fix is a language change that prevents a whole class of bugs.
Compiler errors inline
If you forget to import a package, you get undefined: http from the compiler. If you import a package but don't use it, you get imported and not used. Go enforces clean imports. Remove unused imports immediately.
If you try to pass a string where an io.Reader is expected, the compiler complains with cannot use type string as type io.Reader in argument. You need to wrap the string in strings.NewReader.
Decision matrix
Use http.Get when you are writing a quick script and don't need custom timeouts or transport settings. Use http.Client when you need to configure timeouts, cookies, or TLS settings for production reliability. Use io.Copy when you are streaming data to a file or another writer to keep memory usage constant. Use io.ReadAll when the payload is small and you need random access to the bytes in memory. Use a context when you need to cancel a long-running download or enforce a deadline.
Context is plumbing. Run it through every long-lived call site. If your download function accepts a context.Context, it can respect cancellation signals from the caller. This is essential for web servers and CLI tools that need graceful shutdown.