Files are bytes and handles
You have a CSV export from a database. You write a script to parse it. The script works on your laptop. You deploy it to a server, and it crashes after processing 400 files. The error log says too many open files. Or worse, the script works fine until the file grows to 5GB, at which point your process gets killed by the operating system for using too much memory.
File I/O in Go is explicit. The language forces you to think about where the data comes from, how much memory it consumes, and when the resource is released. There is no magic garbage collector for file handles. You open a file, you get a handle, and you must close it.
A file on disk is just a sequence of bytes. The operating system manages access to these bytes through file descriptors. A file descriptor is a small integer that the kernel uses to track open files. When your Go program opens a file, the kernel allocates a descriptor and hands you a reference. The kernel has a limit on how many descriptors a process can hold. If you open files and never close them, you hit the limit. Every subsequent open fails.
Go exposes this reality directly. You get an error value when things go wrong. You cannot ignore it. The compiler forces you to check the error. This design prevents silent failures. You always know if a file is missing, unreadable, or locked.
Almost every I/O operation in the standard library relies on the io.Reader interface. The interface is tiny:
type Reader interface {
Read(p []byte) (n int, err error)
}
The Read method takes a byte slice and fills it with data. It returns the number of bytes written and an error. The number of bytes n can be less than the length of the slice. The contract requires the caller to keep calling Read until n is zero and err is io.EOF. os.File implements io.Reader. When you call Read on a file, Go makes a system call to the kernel. The kernel copies bytes from the disk cache into your process memory. This copy is expensive. Making thousands of tiny reads is slow. The standard library provides helpers that handle the looping and buffering. You rarely call Read directly. You use io.ReadAll or bufio.Scanner instead.
Reading small files
For small files that fit comfortably in memory, the standard library provides a shortcut. os.ReadFile opens the file, reads all bytes, closes the file, and returns the result. It handles the lifecycle in one call.
package main
import (
"fmt"
"os"
)
func main() {
// os.ReadFile opens, reads, and closes the file automatically
content, err := os.ReadFile("config.json")
if err != nil {
// Return immediately if the file cannot be read
fmt.Println("Error:", err)
return
}
// Convert the byte slice to a string for display
fmt.Println(string(content))
}
This function is convenient because it hides the boilerplate. It opens the file, reads until EOF, closes the file, and returns the bytes. If any step fails, it returns the error. You don't need to manage a file handle or write a defer statement. The community prefers this approach for small files because it reduces the chance of forgetting to close the file.
The if err != nil check is verbose by design. The Go community accepts this boilerplate because it makes the failure path visible. You cannot accidentally swallow an error. You must write the code to handle it. This discipline prevents subtle bugs from hiding in production.
What happens under the hood
When you call os.ReadFile, the function performs three steps. First, it calls os.Open to get a file handle. The kernel checks permissions and allocates a descriptor. Second, it reads bytes until the end of the file. It uses a buffer to minimize system calls. Third, it closes the file handle. The kernel releases the descriptor.
If any step fails, the function returns the error. The file is closed before the error is returned. This wrapper is safe because it guarantees cleanup. You don't need to write defer statements. The function handles it internally.
Under the hood, os.ReadFile uses io.ReadAll. This function creates a buffer and enters a loop. Inside the loop, it calls file.Read(buffer). The kernel copies bytes from the disk into the buffer. Read returns the number of bytes and a nil error. ReadAll appends those bytes to its result slice. It repeats this until Read returns zero bytes and io.EOF.
io.EOF is not an error in the traditional sense. It is a signal that the stream has ended. ReadAll swallows the io.EOF and returns the accumulated bytes. If you call Read manually, you must handle this case yourself. A read can return both data and io.EOF at the same time. You must process the data before treating the EOF as a termination signal.
Managing the lifecycle explicitly
Sometimes you need more control. You might want to check the file size before reading, or you might need to read part of the file, seek to a different position, and read again. In these cases, you manage the file handle explicitly.
package main
import (
"fmt"
"io"
"os"
)
func main() {
// Open returns a file handle and an error
file, err := os.Open("data.txt")
if err != nil {
fmt.Println("Failed to open:", err)
return
}
// Schedule Close to run when main returns
// This ensures the descriptor is released even on error
defer file.Close()
// ReadAll consumes the entire reader into a byte slice
content, err := io.ReadAll(file)
if err != nil {
fmt.Println("Failed to read:", err)
return
}
fmt.Printf("Read %d bytes\n", len(content))
}
The defer statement is critical here. It schedules the file.Close() call to run when the surrounding function returns. This ensures the file descriptor is released even if the function exits early due to an error or a panic. The defer runs after the function's return values are set but before the function actually returns to the caller.
Place the defer immediately after the successful open. If you put it later, you risk leaking the file if an error occurs between the open and the defer. The convention is clear: open, check error, defer close, use file. This pattern appears in every Go codebase that deals with resources.
defer is your safety net. Use it for every resource you open.
Streaming large files
Reading a 10GB log file into memory crashes your program. The operating system kills the process when it runs out of RAM. You need to process the file in chunks. For text files, bufio.Scanner is the standard tool. It reads the file line by line. It maintains an internal buffer. It reads a block of bytes from the file, splits them into lines, and yields one line at a time. Your memory usage stays flat regardless of the file size.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// Open the file for reading
file, err := os.Open("large_log.txt")
if err != nil {
fmt.Println("Error:", err)
return
}
// Close the file when the function returns
defer file.Close()
// Scanner buffers reads to reduce system calls
scanner := bufio.NewScanner(file)
// Scan advances to the next line and returns true if successful
for scanner.Scan() {
// Text returns the current line as a string
line := scanner.Text()
fmt.Println("Line:", line)
}
// Check for errors after the loop ends
if err := scanner.Err(); err != nil {
fmt.Println("Scan error:", err)
}
}
The scanner is efficient. It reduces the number of system calls. Instead of reading one byte at a time, it reads a large buffer and processes it in Go. The Scan method returns true if there is another line. It returns false when the end of the file is reached or an error occurs.
The Text method allocates a new string for each line. For extreme performance, use Bytes instead. It returns a byte slice that is valid only until the next call to Scan. This avoids allocation, but you cannot store the slice across iterations. If you need to keep the data, copy it or convert it to a string immediately.
Pitfalls and conventions
File I/O has specific failure modes. If you forget to import a package, the compiler rejects the program with undefined: os. If you try to pass a file handle where a string is expected, you get cannot use file as string value in argument. These compile-time errors are clear. Runtime errors require attention.
If you open a file that doesn't exist, os.Open returns an error like open data.txt: no such file or directory. You must check this error. Ignoring it leads to panics or undefined behavior. Forgetting defer file.Close() is a silent leak. The compiler does not warn you. The program runs until you exhaust the file descriptor limit. Every subsequent open fails with too many open files. This is hard to debug. Always use defer for file handles.
The scanner has a trap. The loop exits when Scan returns false. This happens when the file ends, but it also happens when an error occurs. If a line is too long, or a disk error happens, Scan returns false. You must check scanner.Err() after the loop to distinguish between a clean end and a failure. Skipping this check causes your program to silently stop processing on errors.
The scanner also has a default maximum token size of 64KB. Lines longer than this cause an error. You can increase the limit with scanner.Buffer, but you must allocate a large byte slice upfront. If your file has variable-length records longer than 64KB, consider using bufio.Reader directly and reading until a delimiter.
Mixing Read and Scanner on the same handle causes data loss. The scanner starts where the previous read left off. Stick to one reading strategy per handle.
If you wrap file reading in a method, name the receiver f or file, not this. Go style favors short, meaningful names. The receiver name is usually one or two letters matching the type.
The worst goroutine bug is the one that never logs. File I/O is often sequential. Don't add concurrency unless you have a reason. If you do spawn goroutines to read files, ensure each goroutine closes its own file handle. Goroutine leaks happen when a goroutine waits on a channel that never gets closed. Always have a cancellation path.
Decision matrix
Use os.ReadFile when the file fits in memory and you need the entire contents. It is the shortest and safest option for small files.
Use os.Open with io.ReadAll when you need to inspect the file object before reading, or reuse the handle for multiple operations. It gives you full control over the lifecycle.
Use bufio.Scanner when processing text line-by-line to keep memory usage flat. It is the standard way to handle large text files without loading them entirely into RAM.
Use io.Copy when streaming data from a file to a network connection or another file without touching the bytes in Go. It moves data directly between readers and writers with minimal allocation.
Use plain sequential code when you don't need concurrency. The simplest thing that works is usually the right thing. Don't add goroutines unless you have a reason.
Where to go next
- How to Use io.MultiWriter in Go
- How to Use io.TeeReader and io.MultiReader
- How the io Package Works in Go: Reader, Writer, and Closer
Memory is cheap. File descriptors are not. Close your files. Check your errors. Trust the scanner.