How to read and write files
You are building a CLI tool that needs to load a configuration file. In some languages, you open a file handle, read data in a loop, close the handle, and hope you didn't forget the close statement when an error occurs. Go's standard library gives you a shortcut that handles the boilerplate for you, but you still need to understand what happens under the hood. Disk operations fail. Permissions matter. Memory is finite. The simplest functions in Go hide complexity, and knowing when to use them versus when to reach for lower-level primitives is what separates a script from a robust program.
The byte slice shortcut
Go treats files as streams of bytes. The most common way to interact with a file is to read the entire content into memory or write a chunk of data to disk in one step. os.ReadFile reads a file and returns a byte slice. os.WriteFile takes a byte slice and writes it to a file, creating or overwriting as needed.
These functions are wrappers around lower-level calls. They open the file, perform the I/O, close the file descriptor, and return the result. You do not need to call Close manually. The functions guarantee the file is closed even if an error occurs halfway through. This design prevents file descriptor leaks, which can crash long-running services when too many files stay open.
The community accepts the if err != nil boilerplate because it forces you to acknowledge that disk operations can fail. Ignoring errors here leads to silent data corruption or panics later. gofmt enforces the indentation of the error check, so you never argue about formatting. You just write the check and move on.
Here is the minimal pattern: read a file, check the error, write the data back out.
package main
import (
"fmt"
"os"
)
func main() {
// Read the entire file into memory as a byte slice
data, err := os.ReadFile("config.json")
if err != nil {
// Print to stderr and exit with non-zero status
fmt.Fprintf(os.Stderr, "failed to read config: %v\n", err)
os.Exit(1)
}
// Write the data to a new file with read/write permissions for owner
err = os.WriteFile("config_backup.json", data, 0644)
if err != nil {
fmt.Fprintf(os.Stderr, "failed to write backup: %v\n", err)
os.Exit(1)
}
}
What happens under the hood
When you call os.ReadFile, the runtime opens the file, allocates a buffer, reads data until the end of the file, closes the file descriptor, and returns the result. The allocation happens on the heap. If the file is small, the garbage collector handles the memory efficiently. If the file is large, the allocation can pressure your program.
The permissions argument in os.WriteFile is an octal number. The leading zero tells the compiler it is octal. 0644 means read and write for the owner, read-only for the group and others. If you omit the leading zero, the compiler treats 644 as a decimal integer. The OS interprets file modes as octal, so passing 644 decimal results in mode 1204 octal, which is invalid. You get a runtime error like invalid argument. Always use the leading zero for file permissions.
Go strings are immutable. Once created, you cannot change a string. os.ReadFile returns []byte because files are streams of bytes and byte slices are mutable. This avoids unnecessary allocations if you are just passing the data around. If you need to treat the content as text, use string(data). The conversion copies the underlying array, so it is not free. If you are doing heavy text manipulation, the bytes package offers functions like bytes.TrimSpace and bytes.Split that work directly on slices without creating intermediate strings.
ReadFile closes the file for you. Trust the function, but check the error.
Realistic processing with bytes
In real code, you rarely just copy a file. You read data, transform it, and write the result. This often involves splitting content, trimming whitespace, or parsing JSON. The bytes package is your friend here. It provides efficient operations on byte slices.
Here is a function that reads a file, trims trailing whitespace from each line, and writes the cleaned content. It uses bytes.Split to break the file into lines and bytes.Buffer to accumulate the result. The function returns an error wrapped with context so the caller knows which file failed.
import (
"bytes"
"fmt"
"os"
)
// ProcessFile reads a file, trims trailing whitespace from each line, and writes the result.
func ProcessFile(inputPath, outputPath string) error {
// Read the source file entirely into memory
content, err := os.ReadFile(inputPath)
if err != nil {
return fmt.Errorf("reading %s: %w", inputPath, err)
}
// Split into lines and trim each line
lines := bytes.Split(content, []byte("\n"))
var result bytes.Buffer
for i, line := range lines {
trimmed := bytes.TrimSpace(line)
result.Write(trimmed)
// Add newline between lines, but not after the last one
if i < len(lines)-1 {
result.WriteByte('\n')
}
}
// Write the processed content with standard permissions
return os.WriteFile(outputPath, result.Bytes(), 0644)
}
The error wrapping uses %w. This preserves the original error chain. The caller can use errors.Is or errors.As to inspect the underlying cause. Returning a wrapped error is the standard Go practice. It gives context without losing the root cause.
Pitfalls and runtime errors
os.ReadFile loads the entire file into memory. If you read a 2GB log file, your program allocates 2GB of RAM. For large files, this causes high memory usage or an out-of-memory panic. The runtime panics with runtime: out of memory or the OS kills the process. Use streaming I/O for large files.
os.WriteFile truncates the file before writing. It does not append. If you want to add data to an existing file, use os.OpenFile with os.O_APPEND. WriteFile is also not atomic. If the program crashes during the write, the file may contain partial data. For critical data, write to a temporary file and rename it. The rename operation is atomic on most filesystems.
os.ReadFile follows symlinks. If the path points to a symlink, the function reads the target file. If you need to inspect the symlink itself, use os.Lstat. Lstat returns metadata about the link, not the target.
os.ReadFile blocks until the read completes. It does not accept a context.Context. If you are in a web handler and the client disconnects, ReadFile keeps running. You cannot cancel it. This is a design choice for simplicity, but it matters in servers. Use os.Open and read with a context-aware reader if you need cancellation.
Common errors include open config.json: no such file or directory when the path is wrong, permission denied when the process lacks access, and invalid argument when permissions are malformed. The compiler rejects code with syntax errors, but file errors happen at runtime. You must handle them.
Memory is finite. Stream what you cannot hold.
Path safety and metadata
File paths can contain user input. Concatenating strings to build paths is dangerous. Use filepath.Join to construct paths safely. It handles separators correctly on all platforms and prevents path traversal attacks when combined with validation. filepath.Abs resolves a path to an absolute location, which helps with logging and debugging.
import (
"fmt"
"os"
"path/filepath"
)
func safeRead(baseDir, userPath string) ([]byte, error) {
// Join paths safely and resolve to absolute path
fullPath := filepath.Join(baseDir, userPath)
absPath, err := filepath.Abs(fullPath)
if err != nil {
return nil, fmt.Errorf("resolving path: %w", err)
}
// Ensure the path is within the base directory
if !filepath.HasPrefix(absPath, baseDir) {
return nil, fmt.Errorf("path escapes base directory")
}
// Read the validated file
return os.ReadFile(absPath)
}
The filepath.HasPrefix check prevents directory traversal. If userPath is ../secret, the joined path might escape baseDir. The absolute check catches this. Always validate paths when dealing with user input.
The swiss army knife: os.OpenFile
os.ReadFile and os.WriteFile cover 90% of use cases. When you need more control, use os.OpenFile. It accepts flags and permissions. Flags like os.O_CREATE, os.O_TRUNC, os.O_APPEND, and os.O_EXCL let you customize behavior. os.O_EXCL ensures the file is created exclusively. If the file exists, the call fails. This is useful for locking or preventing overwrites.
os.OpenFile returns a *os.File. You must call Close on it. Use defer to ensure the file closes when the function returns. Forgetting to close a file leaks a file descriptor. The worst goroutine bug is the one that never logs, and the worst file bug is the one that leaks silently until the system runs out of descriptors.
Decision matrix
Use os.ReadFile and os.WriteFile when the file fits comfortably in memory and you need the simplest code. Use os.Open with io.Copy when streaming large files to avoid high memory usage. Use bufio.Scanner or bufio.Reader when processing a file line by line or token by token. Use os.OpenFile with flags when you need exclusive creation, appending, or non-truncating writes. Use a temporary file plus os.Rename when writing critical data that must not be corrupted by a crash. Use filepath.Join and validation when constructing paths from user input. Avoid ioutil.ReadFile; it was moved to os in Go 1.16 and is deprecated.
Octal permissions start with zero. The compiler will not save you from 644 decimal.