The cost of talking to the operating system
You open a ten gigabyte log file to count error lines. You read one line at a time. The program takes forty minutes. The bottleneck is not your CPU. It is the operating system. Every time your code asks for a single line, the program pauses, switches from user space to kernel space, finds the data on disk, copies it back, and switches again. Context switching costs cycles. Doing it ten million times adds up.
bufio solves this by inserting a memory tray between your code and the operating system. Think of a restaurant kitchen. The waiter does not run to the pantry for every single salt shaker. They grab a tray, fill it with everything the kitchen needs, and bring it back. When the tray empties, they go back for another load. bufio is that tray. It keeps a slice of bytes in RAM. Your code reads from the slice. When the slice runs dry, bufio automatically pulls a fresh chunk from the underlying file or network connection.
How the buffer actually works
The standard library exposes two main types for this pattern. bufio.Reader wraps any io.Reader. bufio.Writer wraps any io.Writer. Both default to a four kilobyte internal buffer. You rarely need to change that size unless you are streaming massive binary blobs over a slow network.
Here is the simplest pattern for reading a line and writing it back out:
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// Wrap stdin to batch system calls into memory
reader := bufio.NewReader(os.Stdin)
// Wrap stdout so writes accumulate before hitting the terminal
writer := bufio.NewWriter(os.Stdout)
defer writer.Flush() // Push remaining buffer to stdout when main exits
// Read until the newline character, including the newline itself
line, err := reader.ReadString('\n')
if err != nil {
fmt.Fprintln(os.Stderr, err)
return
}
// Write to the buffer. Nothing reaches the terminal yet.
writer.WriteString(line)
}
The defer writer.Flush() call is the safety net. Buffered writers hold data in memory until the buffer fills up or you explicitly push it out. If your program crashes or exits early, unflushed data vanishes. Deferring the flush guarantees the tray gets emptied.
Tracking pointers in memory
When bufio.NewReader(os.Stdin) runs, it allocates a four kilobyte byte slice. The first call to ReadString('\n') checks that slice. It is empty, so bufio calls os.Stdin.Read, which triggers a system call and fills the slice with whatever is available in the terminal buffer. ReadString then scans the slice for the newline byte, returns the substring, and advances an internal read pointer. The next call to ReadString starts from that pointer. No new system call happens until the pointer reaches the end of the four kilobyte slice.
Writing works in reverse. WriteString copies bytes into the writer's internal slice and advances a write pointer. The underlying io.Writer never sees the data until the pointer hits the slice capacity. At that point, bufio automatically calls the underlying writer to drain the buffer, resets the pointer to zero, and continues. You do not need to track buffer boundaries manually. The package handles the refill and drain cycles transparently.
Buffered I/O is a performance optimization, not a correctness guarantee. Trust the package to manage the slice. Focus your code on the business logic that runs between the read and the write.
Peeking and rewinding
Parsers often need to look ahead before committing to a token. bufio.Reader provides ReadByte and UnreadByte for exactly this scenario. ReadByte pulls a single byte from the buffer and advances the pointer. UnreadByte moves the pointer back one step, making that byte available for the next read.
// PeekFirstByte reads one byte, checks if it is a comment marker,
// and pushes it back if it is not.
func PeekFirstByte(r *bufio.Reader) (byte, error) {
// Pull the next byte without consuming a full line
b, err := r.ReadByte()
if err != nil {
return 0, err
}
// Check for the hash character used in shell scripts
if b != '#' {
// Push the byte back so the next Read call sees it again
r.UnreadByte()
}
return b, nil
}
You can only unread the last byte you read. The package does not maintain a full history stack. If you need to look ahead multiple bytes, read them into a temporary slice and process them manually. The single-byte rewind is fast because it only adjusts an integer pointer inside the struct.
Reading and writing in production
Real code rarely reads from standard input. You usually wrap files or network connections. The error handling pattern stays the same. Go does not hide errors behind exceptions. You check them immediately after every operation that can fail. The community accepts the repetition because it forces you to acknowledge failure paths instead of hoping they never happen.
Here is a realistic file transformation that reads a CSV, strips whitespace, and writes the result to a new file:
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// CleanFile reads from srcPath, trims spaces, and writes to dstPath.
func CleanFile(srcPath, dstPath string) error {
// Open source file for reading. Fail fast if it does not exist.
src, err := os.Open(srcPath)
if err != nil {
return fmt.Errorf("open source: %w", err)
}
defer src.Close()
// Create destination file with read-write permissions for the owner
dst, err := os.Create(dstPath)
if err != nil {
return fmt.Errorf("create dest: %w", err)
}
defer dst.Close()
// Wrap both files to avoid per-line system calls
reader := bufio.NewReader(src)
writer := bufio.NewWriter(dst)
defer writer.Flush()
for {
// ReadString returns the data plus the delimiter, or an error
line, err := reader.ReadString('\n')
if err != nil {
// io.EOF means we reached the end of the file cleanly
if err.Error() == "EOF" {
break
}
return fmt.Errorf("read line: %w", err)
}
// Trim leading and trailing whitespace before writing
clean := strings.TrimSpace(line)
// Discard the byte count return value since we only care about errors
if _, writeErr := writer.WriteString(clean + "\n"); writeErr != nil {
return fmt.Errorf("write line: %w", writeErr)
}
}
return nil
}
Notice the defer writer.Flush() paired with defer dst.Close(). The flush must happen before the file descriptor closes. Go executes deferred calls in reverse order, so Flush runs first, then Close. If you reverse the order, the buffer drains to a closed file and you get a runtime error. The compiler will not catch this ordering mistake. It only knows you called both functions.
The underscore in _, writeErr := writer.WriteString(...) follows a standard Go convention. It tells the reader and the compiler that you intentionally ignored the first return value. The compiler rejects unused variables with an unused variable error, so the underscore is the explicit opt-out. Use it sparingly with errors, but freely for counters or lengths you do not need.
Where things go wrong
Buffered I/O introduces a few behavioral shifts that trip up developers coming from higher-level languages. The most common mistake is mixing buffered and unbuffered operations on the same stream. If you wrap a file in bufio.NewReader and then call file.Read directly, the buffered reader and the raw file descriptor maintain separate internal pointers. The raw read will skip over data the buffered reader already pulled into memory, or overwrite data waiting to be processed. The compiler rejects this with a type mismatch error if you try to pass the wrong interface, but if you accidentally call methods on the underlying os.File instead of the bufio.Reader, the program compiles fine and silently corrupts your data stream.
Another trap is assuming ReadString always returns a complete line. It returns as soon as it finds the delimiter. If the underlying source sends data without newlines, ReadString blocks until the delimiter arrives or the connection closes. Network streams behave differently than local files. A remote server might send partial lines. bufio does not guess where a line ends. It waits for the exact byte you asked for.
Forgetting to flush is the silent killer. The compiler complains with unused variable if you ignore return values, but it never warns you about missing flush calls. Data stays in the four kilobyte slice until the program exits or the buffer fills. If you are writing to a log file and the process crashes, the last few kilobytes of logs disappear. Always flush before closing, or use defer.
Buffer sizing is another tuning knob that rarely needs adjustment. The default four kilobyte buffer matches typical OS page sizes and network MTU boundaries. Increasing it to sixty-four kilobytes might help with sequential disk reads on modern SSDs, but it also increases memory pressure and latency for small writes. Profile before changing it. Premature optimization of buffer sizes usually hurts readability more than it helps throughput.
When to reach for bufio
Use bufio.Reader when you are parsing text line by line or reading fixed-size records from a file or network connection. Use bufio.Writer when you are generating output in small chunks and want to batch those chunks into fewer system calls. Use io.Copy when you are moving data from one stream to another without inspecting the contents. Use unbuffered os.File or net.Conn when you need precise control over exact byte boundaries or are implementing a custom protocol that requires immediate delivery. Use bufio.Scanner when you want a simpler loop that handles tokenization automatically, but stick to Reader when you need to inspect raw bytes or handle malformed input gracefully.
Buffered I/O is a mechanical layer. Keep it thin. Let the package manage the tray.