Reading from stdin in Go
You are building a command-line tool that processes a list of files piped from find, or a script that asks the user for configuration values. The input arrives from the terminal, but it could also come from a file redirected into your program. Go treats standard input as a stream of bytes. The source does not matter. The keyboard, a pipe, and a file all look the same to your code.
Standard input is exposed as os.Stdin. This is a global variable of type *os.File. It implements the io.Reader interface. That interface is the foundation of all I/O in Go. It defines a single method: Read. Any type that implements Read can be read from. This uniformity means the same reading logic works for network connections, files, and standard input.
The challenge is choosing how to consume the stream. You can read line by line, read the entire blob at once, or read raw chunks. Each approach has trade-offs in memory usage, error handling, and convenience.
The io.Reader interface
The io.Reader interface looks like this:
type Reader interface {
Read(p []byte) (n int, err error)
}
The Read method fills the provided byte slice p with data. It returns the number of bytes read and an error. The error is nil if bytes were read successfully. The error is io.EOF when the stream ends. A reader can return both bytes and io.EOF in the same call, signaling the end of data.
Most code does not call Read directly. Higher-level wrappers handle buffering, splitting, and allocation. The standard library provides tools that wrap io.Reader to make common patterns easy.
Line-by-line with bufio.Scanner
The most common pattern for stdin is reading line by line. bufio.Scanner is the standard tool for this. It wraps a reader, manages a buffer, and splits input into tokens. The default split function breaks input on newlines.
Here is the minimal pattern: wrap stdin in a scanner, loop until the stream ends, and check for errors.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// Scanner buffers input and splits by newlines automatically
scanner := bufio.NewScanner(os.Stdin)
// Scan advances to the next line; returns false on EOF or error
for scanner.Scan() {
// Text returns the current line as a string without the newline
fmt.Println(scanner.Text())
}
// Always check Err after the loop to distinguish EOF from read failures
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "read error:", err)
}
}
The scanner handles buffering internally. It reads chunks from os.Stdin into a private buffer, then extracts lines from that buffer. This reduces system calls and improves performance. The loop runs until Scan returns false. That happens when there is no more data or when a read error occurs. You must call scanner.Err() after the loop to determine which case triggered the exit.
The if err != nil check is verbose by design. The Go community accepts this boilerplate because it forces the programmer to acknowledge the unhappy path. Hiding errors leads to silent failures. Explicit checks make the code robust.
Scanner is the workhorse. Check the error or you will miss the crash.
Walkthrough of the scanner loop
When you call scanner.Scan(), the scanner checks its internal buffer. If the buffer has data, it searches for the next newline. If it finds one, it marks the token and returns true. If the buffer is empty, the scanner calls Read on the underlying reader to fill the buffer. This continues until a token is found or the stream ends.
scanner.Text() returns the token as a string. This allocates a new string for each line. If you process millions of lines, these allocations add up. The scanner also provides scanner.Bytes(), which returns the token as a byte slice. This slice points into the scanner's internal buffer. It is valid only until the next call to Scan. Use Bytes() when you can process the data immediately to avoid string allocation.
The loop terminates when Scan returns false. This can happen for three reasons:
- The stream ended normally.
Err()returnsnil. - A read error occurred.
Err()returns the error. - A token exceeded the maximum size.
Err()returnsbufio.Scanner: token too long.
The scanner has a default maximum token size of 64KB. This limit prevents a single malformed line from consuming all memory. If your input contains lines longer than 64KB, the scanner stops and reports an error. You can increase the limit by calling scanner.Buffer with a larger slice.
Realistic example: processing with buffer tuning
Real-world input often has quirks. Lines might be very long. You might need to count bytes or runes. You might want to avoid allocations. Here is a function that reads stdin, handles long lines, and processes data efficiently.
package main
import (
"bufio"
"fmt"
"os"
"unicode/utf8"
)
// ProcessInput reads lines from stdin and prints statistics for each line.
// It tunes the buffer to handle lines up to 1MB.
func ProcessInput() {
// Create scanner with a custom buffer to support long lines
scanner := bufio.NewScanner(os.Stdin)
// Buffer sets the internal buffer; maxTokenSize is the second argument
scanner.Buffer(make([]byte, 1024), 1024*1024)
lineNum := 0
for scanner.Scan() {
lineNum++
// Bytes returns a slice valid only until the next Scan call
lineBytes := scanner.Bytes()
// len counts bytes; utf8.RuneCountInString counts Unicode characters
runeCount := utf8.RuneCount(lineBytes)
fmt.Printf("Line %d: %d bytes, %d runes\n", lineNum, len(lineBytes), runeCount)
}
// Check for errors after the loop
if err := scanner.Err(); err != nil {
fmt.Fprintf(os.Stderr, "failed to read input: %v\n", err)
}
}
func main() {
ProcessInput()
}
The scanner.Buffer call replaces the internal buffer with a larger one. The first argument is the initial buffer slice. The second argument is the maximum token size. This allows lines up to 1MB. The function uses scanner.Bytes() to get the line data without allocating a string. It passes the byte slice to utf8.RuneCount to count characters correctly. This avoids the overhead of converting to a string when the string is not needed.
The receiver name convention applies to methods. If this were a method on a struct, the receiver would be a short name like (p *Processor). Public functions start with a capital letter. Private functions start lowercase. This naming convention controls visibility across packages.
Tune the buffer only when the data demands it. The default 64KB covers most text files.
Reading everything at once
Sometimes you need the entire input as a single blob. io.ReadAll reads until EOF and returns a byte slice. This is convenient for small inputs like JSON payloads or configuration blocks. It allocates memory for the entire input. Do not use this for large streams or unbounded input.
package main
import (
"fmt"
"io"
"os"
)
func main() {
// ReadAll loads the entire stream into memory
data, err := io.ReadAll(os.Stdin)
if err != nil {
fmt.Fprintln(os.Stderr, "read failed:", err)
return
}
// data contains all bytes from stdin
fmt.Printf("Read %d bytes\n", len(data))
}
io.ReadAll calls Read repeatedly until EOF. It grows the result slice as needed. If the input is large, this can exhaust memory. The compiler will not warn you about memory usage. This is a runtime risk. Use ReadAll only when you know the input size is bounded and small.
ReadAll is a memory bomb for streams. Measure before you use it.
Pitfalls and errors
Reading stdin has several common pitfalls. Understanding these prevents subtle bugs.
The scanner fails on long lines. If a line exceeds the token size, Scan returns false and Err returns bufio.Scanner: token too long. The program stops reading. You must check Err to see this. If you ignore the error, the program appears to hang or exit silently. Increase the buffer or switch to bufio.Reader for unbounded lines.
Unicode handling requires care. scanner.Text() returns a string. len() on a string returns the number of bytes, not characters. UTF-8 characters can be multiple bytes. Use utf8.RuneCountInString to count characters. The compiler does not enforce this. It is a logic error.
Blocking behavior is expected. Scan and Read block until data is available. If the user does not type, the program waits. In a terminal, Ctrl+D sends EOF. In a pipe, EOF arrives when the upstream process closes the stream. If you run reading in a goroutine, ensure you have a way to cancel it. A goroutine that blocks on stdin and never receives EOF will leak. Always have a cancellation path.
The compiler rejects unused variables. If you capture an error and do not use it, you get declared and not used. You can discard a value with _. Use _ sparingly. Discarding errors is usually a mistake. The compiler forces you to make a conscious choice.
stdin is a stream. Treat it like one.
Decision matrix
Choose the reading tool based on your data shape and constraints.
Use bufio.Scanner when you need to process input line by line and lines fit within the token size limit. Use io.ReadAll when you need the entire input as a byte slice and the size is small and bounded. Use bufio.Reader when you need to read arbitrary chunks or handle lines longer than the scanner limit. Use os.Args when you need command-line arguments passed before the program starts, not streamed input. Use os.Stdin.Read directly when you are building a low-level protocol parser that handles raw bytes without buffering.
Pick the reader that matches your data shape. Don't read everything into memory unless you have to.