The memory trap of json.Unmarshal
You are processing a log file full of JSON records. The file is 500 megabytes. You write a quick script using json.Unmarshal to load the data. The script runs, memory usage spikes, and the operating system kills your process with an out-of-memory error. Or you are building an API that receives a stream of events. The client sends data over a long-lived connection. You try to buffer the entire request body before parsing. The connection times out, or the buffer grows until the server stalls.
json.Unmarshal expects the entire JSON payload in a byte slice. It loads everything into RAM at once. This works for small configuration files or single API responses. It fails for streams, large files, or unbounded data. You need to parse values incrementally, keeping memory usage flat regardless of input size.
Go's encoding/json package provides json.Decoder and json.Encoder for this exact problem. These types wrap io.Reader and io.Writer interfaces. They read or write data in chunks, parsing or serializing one value at a time. The internal buffer stays small. You can process terabytes of JSON with a few megabytes of RAM.
Streaming keeps memory flat. Unmarshal fills the bucket; Decoder drinks from the tap.
Streaming with io.Reader and io.Writer
json.Decoder attaches to any io.Reader. A reader is anything that provides bytes on demand: a file handle, a network connection, a strings.Reader, or a pipe. The decoder maintains a small internal buffer. It reads bytes from the source, parses JSON tokens, and stops as soon as it has reconstructed a complete value. The next call to Decode continues exactly where the last one left off.
json.Encoder attaches to any io.Writer. A writer accepts bytes: a file, os.Stdout, an HTTP response, or a channel buffer. The encoder serializes a Go value to JSON and writes the bytes to the destination. By default, Encode appends a newline after each value. This behavior produces JSON Lines output automatically, which is the standard format for streaming logs and data pipelines.
The flexibility comes from the interfaces. You can stream JSON from a remote URL by wrapping http.Response.Body in a decoder. You can stream to a compressed file by wrapping a gzip.Writer in an encoder. The encoding/json package doesn't care where the bytes come from or where they go. It only cares about the stream.
Use the stream interface when the data size is unknown or exceeds available memory. Use the byte-slice functions when you already have the data in RAM and need a quick conversion.
The decoder loop
Here is the canonical pattern for consuming a stream of JSON values. The decoder reads one value per iteration. The loop terminates when the stream ends.
package main
import (
"encoding/json"
"fmt"
"io"
"strings"
)
func main() {
// Simulate a stream of concatenated JSON objects
data := `{"id":1,"name":"Alice"}{"id":2,"name":"Bob"}`
reader := strings.NewReader(data)
// NewDecoder wraps the reader and parses tokens incrementally
dec := json.NewDecoder(reader)
for {
var item map[string]interface{}
// Decode reads exactly one JSON value and stops at the boundary
err := dec.Decode(&item)
// io.EOF signals the stream ended cleanly without a trailing value
if err == io.EOF {
break
}
if err != nil {
// Handle parse errors; the stream is corrupted or malformed
fmt.Println("Parse error:", err)
break
}
fmt.Println("Got:", item["name"])
}
}
The loop breaks on io.EOF. This error means the decoder reached the end of the input while expecting a new value. It is the normal exit path, not a failure. If you treat io.EOF as an error, your program will report a false failure on every successful stream. Always check for io.EOF before handling other errors.
The decoder handles concatenated JSON seamlessly. It does not require an array wrapper. The input {"a":1}{"a":2} works perfectly. The decoder finds the first object, decodes it, stops at the closing brace, and waits for the next call. On the next call, it sees the second object and decodes that. This behavior matches JSON Lines format, where each line is a standalone JSON value.
The loop breaks on io.EOF. Always distinguish end-of-stream from parse errors.
How the decoder buffers data
The decoder does not read one byte at a time. That would be too slow. It maintains an internal buffer, typically a few kilobytes. When the buffer is empty, it performs a read from the underlying reader to fill it. Parsing happens on the buffer contents. When the buffer runs low, it refills.
This buffering strategy minimizes system calls. Reading a file byte-by-byte triggers a kernel transition for every byte. Buffering batches those reads into large chunks. The performance difference is massive for disk I/O and network connections.
The decoder also handles partial values. If the stream provides {"id": and then pauses, the decoder blocks waiting for more data. It does not return an error until the timeout expires or the connection closes. This blocking behavior is correct for streams. It ensures you get complete values.
When you decode into a struct, the decoder matches JSON keys to struct fields using tags. The tags follow the convention json:"field_name". If a tag is missing, the decoder uses the field name with case-insensitive matching. Structs give you type safety and better performance than map[string]interface{}. The map requires reflection for every field access and allocates memory for each value. Structs compile to direct field assignments.
Structs give you compile-time safety. map[string]interface{} is a runtime hazard.
Processing a file with structs
Here is a realistic example. The function reads a JSON Lines file containing event records. It uses a struct for type safety and processes only high-priority events. The code follows Go conventions for error handling and resource management.
package main
import (
"encoding/json"
"fmt"
"io"
"os"
)
// Event represents a structured log entry from the stream
type Event struct {
ID int `json:"id"`
Level string `json:"level"`
Msg string `json:"msg"`
}
// ProcessEvents reads a JSON Lines file and prints high-priority messages
func ProcessEvents(filename string) error {
// Open the file for reading; return early on permission errors
f, err := os.Open(filename)
if err != nil {
return fmt.Errorf("open file: %w", err)
}
// Ensure the file handle closes when the function returns
defer f.Close()
// Decoder reads incrementally from the file handle
dec := json.NewDecoder(f)
for {
var event Event
// Decode fills the struct from the next JSON object in the stream
if err := dec.Decode(&event); err == io.EOF {
// Stream finished; this is the normal exit path
break
}
if err != nil {
// Return early on parse errors; wrap for context
return fmt.Errorf("decode event: %w", err)
}
if event.Level == "ERROR" {
fmt.Printf("Alert: %s\n", event.Msg)
}
}
return nil
}
The defer f.Close() call guarantees the file handle releases even if an error occurs. This prevents file descriptor leaks. The error wrapping with %w preserves the error chain, allowing callers to inspect the root cause with errors.Is or errors.As.
The verbose if err != nil check is standard Go. The community accepts the boilerplate because it makes the unhappy path visible. You cannot accidentally swallow an error behind a silent assignment. Every error gets a decision point.
When processing long-running streams, consider cancellation. If this function runs in a goroutine, the caller might want to stop it early. Wrap the loop with a context check. Call ctx.Err() after each decode. If the context is cancelled, break the loop and return. Context is plumbing. Run it through every long-lived call site.
Encoding streams and JSON Lines
json.Encoder writes values to a stream. It is the counterpart to the decoder. You use it when generating output for a file, a network response, or a log pipeline.
Here is an encoder that writes a sequence of structs to standard output. The code disables HTML escaping, which is a common requirement for JSON Lines output.
package main
import (
"encoding/json"
"fmt"
"os"
)
func main() {
// Encoder writes to stdout, adding a newline after each value
enc := json.NewEncoder(os.Stdout)
// SetEscapeHTML disables escaping <, >, and & for cleaner output
enc.SetEscapeHTML(false)
items := []struct {
Name string `json:"name"`
Score int `json:"score"`
}{
{Name: "Alice", Score: 95},
{Name: "Bob", Score: 88},
}
for _, item := range items {
// Encode writes the JSON representation followed by a newline
if err := enc.Encode(item); err != nil {
fmt.Fprintf(os.Stderr, "encode error: %v\n", err)
return
}
}
}
By default, json.Encoder escapes HTML special characters. The string <script> becomes u003cscriptu003e. This behavior exists for security when serving JSON directly to browsers. It breaks downstream parsers that expect raw strings. If your JSON goes to logs, APIs, or files, call SetEscapeHTML(false). The output becomes readable and compatible with standard tools.
The encoder adds a newline after each value. This matches the JSON Lines specification. Each line is a valid JSON document. You can concatenate multiple encoder outputs and the result is still a valid stream. This makes encoders perfect for building pipelines where one process writes JSON Lines and another reads them.
Encoders produce JSON Lines by default. This is perfect for logs and pipelines.
Pitfalls: precision, escaping, and errors
Streaming JSON has a few traps. The first is numeric precision. Go maps JSON numbers to float64 by default. A float64 can represent integers exactly only up to 2^53. If your JSON contains larger integers, such as Snowflake IDs or financial amounts, the decoder silently loses precision.
Fix this by calling dec.UseNumber() before decoding. The decoder then stores numbers as json.Number, which is a string wrapper. You can parse the string to int64 or big.Int later. This preserves exact values.
dec.UseNumber()
var item map[string]interface{}
dec.Decode(&item)
// item["id"] is now json.Number, not float64
idStr, ok := item["id"].(json.Number)
if !ok {
// handle type assertion failure
}
id, err := idStr.Int64()
The second trap is the io.EOF check. If you decode into a slice, the behavior changes. Decode fills the slice with one array from the stream. If the stream ends, Decode returns io.EOF. If you forget to check io.EOF, you might treat the end of stream as a failure. Always check io.EOF explicitly.
The third trap is error messages. The compiler rejects type mismatches with clear text. If you try to decode an object into a string, you get json: cannot unmarshal object into Go value of type string. If you pass the wrong type to Decode, the compiler complains with cannot use x (type string) as type *Event in argument. These errors happen at compile time or runtime depending on the mistake. Read them carefully. They point directly to the type mismatch.
Big integers overflow float64. Use UseNumber when precision matters.
Decision matrix
Use json.Unmarshal when the payload fits in memory and you need the whole object at once. Use json.Decoder when reading a stream of JSON values or a file larger than available RAM. Use json.Encoder when writing a sequence of JSON values to a stream or file. Use json.Marshal when you need a JSON byte slice for a single value, like storing in a cache or sending as a response body.
Match the tool to the data shape. Stream the stream. Marshal the object.