Iterator Patterns: Map, Filter, Take, Skip in Go
You are processing a massive log file. You need the first ten lines that contain "PANIC", and for each match, you want to extract the timestamp and write it to a database. Loading the entire file into a slice of strings eats memory you don't need. Writing a manual loop with state variables for counting, filtering, and formatting creates a tangle of logic that is hard to read and harder to reuse. You want to describe the pipeline as a chain of steps. You want the program to stop reading the file the moment it finds ten matches. You want to keep memory usage flat regardless of file size.
Go's iter package solves this with lazy iterators. Functions like Map, Filter, Take, and Drop let you chain transformations on a sequence. The chain runs item by item. Nothing happens until you consume a value. If you stop consuming, the chain stops producing. This pull-based design gives you efficiency and modularity without the overhead of objects or interfaces.
How iterators work in Go
An iterator in Go is not an object with a Next method. It is a function. The type iter.Seq[T] is defined as func(yield func(T) bool). This signature looks unusual at first. The iterator function takes another function as an argument. That inner function is called yield.
When the iterator has a value ready, it calls yield(value). The yield function returns a boolean. If it returns true, the iterator should keep going. If it returns false, the consumer has stopped, and the iterator should return immediately. This mechanism lets the consumer control the pace. The iterator pushes values, but the consumer pulls by deciding when to stop.
This design avoids allocations. iter.Seq is a function type, not an interface. Interfaces require a heap allocation for the receiver and prevent inlining. Functions are cheaper and allow the compiler to inline the iteration logic. The trade-off is that you cannot define a custom iterator as a struct with methods. You define it as a function that returns an iter.Seq. This keeps the abstraction zero-cost.
Iterators are lazy. The chain runs only when you pull.
Minimal example
Here is the simplest way to chain iterators. We start with a slice, convert it to a sequence, filter for even numbers, double the values, and take only the first two results.
package main
import (
"fmt"
"iter"
"slices"
)
func main() {
// slices.Values creates an iterator over the slice without copying data.
// This avoids allocating a new slice for the sequence.
nums := slices.Values([]int{1, 2, 3, 4, 5, 6})
// Filter keeps only elements where the predicate returns true.
// The chain stops early if Take limits the output.
evens := iter.Filter(nums, func(n int) bool {
return n%2 == 0
})
// Map transforms each element.
// The type stays int here, but Map can change types too.
doubled := iter.Map(evens, func(n int) int {
return n * 2
})
// Take limits the output to exactly two items.
// Once two items are yielded, the source iterator stops running.
limited := iter.Take(doubled, 2)
// Range consumes the iterator.
// Laziness means Filter and Map only execute when Range asks for a value.
for v := range limited {
fmt.Println(v)
}
}
The output is 4 and 8. The iterator produces 2, doubles it to 4, yields it. It produces 4, doubles it to 8, yields it. Take has reached its limit. The chain halts. The values 6 and beyond are never processed.
Walking through the runtime
When you write for v := range seq, the compiler generates code that calls seq with a closure. The closure captures the loop variable. Inside seq, you call yield. The closure assigns the value to v and continues the loop. The closure returns true. If the loop breaks, the closure returns false. The next time yield is called, it returns false, and seq exits.
This mechanism enables Take to work efficiently. Take wraps the source iterator. It counts how many times yield is called. Once the count hits the limit, Take stops calling the source's yield. The source sees false and stops. The chain halts instantly. There is no buffering. There is no extra work.
The yield function is the remote control. Respect the stop signal.
Realistic pipeline
In real code, you often process streams of data. The bytes package provides Lines, which returns an iterator over []byte values. This is more efficient than splitting a string into a slice of strings. bytes.Lines yields slices that point directly into the original buffer. No allocation per line. You can chain Filter and Map to process the data lazily.
Here is a log processing pipeline. It reads bytes, filters for errors, converts to strings, and limits the output.
package main
import (
"bytes"
"fmt"
"iter"
"strings"
)
// ProcessLog streams log lines, filters errors, and extracts messages.
// It demonstrates chaining iterators on byte slices for efficiency.
func ProcessLog(data []byte) iter.Seq[string] {
// bytes.Lines yields []byte for each line.
// This avoids allocating a string for every line in the input.
lines := bytes.Lines(data)
// Filter keeps only lines containing "ERROR".
// bytes.Contains works on []byte, keeping the pipeline allocation-free.
errors := iter.Filter(lines, func(line []byte) bool {
return bytes.Contains(line, []byte("ERROR"))
})
// Map converts []byte to string and trims whitespace.
// This is where the type changes from []byte to string.
// Allocation happens here, but only for matching lines.
messages := iter.Map(errors, func(line []byte) string {
return strings.TrimSpace(string(line))
})
return messages
}
func main() {
logData := []byte("INFO: start\nERROR: disk full\nDEBUG: check\nERROR: timeout\nINFO: end\n")
// Take the first two error messages.
// The iterator stops reading logData after finding two matches.
// This prevents scanning the rest of the buffer unnecessarily.
topErrors := iter.Take(ProcessLog(logData), 2)
for msg := range topErrors {
fmt.Println(msg)
}
}
The output is ERROR: disk full and ERROR: timeout. The pipeline processes bytes until it finds two errors. The INFO: end line is never touched. Memory usage stays constant. The Map function allocates strings only for the lines that pass the filter.
Order matters in a pipeline. Filter before Map if you can skip work.
Pitfalls and compiler errors
Iterators are powerful, but they have quirks. Understanding the pitfalls prevents subtle bugs.
Closure capture. Iterators often use closures for Filter and Map. If your closure captures a variable that changes, you get bugs. Go 1.22 fixed loop variable capture in for loops, but iterators can still capture state from outer scopes. If you define a filter inside a loop and capture the loop variable, ensure the variable does not change before the iterator runs.
Infinite sources. The iter package provides iter.Infinite for generating endless sequences. If you chain Map on an infinite source and range without Take, your program runs forever. The compiler cannot detect this. You must ensure the chain terminates. Always pair infinite iterators with Take or a Filter that eventually stops.
Type mismatches. iter.Map requires the function to return the target type. If you pass a function that returns the wrong type, the compiler rejects it. The error message is verbose but clear.
The compiler rejects the code with
cannot use func(int) int { ... } as func(int) string value in argument to iter.Mapif the return types do not match.
Seq vs Seq2. The iter package defines iter.Seq[T] for single values and iter.Seq2[K, V] for key-value pairs. Functions like maps.All return Seq2. The transformation functions in iter work on Seq, not Seq2. You cannot pass a Seq2 to iter.Map. If you need to transform key-value pairs, you must write a custom iterator or use helper functions.
Passing a
Seq2toMaptriggerscannot use ... (variable of type iter.Seq2[K, V]) as iter.Seq[T] value in argument.
Discarding values. If you have a Seq2 but only need keys, you might be tempted to ignore values. Use the underscore to discard intentionally.
// keysOnly iterates over a map and yields only keys.
// The underscore discards the value, signaling intentional ignore.
func keysOnly(m map[string]int) iter.Seq[string] {
return func(yield func(string) bool) {
for k := range m {
if !yield(k) {
return
}
}
}
}
The underscore says "I considered the value and chose to drop it." Use it sparingly with errors, but freely with iterator values you don't need.
Iterators stream data. Loops stream control. Pick the right tool.
When to use iterators
Go gives you multiple ways to process collections. Iterators are not always the best choice. Use the right tool for the job.
Use iter.Map when you need to transform every element in a sequence without changing the count. Use iter.Filter when you need to discard elements based on a condition while preserving the order of the rest. Use iter.Take when you only need a fixed number of items and want to stop processing early to save resources. Use iter.Drop when you need to ignore the first N items before processing the rest. Use iter.Concat when you need to join two sequences into one stream. Use a plain for loop when the logic involves complex state, side effects, or early breaks that iterators make awkward. Use slices or maps package functions when you have a concrete collection and need a result all at once, rather than streaming.
Iterators are cheap. Loops are explicit. Choose based on clarity and performance.