Lazy sequences with range-over-function
You are processing a stream of log entries from a high-throughput service. The stream never ends, and you only need the first fifty lines containing a specific error code. Loading the entire stream into a slice would exhaust memory. You need a mechanism that produces values one by one, computes each value only when requested, and stops immediately once you have what you need.
Go 1.23 introduced range-over-function iterators to solve this. A lazy sequence defers computation until the consumer asks for the next item. This contrasts with a slice, which holds all items in memory at once. Think of a slice as a bucket filled with water. Think of an iterator as a hose connected to a tap. The hose delivers water on demand. You can turn the tap off at any moment without wasting water. In Go, an iterator is a function that returns a closure. The closure accepts a yield function and calls it for each item. The yield function runs the loop body. If the loop body breaks, yield returns false. The closure sees false and returns. The iteration stops.
The mechanism: yield and break
The type iter.Seq[T] is defined as func(yield func(T) bool). This signature encodes the entire protocol. The outer function returns a closure. The closure takes a yield function as an argument. The closure calls yield with each value. The yield function executes the loop body and returns true if the loop should continue, or false if the loop broke.
Here's the simplest lazy iterator: a function that generates integers up to a limit, yielding them one by one.
package main
import "fmt"
// RangeInts returns an iterator that yields integers from start up to limit.
func RangeInts(start, limit int) func(yield func(int) bool) {
// closure captures start and limit from the outer scope
return func(yield func(int) bool) {
for i := start; i < limit; i++ {
// yield returns false if the loop breaks, signaling the iterator to stop
if !yield(i) {
return
}
}
}
}
func main() {
// range calls RangeInts, passes the loop body as yield, and iterates
for v := range RangeInts(1, 5) {
fmt.Println(v)
}
}
The for v := range RangeInts(1, 5) syntax hides the complexity. The compiler rewrites this call. It invokes RangeInts(1, 5) to get the closure. It passes the loop body as the yield argument. The closure runs. It calls yield(1). The loop body executes. v is 1. The loop continues. yield returns true. The closure calls yield(2). This repeats until i reaches limit.
If the loop body calls break, yield returns false. The closure checks the return value. It returns immediately. No more calls to yield. The values 3 and 4 are never computed. This is the lazy behavior. The iterator does not precompute values. It computes only what the consumer requests.
Lazy sequences compute on demand. Slices compute upfront. Match the strategy to the data size.
Building a pipeline
Iterators shine when you compose operations. You can filter, map, and take items without allocating intermediate slices. Each operation wraps the previous iterator. The composition builds a chain of closures. When you iterate, the chain executes from the end to the start. The consumer pulls a value. The last operation asks the previous one for a value. That operation asks the one before it. Values flow backward through the chain until the source produces one.
Here's a filter function that keeps only items satisfying a predicate.
// Filter returns an iterator that yields items from seq where pred returns true.
func Filter[T any](seq func(yield func(T) bool), pred func(T) bool) func(yield func(T) bool) {
return func(yield func(T) bool) {
// seq drives the iteration; Filter intercepts values via the inner yield
seq(func(item T) bool {
if pred(item) {
// yield returns the result of the loop body, propagating break signals
return yield(item)
}
// return true to continue iterating even if this item is dropped
return true
})
}
}
The Filter function returns a closure. When the consumer calls yield, the closure calls seq with a wrapper function. The wrapper checks the predicate. If the item passes, it calls the consumer's yield. If the item fails, it returns true to keep iterating. This ensures the source continues producing values even when items are filtered out.
Here's a take function that limits the number of items.
// Take returns an iterator that yields the first n items from seq.
func Take[T any](seq func(yield func(T) bool), n int) func(yield func(T) bool) {
return func(yield func(T) bool) {
count := 0
seq(func(item T) bool {
if count >= n {
// return false to stop the source iterator once limit is reached
return false
}
count++
return yield(item)
})
}
}
The Take function tracks a count. It stops the source iterator once n items have been yielded. This prevents unnecessary computation in the source. If the source is an infinite sequence, Take ensures the iteration terminates.
You can compose these functions to build a pipeline. The pipeline filters errors and takes the first two.
func main() {
// source simulates a stream of log lines
lines := func(yield func(string) bool) {
for _, line := range []string{"INFO start", "ERROR crash", "INFO ok", "ERROR panic"} {
if !yield(line) {
return
}
}
}
// pipeline composes Filter and Take to process the stream lazily
pipeline := Take(Filter(lines, func(s string) bool {
return len(s) > 4 && s[:5] == "ERROR"
}), 2)
for line := range pipeline {
fmt.Println(line)
}
}
The pipeline executes lazily. The for loop calls yield. Take checks the count. Take calls Filter. Filter checks the predicate. Filter calls lines. lines yields a value. The value flows back through Filter and Take to the loop body. If the loop breaks, the signal propagates back through the chain, stopping all iterators. No intermediate slices are allocated. The memory usage stays constant regardless of the stream size.
Accept interfaces, return structs. Iterators follow a similar pattern: accept iter.Seq as input, return iter.Seq as output. This keeps functions composable and flexible.
Pitfalls and runtime checks
Iterators introduce subtle bugs if you violate the protocol. The most common error is calling yield twice. The range loop expects exactly one value per iteration. If your iterator calls yield twice, the loop state gets corrupted. The runtime detects this and panics.
The runtime panics with range-over-func: yield called twice if your iterator invokes the yield function more than once per iteration. This usually happens when you forget to return after calling yield, or when you call yield in multiple branches without guards.
Another pitfall is infinite loops. If your iterator never calls yield and never returns, the range loop hangs. This can happen if you filter out all items but forget to return true to continue. The iterator waits for the next item, but the source has finished. The program deadlocks.
Goroutine leaks are a risk when iterators spawn background work. If an iterator starts a goroutine to produce values, but the consumer breaks early, the goroutine might keep running. You need a cancellation path. Pass a context.Context to the iterator, or use a channel to signal cancellation. The context should always go as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines.
The worst goroutine bug is the one that never logs. Always ensure iterators clean up resources when the iteration stops.
Call yield once per item. Respect the break signal. Keep the iterator stateless.
Choosing the right iteration strategy
Go offers multiple ways to iterate. Pick the tool that matches your data flow and performance requirements.
Use a slice when you have a small, fixed dataset that fits in memory and you need random access or multiple passes. Slices are fast and simple. The compiler optimizes slice range loops aggressively.
Use iter.Seq when you need lazy evaluation to save memory or support infinite sequences. Iterators defer computation and compose well. They are ideal for pipelines where you filter, map, or limit data without allocating intermediates.
Use a channel when you need concurrent production and consumption across goroutines. Channels buffer data and synchronize access. They are essential for fan-out, fan-in, and worker pool patterns.
Use a callback function when you need a simple one-off iteration without the overhead of an iterator type. Callbacks are lightweight and explicit. They work well for one-time traversals where composition is not needed.
Slices are fast. Iterators are flexible. Channels are concurrent. Choose the tool that matches your data flow.