The kitchen with too many orders
The dinner rush hits. The head chef holds a stack of tickets. Cooking every dish personally creates a bottleneck. The kitchen needs line cooks to grab tickets, cook, and plate dishes while the head chef manages the flow. This is the fan-out/fan-in pattern. You distribute a batch of work to multiple workers and collect the results back to a single point. In Go, goroutines handle the concurrent work and channels move the data between them.
How the pattern maps to Go
Fan-out splits tasks across concurrent workers. Fan-in merges results back together. The pattern converts a sequential loop into a concurrent pipeline. Go provides goroutines and channels to build this naturally. You need three components. A job channel feeds work to workers. A results channel collects output. A synchronization mechanism ensures the results channel closes only after every worker finishes. Without synchronization, the collector might read from a channel that never closes, or a worker might send on a channel that the collector already closed. Concurrency structures the code. Parallelism speeds it up. Fan-out/fan-in gives you both by running independent tasks simultaneously.
Channels have direction. jobs <-chan int means the worker can only receive. results chan<- int means the worker can only send. This restricts misuse. The compiler rejects code that tries to send on a receive-only channel with invalid operation: cannot send to receive-only type. This makes the interface explicit and prevents accidental writes.
Minimal example: doubling numbers
Here is the worker function. It reads from the job channel, processes each item, and sends the result back.
// worker reads jobs, doubles them, and sends results back.
func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
defer wg.Done() // Decrement counter when worker exits
for j := range jobs { // Range exits automatically when jobs channel closes
results <- j * 2 // Send result back to main goroutine
}
}
The worker uses defer wg.Done() to signal completion. This runs even if the function panics. The range loop blocks until the job channel closes. When the channel closes, the loop exits and the worker returns.
The main function sets up the channels and spawns the workers.
func main() {
jobs := make(chan int, 100) // Buffer prevents blocking if main sends faster than workers consume
results := make(chan int, 100)
var wg sync.WaitGroup
for w := 1; w <= 3; w++ {
wg.Add(1) // Track one more worker
go worker(w, jobs, results, &wg)
}
go func() {
for j := 1; j <= 10; j++ {
jobs <- j
}
close(jobs) // Signal workers that no more jobs exist
wg.Wait() // Block until all workers finish
close(results) // Signal main loop that results are done
}()
for r := range results {
fmt.Println(r)
}
}
The main function creates buffered channels. The buffer size prevents blocking if the sender is faster than the receiver. The loop spawns three goroutines. Each call to wg.Add(1) increments the counter before the goroutine starts. This prevents a race condition where the worker calls Done before Add runs.
The anonymous goroutine handles the lifecycle. The main goroutine cannot close the results channel because it is busy reading from it. The closer must run in a separate goroutine. It sends all jobs, closes the job channel, waits for workers, and then closes the results channel.
gofmt formats this code consistently. Run gofmt on save. The community expects standard formatting. Argue about logic, not indentation.
Channels flow data. WaitGroups track completion. Keep them separate.
Realistic example: checking service health
Real-world fan-out/fan-in involves I/O. This example checks the health of multiple URLs concurrently. It uses context to enforce a timeout and follows Go conventions for error handling.
// Result holds the outcome of a health check.
type Result struct {
URL string
Status int
}
// checkURL fetches a URL and sends the status code to results.
func checkURL(ctx context.Context, url string, results chan<- Result, wg *sync.WaitGroup) {
defer wg.Done()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
results <- Result{URL: url, Status: -1} // Send error code for invalid URL
return
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
results <- Result{URL: url, Status: -1} // Send error code for network failure
return
}
defer resp.Body.Close() // Prevent connection leak
results <- Result{URL: url, Status: resp.StatusCode}
}
The Result struct uses public fields. Go convention requires capital letters for exported names. The struct is passed by value because structs are cheap to copy in Go. The checkURL function accepts context.Context as the first parameter. Context is plumbing. Run it through every long-lived call site. The HTTP client respects context cancellation. If the timeout fires, pending requests abort immediately.
The function checks errors immediately. The pattern if err != nil { return err } is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible. Here, the function sends a result with a status code of -1 to indicate failure. The defer resp.Body.Close() call is mandatory. Leaked response bodies consume file descriptors and connections. The runtime reclaims them eventually, but a busy server exhausts resources long before garbage collection runs.
The main function creates a context with a five-second timeout.
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
urls := []string{"https://golang.org", "https://example.com"}
results := make(chan Result, len(urls))
var wg sync.WaitGroup
for _, u := range urls {
wg.Add(1)
go checkURL(ctx, u, results, &wg)
}
go func() {
wg.Wait()
close(results)
}()
for r := range results {
fmt.Printf("%s: %d\n", r.URL, r.Status)
}
}
The defer cancel() call releases resources when main returns. The results channel is buffered to len(urls). This ensures workers never block on sending results. The anonymous closer goroutine waits for all workers before closing results. This prevents a panic from sending on a closed channel.
Always close response bodies. Leaked connections kill servers.
Pitfalls and runtime panics
The pattern is simple to write but easy to break. The most common errors involve channel lifecycle and synchronization.
Sending on a closed channel causes a panic. If the closer goroutine closes the results channel before a worker finishes, the worker crashes when it tries to send. The runtime stops the program with panic: send on closed channel. This happens if you move close(results) outside the anonymous goroutine or if you forget wg.Wait(). The fix is strict ordering. The results channel must close only after every worker calls wg.Done().
Forgetting to close the job channel causes a hang. If you never close the job channel, the workers never exit. The range loop blocks forever waiting for more jobs. The program hangs silently. There is no compiler error. The worst goroutine bug is the one that never logs. Always close the input channel after sending all jobs.
Deadlock occurs when goroutines block on channels with no progress. If the results channel is unbuffered and the main goroutine is slow to read, workers block on sending. If all workers block and the main goroutine waits for wg.Wait(), the program deadlocks. The runtime detects this and panics with all goroutines are asleep - deadlock!. Buffer the results channel to the number of workers or jobs. This gives workers room to deposit results without blocking.
Goroutine leaks happen when a goroutine waits on a channel that never gets closed. If you forget to close the job channel, workers leak. If you forget to cancel a context, HTTP requests leak. Always have a cancellation path. Use context for timeouts and close for channels.
A closed channel is a signal, not a cleanup. Send on it and you crash.
When to use fan-out/fan-in
Choosing the right concurrency pattern depends on the task structure and error requirements.
Use fan-out/fan-in when you have a batch of independent tasks that can run concurrently and you need to collect all results.
Use a sequential loop when the tasks depend on each other or the overhead of goroutines outweighs the benefit.
Use a worker pool with a bounded job channel when you need to limit concurrency to protect a downstream service from overload.
Use golang.org/x/sync/errgroup when you want to stop processing immediately if any single task fails.