How to Fan Out and Fan In with Channels in Go
You have a directory of 1,000 images that need resizing. Processing them sequentially takes ten minutes. You have eight cores sitting idle. You spawn eight goroutines to chew through the work in parallel. Now you face a new problem: how do you gather the resized images back into a single stream without race conditions, deadlocks, or losing data? Fan out and fan in solves this by splitting work across workers and merging results into one output.
Think of a restaurant kitchen. Orders arrive on a ticket rail. Multiple cooks stand behind the rail, each grabbing the next available ticket. That's the fan out: one source of work distributed to many workers. When a dish is ready, the cook places it on a hot holding shelf. The server stands at the shelf, taking dishes as they appear to deliver to tables. That's the fan in: many producers merging into a single stream for the consumer. Channels implement this pattern without locks. The ticket rail is a channel for tasks. The holding shelf is a channel for results.
Minimal worker pool
Here's the skeleton: a worker pool that squares numbers and merges results.
package main
import "fmt"
func worker(id int, jobs <-chan int, results chan<- int) {
// Range over jobs blocks until a value arrives or the channel closes.
for j := range jobs {
// Simulate work by squaring the input value.
result := j * j
// Send result to the shared results channel.
results <- result
}
}
func main() {
jobs := make(chan int, 100)
results := make(chan int, 100)
// Fan out: start three workers pulling from jobs.
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
// Send five jobs into the channel.
for j := 1; j <= 5; j++ {
jobs <- j
}
close(jobs) // Signal no more jobs so workers can exit.
// Fan in: collect all results.
for a := 1; a <= 5; a++ {
fmt.Println(<-results)
}
}
The jobs channel acts as the queue. When main sends values, they buffer until a worker receives them. The range loop in worker blocks until a job arrives. Once main closes jobs, the range loop detects the close and exits, allowing the goroutine to finish. The results channel collects outputs. The final loop in main pulls exactly five results. If you try to pull six, the program deadlocks because no more values are coming. Channels enforce the contract: the number of sends must match the number of receives, or you hang.
Channels are pipes, not magic. Close the input or the workers starve.
Buffering strategy
Channels can be buffered or unbuffered. The choice changes the flow control. An unbuffered channel synchronizes the sender and receiver. The send blocks until a receive happens. A buffered channel allows sends to proceed until the buffer fills.
In fan-out, an unbuffered jobs channel means main pauses while workers catch up. This provides natural backpressure. If workers are slow, main slows down. A buffered jobs channel lets main dump all jobs quickly. This improves throughput if workers are fast, but risks memory exhaustion if workers lag.
In fan-in, the results channel often needs buffering. If the results channel is unbuffered, a worker finishes and tries to send, but main hasn't started reading yet. The worker blocks. If all workers block, and main is waiting for all workers to finish before reading, you deadlock. Buffering the results channel to the number of workers prevents this. Each worker can send one result and exit, even if main hasn't started collecting.
Buffer the results channel to the worker count. Unbuffered results invite deadlock when workers finish faster than the collector starts.
Realistic example with errors and context
Here's a realistic scenario: fetching URLs in parallel and collecting responses with error handling.
package main
import (
"context"
"fmt"
"net/http"
)
type Result struct {
URL string
Code int
Error error
}
func fetch(ctx context.Context, url string, results chan<- Result) {
// Context is plumbing. Run it through every long-lived call site.
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
results <- Result{URL: url, Error: err}
return
}
// Perform the HTTP request.
resp, err := http.Get(req.URL.String())
result := Result{URL: url}
if err != nil {
// Wrap error context if needed, though http.Get usually suffices.
result.Error = err
} else {
// Capture status code and close body to prevent resource leaks.
result.Code = resp.StatusCode
resp.Body.Close()
}
// Send result to unblock the worker and allow the fan-in loop to proceed.
results <- result
}
The if err != nil check is verbose by design. It forces you to acknowledge the failure path. In a real service, you might wrap the error with fmt.Errorf("fetch %s: %w", url, err) to add context. The resp.Body.Close() call is mandatory. Forgetting it leaks file descriptors. The community accepts the boilerplate because it makes the unhappy path visible.
Context is plumbing. Run it through every long-lived call site.
func main() {
urls := []string{"https://golang.org", "https://example.com"}
// Buffer matches input size so workers never block on send.
results := make(chan Result, len(urls))
// Fan out: spawn a goroutine per URL.
for _, url := range urls {
go fetch(context.Background(), url, results)
}
// Fan in: collect results matching the input count.
for range urls {
res := <-results
if res.Error != nil {
fmt.Printf("%s failed: %v\n", res.URL, res.Error)
} else {
fmt.Printf("%s returned %d\n", res.URL, res.Code)
}
}
}
The results channel buffers to len(urls). This guarantees that every worker can send its result and exit without waiting for main to read. The fan-in loop iterates exactly len(urls) times. The range urls loop is safe in Go 1.22+ because the loop variable is scoped per iteration. In older versions, you would need to capture the variable explicitly to avoid all goroutines receiving the last URL.
Trust gofmt. The indentation in the worker loop is decided by the tool, not your preference. Most editors run it on save.
Pitfalls and compiler errors
Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Always have a cancellation path. If you forget to close the jobs channel, workers block forever on range. The runtime panics with fatal error: all goroutines are asleep - deadlock! if the main goroutine exits while workers are still blocked on a channel receive.
Directional channel types prevent accidental misuse. If you try to send to a receive-only channel, the compiler rejects the code with cannot send to receive-only type <-chan int. Use directional types in function signatures to enforce flow. The worker function accepts <-chan int for jobs and chan<- int for results. This tells the compiler and the reader that the worker only receives jobs and only sends results.
Unbuffered channels can cause deadlocks in fan-in if the collector starts after workers finish. If results is unbuffered and main does close(jobs) then loops to read results, a fast worker might finish, try to send, and block because main hasn't started the receive loop yet. If all workers block, and main is waiting for workers to finish before reading, the program hangs. Buffering or running the fan-in in a separate goroutine solves this.
The worst goroutine bug is the one that never logs. Add logging or metrics to track worker completion and channel activity.
When to use fan out and fan in
Use a fan-out/fan-in pattern when you have independent tasks that can run in parallel and you need to aggregate results. Use a worker pool with a bounded channel when you must limit concurrency to protect a downstream service from overload. Use errgroup when you need structured error handling and cancellation across a group of goroutines. Use sequential code when the tasks are fast or dependent on each other: the overhead of channels and goroutines outweighs the benefit. Use a single goroutine with a pipeline when one stage feeds the next without branching.
Fan out and fan in scales work, but it also scales complexity. Keep the topology simple.