How to Fan Out and Fan In with Channels in Go

Fan out distributes work to multiple goroutines via a shared channel, while fan in collects results from those goroutines into a single channel. The compileFunctions pattern in the Go compiler demonstrates this by spawning workers that pull tasks from compilequeue and pushing closures back into it.

var mu sync.Mutex
var wg sync.WaitGroup

// Fan out: spawn workers
for i := 0; i < numWorkers; i++ {
    wg.Add(1)
    go func() {
        defer wg.Done()
        for {
            mu.Lock()
            if len(compilequeue) == 0 {
                mu.Unlock()
                return
            }
            fn := compilequeue[len(compilequeue)-1]
            compilequeue = compilequeue[:len(compilequeue)-1]
            mu.Unlock()
            
            // Process
            ssagen.Compile(fn, i, profile)
            
            // Fan in: push closures back to queue
            mu.Lock()
            compilequeue = append(compilequeue, fn.Closures...)
            mu.Unlock()
        }
    }()
}

wg.Wait()

Fan out is like a manager handing out tasks to a team of workers so they can work in parallel. Fan in is the process of gathering all the completed work back together into one place. This pattern lets you process a large list of items quickly by using all your CPU cores at once.