Worker pool with channels

The coffee shop queue

You have a list of five thousand URLs to scrape. You write a loop, spawn a goroutine for each URL, and hit run. The program crashes. The garbage collector screams. The remote server rate-limits you into oblivion. You just created fifty thousand goroutines to do work that could have been handled by ten.

Unbounded concurrency is a trap. It feels fast until it isn't. The system runs out of file descriptors, memory fragments, and the CPU spends more time scheduling goroutines than executing them. You need a throttle. You need a way to say, "I have infinite work, but I only want ten workers running at a time."

That is a worker pool. A worker pool uses a fixed number of goroutines to consume tasks from a shared queue. The queue is a channel. The workers are goroutines. The pool size is the concurrency limit. This pattern protects your resources, respects downstream rate limits, and keeps your memory footprint predictable.

How the pool works

Think of a coffee shop. Customers arrive and join a line. The line is the channel. Baristas stand behind the counter making drinks. The baristas are the goroutines. If you hire a new barista for every customer who walks in, the shop explodes. If you have three baristas, the line grows, but the shop stays manageable. The line length determines how much work can be queued before new customers have to wait outside.

In Go, the channel holds the jobs. The goroutines read from the channel. When a goroutine finishes a job, it loops back and grabs the next one. When the channel is empty and closed, the goroutine exits. The main function feeds jobs into the channel, waits for all workers to finish, and collects the results.

The synchronization relies on three mechanics:

1. Range over a channel: A for range loop blocks until the channel is closed and drained. This keeps workers alive and waiting for work.
2. Close: Closing a channel signals to receivers that no more values will be sent. It unblocks the range loop.
3. WaitGroup: A sync.WaitGroup tracks how many workers are still running. The main function waits on the group before closing the results channel.

Minimal worker pool

Here is the simplest worker pool. It spawns three workers, feeds them five jobs, and collects the results. The code is compact, but every line has a job.

package main

import (
	"fmt"
	"sync"
)

// worker reads from jobs, processes, and writes to results
// Receiver name is omitted because this is a plain function, not a method
func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
	defer wg.Done() // Decrement counter when the worker exits
	for j := range jobs { // Range blocks until jobs is closed and empty
		results <- j * 2 // Send result; blocks if results channel is full
	}
}

func main() {
	jobs := make(chan int, 100) // Buffer prevents main from blocking on first send
	results := make(chan int, 100)
	var wg sync.WaitGroup

	// Launch fixed number of workers
	for w := 1; w <= 3; w++ {
		wg.Add(1) // Increment counter before starting goroutine
		go worker(w, jobs, results, &wg)
	}

	// Feed jobs into the pool
	for j := 1; j <= 5; j++ {
		jobs <- j
	}
	close(jobs) // Signal no more jobs; workers will finish their range loops

	// Close results after all workers are done
	go func() {
		wg.Wait() // Block until all workers call Done
		close(results) // Safe to close now that no one is sending
	}()

	// Collect output
	for r := range results {
		fmt.Println(r)
	}
}

The buffer size of 100 is arbitrary. It allows the main function to send all five jobs without blocking, even if the workers haven't started yet. If you make the channel unbuffered, the main function blocks on the first send until a worker is ready to receive. That's fine for small workloads, but a buffer decouples the producer from the consumer and smooths out bursts.

Walking through the lifecycle

The program starts by creating two channels and a WaitGroup. The WaitGroup counter is zero. The main function launches three goroutines. Each goroutine increments the counter with wg.Add(1) and starts the worker function.

The workers immediately hit the for j := range jobs loop. The jobs channel is empty, so the workers block. They are alive, waiting for data.

The main function sends five integers into jobs. Because the channel is buffered, these sends succeed immediately. The workers wake up, grab values, double them, and send them to results. The results channel is also buffered, so the workers can send without blocking, as long as the buffer isn't full.

After sending all five jobs, the main function calls close(jobs). This is the signal. The workers are still processing the items in the channel. When a worker finishes an item and loops back to range, it sees the channel is closed and empty. The loop exits. The defer wg.Done() runs, decrementing the counter.

Meanwhile, the main function launches a goroutine to wait on the WaitGroup. This goroutine blocks until the counter hits zero. Once all three workers have exited, wg.Wait() returns, and the goroutine calls close(results).

The main function is already ranging over results. It collects values until the channel closes, then the loop ends. The program exits cleanly.

Goroutines are cheap. Channels are not magic. The discipline comes from closing channels in the right order.

Realistic example with context and errors

Real code deals with errors, cancellation, and structured data. You also need to respect the context.Context convention. Contexts always go as the first parameter, conventionally named ctx. Functions that take a context should check for cancellation and respect deadlines.

Here is a pool that processes jobs with a Processor interface. The interface allows you to swap out the work logic without changing the pool machinery. This follows the "accept interfaces, return structs" mantra.

package main

import (
	"context"
	"fmt"
	"sync"
)

// Job represents a unit of work
type Job struct {
	ID    int
	Value int
}

// Result holds the outcome
type Result struct {
	JobID int
	Output int
}

// Processor defines the work contract
// Accept interfaces: the worker accepts a Processor, not a concrete type
type Processor interface {
	Process(ctx context.Context, job Job) (Result, error)
}

// HeavyLifter implements Processor
type HeavyLifter struct{}

// Process handles computation
// Receiver name is short: (h HeavyLifter) matches the type HeavyLifter
func (h HeavyLifter) Process(ctx context.Context, job Job) (Result, error) {
	// Check context cancellation before starting work
	select {
	case <-ctx.Done():
		return Result{}, ctx.Err()
	default:
	}

	// Simulate work
	output := job.Value * 2
	return Result{JobID: job.ID, Output: output}, nil
}

// worker consumes jobs and produces results
func worker(ctx context.Context, id int, jobs <-chan Job, results chan<- Result, wg *sync.WaitGroup) {
	defer wg.Done()
	for job := range jobs {
		// Respect context cancellation per iteration
		select {
		case <-ctx.Done():
			return
		default:
		}

		p := HeavyLifter{}
		res, err := p.Process(ctx, job)
		if err != nil {
			// Log error and continue; don't crash the pool
			fmt.Printf("worker %d failed job %d: %v\n", id, job.ID, err)
			continue
		}
		results <- res
	}
}

The worker function checks ctx.Done() inside the loop. If the context is cancelled, the worker exits immediately. This prevents goroutine leaks. A goroutine leak happens when a goroutine waits on a channel that never gets closed. Always have a cancellation path.

The error handling is explicit. if err != nil is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible. The worker logs the error and continues. It doesn't panic. If one job fails, the pool keeps running.

Don't fight the type system. Wrap the value or change the design.

Pitfalls and compiler errors

Worker pools introduce synchronization complexity. A single mistake causes deadlocks or panics.

Sending on a closed channel If you close a channel and then try to send to it, the program panics. The runtime stops with panic: send on closed channel. This happens if you close jobs before all workers have finished reading, or if you close results while a worker is still sending. The WaitGroup pattern prevents this by ensuring results closes only after all workers exit.

Deadlocks A deadlock occurs when all goroutines are blocked. The runtime detects this and panics with all goroutines are asleep - deadlock!. Common causes:

• Unbuffered channels where the sender and receiver are out of sync.
• Forgetting to close a channel, leaving workers blocked on range.
• Closing a channel too early, causing workers to exit before the main function is ready.

Goroutine leaks If the main function exits while workers are still running, the workers become zombies. They consume memory and file descriptors until the process dies. This happens if you don't close the jobs channel, or if you don't wait on the WaitGroup. The worst goroutine bug is the one that never logs.

Loop variable capture If you use a for loop to spawn workers and pass the loop variable by reference, you might capture the wrong value. The compiler rejects this with loop variable i captured by func literal in Go 1.22+. Always pass the variable by value, or use the loop variable directly in the goroutine call.

Trust gofmt. Argue logic, not formatting. The formatter handles indentation and style. You handle the concurrency.

When to use a worker pool

Concurrency is a tool, not a default. Choose the right pattern for the workload.

Use a worker pool when you have a large batch of independent tasks and need to limit concurrency to protect a downstream service or your own resources. Use a worker pool when the tasks are CPU-bound and you want to match the number of workers to the number of CPU cores. Use a worker pool when you need to process items in parallel but maintain a bounded memory footprint.

Use a single goroutine plus a channel when one task feeds another in a pipeline. Use a single goroutine when the work is sequential and concurrency adds no value. Use plain sequential code when you don't need concurrency: the simplest thing that works is usually the right thing.

Use errgroup.Group from golang.org/x/sync/errgroup when you need to stop the pool if any worker fails. The standard WaitGroup doesn't propagate errors. errgroup adds error handling and context cancellation to the pool pattern. Use errgroup when the first error should cancel all remaining work.

Context is plumbing. Run it through every long-lived call site.

Where to go next

• How to Apply the Principle of Least Privilege in Go API Design
• How the Go Compiler Works: Phases and Pipeline
• How to Use Multiple Go Versions on the Same Machine

A worker pool is like a team of employees (goroutines) waiting at a single desk (channel) for tasks. When a task arrives, the next available employee grabs it and works on it immediately. This ensures you never have more people working than you have desks, keeping your system stable and efficient.