The problem with blocking code
You write a script that processes a list of files. It reads one, transforms it, writes it out, then moves to the next. It works fine for ten files. You run it on a thousand files and the terminal just sits there. The CPU sits idle while the disk spins. The program is stuck waiting for the slowest step to finish before it can start the next one. That is the exact moment you need a background worker.
How workers actually work
A background worker is just a separate thread of execution that waits for work to arrive, processes it, and hands the result back. Think of a coffee shop with one barista. Customers line up. The barista takes an order, makes it, hands it over, then takes the next order. Now imagine you hire three baristas. Customers still line up, but three orders get processed at the same time. The line moves faster. The kitchen does not explode because you control how many baristas you hire. In Go, the baristas are goroutines. The line is a channel. The orders are your jobs.
Goroutines are lightweight. The runtime multiplexes thousands of them onto a handful of operating system threads. Spawning one costs a few kilobytes of stack space and a fraction of a millisecond. You do not need to worry about thread exhaustion the way you would in Java or C. The tradeoff is coordination. You cannot share memory between goroutines safely without explicit synchronization. Channels provide that synchronization. They are the conveyor belt that moves work from the main program into the workers and back out again.
Channels coordinate. Goroutines execute.
The minimal worker pool
Here is the simplest worker pattern: spawn a fixed number of goroutines, feed them jobs through a channel, and collect the results.
package main
import "fmt"
// worker reads jobs from a channel and sends results back.
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs { // blocks until a job arrives or the channel closes
fmt.Printf("worker %d processing job %d\n", id, j)
results <- j * 2 // sends result back; blocks if results channel is full
}
}
func main() {
jobs := make(chan int, 10) // buffered so senders don't block immediately
results := make(chan int, 10)
for w := 1; w <= 3; w++ { // spawn three independent goroutines
go worker(w, jobs, results)
}
for j := 1; j <= 5; j++ { // feed five jobs into the system
jobs <- j
}
close(jobs) // signal that no more jobs are coming
for a := 1; a <= 5; a++ { // collect exactly five results
fmt.Println("result:", <-results)
}
}
The code above demonstrates the core loop. The range keyword on a channel keeps pulling values until the channel is both closed and empty. When that happens, the loop exits cleanly and the goroutine terminates. The buffer size of ten on both channels prevents the main goroutine from blocking while waiting for a worker to pick up a job or finish processing one. You adjust the buffer to match your throughput needs.
What happens under the hood
When the program starts, main runs on the primary goroutine. The three go worker calls ask the scheduler to run worker concurrently. The scheduler places them in a queue and assigns them to available OS threads. Meanwhile, main continues executing the job-feeding loop.
Each jobs <- j attempt checks the channel buffer. If space exists, the value lands in the buffer and main moves to the next iteration. If the buffer fills up, main blocks until a worker pulls a value out. Workers are blocked on range jobs until a value appears. When a value arrives, the worker wakes up, processes it, and attempts to send to results. The same buffer logic applies. When main calls close(jobs), it does not drain the channel. It simply marks it as closed. Workers continue processing any remaining buffered jobs. Once the buffer empties, range detects the closed state and exits the loop.
The runtime handles all the thread switching. You never see pthread_create or std::thread. You just get deterministic blocking behavior wrapped in a highly optimized scheduler. If you forget to close the input channel, the workers sit forever waiting for more jobs. That is a goroutine leak. Always have a cancellation path.
A realistic background processor
Production code rarely processes bare integers. It handles structs, makes network calls, and deals with failures. Here is how a worker looks when it touches real infrastructure.
package main
import (
"context"
"fmt"
)
// Job represents a unit of background work.
type Job struct {
ID string
Data string
}
// process handles a single job and returns an error.
func process(ctx context.Context, j Job) error {
select { // check for cancellation before doing expensive work
case <-ctx.Done():
return ctx.Err()
default:
fmt.Printf("processing %s with data %s\n", j.ID, j.Data)
return nil
}
}
// worker loops over jobs and reports errors back to the caller.
func worker(ctx context.Context, jobs <-chan Job, errs chan<- error) {
for j := range jobs {
if err := process(ctx, j); err != nil {
errs <- err // send error back; non-blocking if buffered
}
}
}
This version introduces context.Context as the first parameter. That is a Go convention. Functions that perform long-running or cancellable work accept a context so the caller can enforce deadlines or cancel the operation early. The select statement inside process checks ctx.Done() before proceeding. If the context expires, the worker stops immediately instead of wasting CPU cycles.
Error handling follows the standard pattern. if err != nil is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible. You send errors back through a dedicated channel. In a real system, you would wrap the error with fmt.Errorf("job %s failed: %w", j.ID, err) to preserve the stack trace. The receiver name in a method would be one or two letters matching the type, but here we use plain functions to keep the signature clean.
Context is plumbing. Run it through every long-lived call site.
Where things go wrong
Worker pools look simple until they break. The most common failure mode is a deadlock. If you create an unbuffered channel and try to send to it without a receiver ready, the program halts. The runtime prints fatal error: all goroutines are asleep - deadlock! and exits. You avoid this by buffering the channel or ensuring a receiver is actively waiting.
Another trap is sending on a closed channel. Channels are meant to be closed by the sender, not the receiver. If a worker tries to send to a channel that main already closed, the program panics with panic: send on closed channel. You fix this by keeping channel ownership strict. The producer closes. The consumer reads. Never mix the two roles.
Goroutine leaks happen when a worker waits on a channel that never gets closed. The main program finishes, but the background goroutine stays alive, holding onto memory and file descriptors. You prevent leaks by passing a cancellable context and checking it inside the loop, or by ensuring the input channel closes exactly once. The compiler cannot catch these mistakes. They only show up at runtime.
The worst goroutine bug is the one that never logs.
When to reach for workers
Concurrency adds complexity. You only want it when it solves a real bottleneck.
Use a worker pool when you have independent tasks that can run in parallel and you need to limit concurrency to protect downstream services. Use a single goroutine plus a channel when one task feeds another in a pipeline and you want to decouple production from consumption. Use sequential code when you do not need concurrency. The simplest thing that works is usually the right thing. Use a dedicated job queue library when you need persistence, retries, and horizontal scaling across multiple machines.
Trust the scheduler. Measure before you optimize.