When concurrency eats your resources
You are building a scraper. You have a list of 10,000 URLs. You write a loop, spawn a goroutine for each URL, and let them fly. Your program crashes with an out-of-memory error, or the target server blocks your IP address because you sent 10,000 requests in one second. You need a throttle. You need a way to say "run at most 10 goroutines at the same time."
A semaphore is the tool for this job. It is a concurrency primitive that limits the number of goroutines that can enter a critical section simultaneously. Unlike a mutex, which allows only one goroutine at a time, a semaphore allows N goroutines. When the limit is reached, new goroutines wait until a slot opens up.
Go's standard library does not include a semaphore. The community standard is the golang.org/x/sync/semaphore package. It provides a weighted semaphore that handles blocking, context cancellation, and variable resource costs.
The concept in plain words
Think of a nightclub with a strict capacity limit. The bouncer stands at the door with a counter. The club holds 50 people. When someone arrives, the bouncer checks the counter. If fewer than 50 people are inside, the bouncer lets them in and increments the counter. If the club is full, the person waits in line. When someone leaves, they tell the bouncer, the counter decrements, and the next person in line gets in.
In Go, the "club" is your resource limit. The "people" are goroutines. The "counter" is the semaphore.
The semaphore in x/sync is weighted. This means different tasks can consume different amounts of capacity. A normal request might take one slot. A heavy batch operation might take five slots. The semaphore tracks the total weight of active tasks. If the limit is 10, and two tasks of weight 5 are running, no new tasks can start, even if only two goroutines are active.
Minimal example
Here is the skeleton of a weighted semaphore. You create it with a capacity, acquire a slot before doing work, and release the slot when finished.
package main
import (
"context"
"fmt"
"time"
"golang.org/x/sync/semaphore"
)
func main() {
// Capacity of 3 means at most 3 slots can be held at once.
sem := semaphore.NewWeighted(3)
// Simulate 5 tasks trying to run concurrently.
for i := 0; i < 5; i++ {
go func(id int) {
// Acquire blocks until a slot is available or context is canceled.
// Weight of 1 means this task consumes one slot.
if err := sem.Acquire(context.Background(), 1); err != nil {
fmt.Printf("Task %d canceled\n", id)
return
}
// Release returns the slot to the semaphore.
// Defer ensures release happens even if the task panics.
defer sem.Release(1)
fmt.Printf("Task %d running\n", id)
// Simulate work.
time.Sleep(1 * time.Second)
}(i)
}
// Wait for goroutines to finish.
time.Sleep(2 * time.Second)
}
Walkthrough of the runtime behavior
When sem.Acquire is called, the semaphore checks its internal counter. If the current weight is less than the capacity, it increments the counter and returns immediately. The goroutine proceeds into the critical section.
If the counter is already at capacity, Acquire blocks. The goroutine parks itself and waits for a signal. It does not spin-loop and waste CPU cycles. It sleeps until another goroutine calls Release.
When Release is called, the semaphore decrements the counter by the specified weight. If there are goroutines waiting in the queue, the semaphore wakes one up (or several, if the released weight is large enough) and allows them to proceed.
The context.Context argument is the kill switch. If the context is canceled while a goroutine is waiting in Acquire, the function returns immediately with an error. The goroutine does not acquire a slot. This prevents goroutine leaks when a parent operation is aborted.
Convention aside: context.Context is always the first parameter in Go functions that perform long-running operations. Functions that take a context should respect cancellation and deadlines. Passing context to Acquire follows this rule.
Realistic example with weights and errors
In production code, you often wrap the semaphore in a struct. You might have different types of jobs that consume different resources. A database query might take one slot. A complex report generation might take three slots.
Here is a worker struct that manages a semaphore. It shows how to handle errors from Acquire and how to use weights.
package main
import (
"context"
"fmt"
"sync"
"time"
"golang.org/x/sync/semaphore"
)
// Worker manages concurrent tasks with a resource limit.
type Worker struct {
sem *semaphore.Weighted
}
// NewWorker creates a worker with the given concurrency limit.
func NewWorker(limit int64) *Worker {
return &Worker{
// Initialize the semaphore with the max weight.
sem: semaphore.NewWeighted(limit),
}
}
// Run executes a task with the given weight.
// If the context is canceled, it returns early.
func (w *Worker) Run(ctx context.Context, id int, weight int64) error {
// Acquire blocks until weight slots are available.
// Returns error if ctx is canceled.
if err := w.sem.Acquire(ctx, weight); err != nil {
return fmt.Errorf("task %d: %w", id, err)
}
// Release must be deferred to prevent leaks.
// If the function panics, the slot is still returned.
defer w.sem.Release(weight)
fmt.Printf("Task %d (weight %d) running\n", id, weight)
time.Sleep(500 * time.Millisecond)
return nil
}
func main() {
w := NewWorker(5)
var wg sync.WaitGroup
// Launch 10 tasks. Some are heavy (weight 3), some are light (weight 1).
for i := 0; i < 10; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
// Heavy tasks take more capacity.
weight := int64(1)
if id%3 == 0 {
weight = 3
}
// Handle error from Acquire.
if err := w.Run(context.Background(), id, weight); err != nil {
fmt.Printf("Task %d failed: %v\n", id, err)
}
}(i)
}
wg.Wait()
}
The receiver name is usually one or two letters matching the type: (w *Worker), not (this *Worker). This keeps method signatures clean.
The channel alternative
Go developers often implement a semaphore using a buffered channel. This is the "classic" Go way before x/sync/semaphore existed.
// Channel-based semaphore pattern.
// Capacity of 10 means 10 tokens are available.
ch := make(chan struct{}, 10)
// To acquire a slot, send a token.
// Blocks if the channel is full.
ch <- struct{}{}
// To release a slot, receive a token.
// Unblocks a waiting sender if any.
<-ch
This works for simple counting. It is lightweight and uses only the standard library. However, it has limitations. You cannot assign weights. A channel token is always one unit. You cannot pass a context to ch <- struct{}{}. If you need to cancel a waiting goroutine, you have to use a separate select statement with a ticker or a done channel, which adds boilerplate.
The x/sync/semaphore package handles weights and context cancellation internally. It is more expressive and less error-prone for complex scenarios.
Pitfalls and compiler errors
Forgetting to release a slot is the most common bug. If a goroutine acquires a slot and then returns early without releasing, the semaphore counter stays high. Eventually, the semaphore fills up, and all new goroutines block forever. This is a goroutine leak. Always use defer sem.Release(weight) immediately after a successful Acquire.
Panicking inside the critical section is safe if you use defer. The deferred release runs during the panic unwind, returning the slot. If you do not use defer, the panic terminates the goroutine, and the slot is lost.
The compiler enforces type safety on weights. The Acquire and Release methods expect int64 for the weight. If you pass an int literal without casting, the compiler rejects the program.
The compiler complains with
cannot use 1 (untyped int constant) as int64 value in argumentif you pass the wrong type.
You can fix this by using an int64 literal or a cast. sem.Acquire(ctx, int64(1)) works. sem.Acquire(ctx, 1) also works because untyped constants are flexible, but explicit types are clearer in larger expressions.
Context cancellation is not automatic. If you pass context.Background(), the Acquire call will wait forever until a slot opens. If you pass a context with a deadline, Acquire returns an error when the deadline passes. The goroutine must check this error and handle it. Ignoring the error and proceeding assumes you have a slot, which is false. The program will likely deadlock or corrupt state.
Decision matrix
Use a semaphore when you need to limit concurrency to a specific number but want to keep the goroutine lifecycle simple. Use a buffered channel when you want to implement a worker pool with a fixed number of workers and no weights. Use a mutex when you need strict mutual exclusion for a shared variable. Use a plain loop when you don't need concurrency: the simplest thing that works is usually the right thing.
Use golang.org/x/sync/semaphore when you need weighted limits or context cancellation. Use sync.Mutex when you only need one goroutine at a time. Use a worker pool pattern when you want to reuse a fixed set of goroutines instead of spawning new ones.
Goroutines are cheap. Semaphores are not magic. They are a counter with a queue. Trust the counter. Always release what you acquire.