The problem with doing everything in the request cycle
You build an API endpoint that sends a welcome email. The user clicks a button, your server connects to an SMTP provider, waits for the network to respond, and finally returns a 200 OK. If the email provider takes three seconds, your HTTP handler blocks for three seconds. Ten users click at once, and your server runs out of goroutines. The solution is not to optimize the SMTP call. The solution is to hand the work off to a separate system that processes it later, while your API returns immediately.
What a distributed task queue actually does
A distributed task queue solves this by decoupling the request from the work. Think of it like a restaurant kitchen. The waiter takes the order, writes it on a ticket, drops it in the expander, and walks away to take the next table. The cooks pull tickets from the expander, prepare the food, and mark them done. The waiter never waits for the steak to cook. Asynq implements this pattern in Go. It uses Redis as the ticket expander. Redis handles the heavy lifting of job storage, priority sorting, and atomic state transitions through Lua scripts. Your Go code only writes the ticket and processes it.
The queue introduces durability. If your worker process crashes, the job stays in Redis. When the process restarts, it picks up where it left off. In-process goroutines vanish when the binary exits. Distributed queues survive restarts, deployments, and network partitions. They also give you visibility. You can inspect pending jobs, view failed attempts, and manually retry work without writing custom admin panels.
The minimal Asynq setup
Here is the simplest task definition and handler. The handler receives a context and a task wrapper, unmarshals the payload, does the work, and returns an error or nil.
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"github.com/hibiken/asynq"
)
// EmailTask holds the data needed to send a message
type EmailTask struct {
To string `json:"to"`
Subject string `json:"subject"`
}
// EmailHandler processes incoming email jobs
func EmailHandler(ctx context.Context, t *asynq.Task) error {
var payload EmailTask
// Decode the JSON payload into our struct
if err := json.Unmarshal(t.Payload(), &payload); err != nil {
return fmt.Errorf("invalid payload: %w", err)
}
// Check if the caller cancelled the job while we were working
if err := ctx.Err(); err != nil {
return fmt.Errorf("job cancelled: %w", err)
}
// Simulate network call to SMTP provider
log.Printf("sending email to %s with subject %s", payload.To, payload.Subject)
return nil
}
How the pieces connect at runtime
The handler signature follows Go conventions. The context arrives first, giving the worker a way to respect deadlines and cancellation signals. The *asynq.Task carries the type string and the raw byte payload. If the handler returns nil, Asynq marks the job as completed. If it returns an error, Asynq schedules a retry based on the configured backoff strategy. The payload is just JSON bytes. You control the schema. Asynq does not enforce a specific format, which means you can store simple strings, complex structs, or protobuf messages as long as your handler knows how to decode them.
When a job enters the system, it moves through distinct states. It starts as pending. The worker pulls it and marks it active. If the handler succeeds, it becomes completed. If the handler fails, it moves to retry. After exhausting retries, it lands in dead. Asynq tracks these transitions using Redis sorted sets and hashes. The library executes Lua scripts to guarantee that state changes happen atomically. This prevents race conditions where two workers accidentally claim the same job.
Convention aside: Go functions that accept a context always take it as the first parameter, conventionally named ctx. Asynq follows this rule strictly. Your handler receives context.Context first, which means you can chain deadlines, attach request tracing IDs, or respect graceful shutdown signals without fighting the library.
A realistic production pattern
A real application needs a worker process that listens for jobs and a client that pushes them. The worker runs as a long-lived process, often deployed alongside your API or as a separate service. The client lives in your request handlers.
Here is the worker setup. It registers the handler on a serve mux and starts processing with a fixed concurrency limit.
// StartWorker initializes and runs the background processor
func StartWorker(redisAddr string) error {
redisOpt := asynq.RedisClientOpt{Addr: redisAddr}
// Route task types to their corresponding handlers
mux := asynq.NewServeMux()
mux.HandleFunc("email:send", EmailHandler)
// Configure queue priorities and worker concurrency
srv := asynq.NewServer(
redisOpt,
asynq.Config{
Concurrency: 10,
Queues: map[string]int{
"default": 10,
"critical": 5,
},
},
)
// Run blocks until the server shuts down
return srv.Run(mux)
}
The client side lives in your HTTP handlers or CLI tools. It creates a task, attaches options like queue routing and retry limits, and pushes it to Redis.
// EnqueueEmail pushes a new email job to the queue
func EnqueueEmail(redisAddr string, to, subject string) error {
client := asynq.NewClient(asynq.RedisClientOpt{Addr: redisAddr})
defer client.Close()
// Serialize the payload into JSON bytes
payload, err := json.Marshal(EmailTask{To: to, Subject: subject})
if err != nil {
return fmt.Errorf("marshal failed: %w", err)
}
// Build the task with routing and retry rules
task := asynq.NewTask("email:send", payload,
asynq.Queue("default"),
asynq.MaxRetry(3),
)
// Push to Redis and return immediately
return client.Enqueue(task)
}
The Queues map in the server config controls how many goroutines pull from each named queue. The default queue gets ten workers. The critical queue gets five. This lets you isolate noisy workloads. If email sending slows down, it does not starve your payment processing jobs. Asynq also supports scheduled execution. You can attach asynq.ProcessIn(time.Duration) or asynq.ProcessAt(time.Time) to delay a job until a specific moment. The library uses Redis keys to track scheduled jobs and moves them to the pending set when the time arrives.
Where things break and how to catch them
Distributed queues introduce failure modes that in-process goroutines do not have. Redis is the single point of truth. If the Redis connection drops, the worker stops processing and the client cannot enqueue. Asynq handles reconnection automatically, but you should monitor the connection state. The compiler will catch missing imports with an undefined: asynq error, but runtime failures usually come from payload mismatches or unhandled context cancellation. If your handler ignores ctx.Err(), the job keeps running even after Asynq marks it for cancellation. That wastes CPU and blocks the concurrency slot. Always check the context early and often.
Another common trap is assuming jobs run in order. Asynq guarantees FIFO ordering within a single queue, but multiple workers pulling from the same queue will process jobs in parallel. If job B depends on job A finishing, you need to chain them explicitly or use a different pattern. The library also relies on Redis Lua scripts for atomic operations. If your Redis instance runs an older version that lacks Lua support, the server fails to start with a Lua scripting not supported panic. Stick to Redis 6.2 or newer.
Convention aside: Go error handling looks verbose here, but it is intentional. Returning fmt.Errorf("wrap: %w", err) preserves the error chain. Asynq logs the full chain on failure, which makes debugging production jobs much easier than swallowing errors. The if err != nil { return err } pattern is boilerplate by design. It forces you to acknowledge the unhappy path instead of hiding it behind a silent fallback.
The built-in monitoring tool, asynqmon, removes the need to write custom admin dashboards. It connects to the same Redis instance and reads the internal data structures. You can view pending jobs, inspect failed payloads, trigger manual retries, and pause queues during deployments. It runs as a separate binary or embeds in your app. The UI updates in real time because it polls Redis at a fixed interval.
Queues trade immediacy for reliability. Design your handlers to be idempotent, and let the queue handle the rest.
When to reach for Asynq
Use Asynq when you need durable job storage that survives process restarts. Use a simple in-process channel when the work only needs to live as long as the application does. Use a database-backed queue like River when you already run PostgreSQL and want to avoid managing a separate Redis cluster. Use a cloud provider's managed queue when you want to offload infrastructure maintenance entirely. Use plain sequential code when the task takes less than fifty milliseconds and does not block the request cycle.