How to Use Asynq for Distributed Task Queues in Go

Asynq is a robust Go library for building distributed task queues that relies on Redis as its backend to handle job scheduling, retries, and concurrency. You initialize a server with a Redis connection and a task processor, then push tasks to specific queues where workers pick them up based on priority and concurrency limits.

Here is a minimal setup to get a worker running and a client pushing jobs:

1. Define your task and worker logic First, define a task type and a handler function. The handler receives the task payload and returns an error; if it returns nil, the job is marked as completed.

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"log"

	"github.com/hibiken/asynq"
)

// Define a custom task type
type EmailTask struct {
	To      string
	Subject string
}

// TaskHandler processes the job
func EmailHandler(ctx context.Context, t *asynq.Task) error {
	var payload EmailTask
	if err := json.Unmarshal(t.Payload(), &payload); err != nil {
		return fmt.Errorf("failed to unmarshal payload: %w", err)
	}

	// Simulate sending email
	log.Printf("Sending email to %s: %s", payload.To, payload.Subject)
	return nil
}

2. Run the worker and client Start the worker to listen for tasks and use the client to enqueue them.

func main() {
	// Initialize Redis connection
	redisOpt := asynq.RedisClientOpt{
		Addr: "localhost:6379",
	}

	// Start the worker
	mux := asynq.NewServeMux()
	mux.HandleFunc("email:send", EmailHandler)

	srv := asynq.NewServer(
		redisOpt,
		asynq.Config{
			Concurrency: 10,
			Queues: map[string]int{
				"default": 10,
				"critical": 5,
			},
		},
	)

	go func() {
		if err := srv.Run(mux); err != nil {
			log.Fatalf("failed to run server: %v", err)
		}
	}()

	// Create a client to push tasks
	client := asynq.NewClient(redisOpt)
	defer client.Close()

	// Prepare payload
	payload, _ := json.Marshal(EmailTask{To: "user@example.com", Subject: "Hello"})

	// Enqueue the task
	task := asynq.NewTask("email:send", payload,
		asynq.Queue("default"),
		asynq.MaxRetry(3), // Retry up to 3 times on failure
	)

	if err := client.Enqueue(task); err != nil {
		log.Fatalf("failed to enqueue task: %v", err)
	}

	log.Println("Task enqueued successfully")
}

Key features to leverage immediately include asynq.MaxRetry for automatic failure handling and asynq.Queue to route high-priority jobs to a separate queue with dedicated concurrency. Asynq also provides a built-in web UI (asynqmon) to inspect job states, view errors, and manage retries without writing custom admin tools. Ensure your Redis instance is stable, as Asynq relies heavily on Redis Lua scripts for atomic job operations and scheduling.

Asynq is a tool that helps your Go applications handle heavy work in the background without slowing down the main program. It works like a digital to-do list where tasks are added to a queue and processed one by one by a separate worker. You cannot find this tool inside the standard Go files you provided; it is an external library you must download and add to your project.