How to Use Machinery for Distributed Task Processing in Go

Async
2026-04-17T16:39:40+00:00
Machinery is a robust asynchronous task queue and job scheduler for Go that allows you to distribute work across multiple workers using a message broker like Redis or RabbitMQ.

Machinery is a robust asynchronous task queue and job scheduler for Go that allows you to distribute work across multiple workers using a message broker like Redis or RabbitMQ. You define tasks as standard Go functions, register them with the server, and then send jobs to the broker where workers pick them up and execute them concurrently.

To get started, install the library and set up a Redis connection, then define your worker functions and register them with the server. Here is a minimal example of a worker that processes a simple addition task:

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/RichardKnop/machinery/v2"
	"github.com/RichardKnop/machinery/v2/config"
	"github.com/RichardKnop/machinery/v2/tasks"
)

// Define a task function. It must return []interface{} and error.
func Add(x, y int) (int, error) {
	return x + y, nil
}

func main() {
	// Configure the broker (Redis in this case)
	cfg := &config.Config{
		Broker:        "redis://localhost:6379",
		Backend:       "redis://localhost:6379",
		DefaultQueue:  "machinery_queue",
		ResultsExpireTime: 3600,
	}

	// Create the server
	server, err := machinery.NewServer(cfg)
	if err != nil {
		log.Fatalf("Failed to create server: %v", err)
	}

	// Register the task
	server.RegisterTask("add", Add)

	// Start the worker
	worker := server.NewWorker("add_worker", 2) // 2 concurrent processes
	if err := worker.Launch(); err != nil {
		log.Fatalf("Failed to launch worker: %v", err)
	}

	// Keep the process running
	select {}
}

To send a job to the queue from a separate producer application, you use the same server configuration but call NewTask and Send:

// In your producer code
server, _ := machinery.NewServer(cfg)
task := server.NewTask("add", 10, 20)
signature := server.NewSignature(task, "machinery_queue", nil)
_, err := server.SendTask(signature)
if err != nil {
	log.Fatalf("Failed to send task: %v", err)
}

The key to using Machinery effectively is ensuring your task functions are pure (no global state) and that your broker is running. You can scale horizontally by running multiple worker processes on different machines pointing to the same Redis/RabbitMQ instance; Machinery handles the distribution automatically. For production, always set up proper error handling in your tasks and consider using server.NewSignature options to configure retries, delays, and timeouts for critical jobs.