--- title: How to Implement Idempotent Job Processing in Go summary: Prevent duplicate job execution in Go by checking a persistent store for the job ID before running the task logic. lang: en slug: how-to-implement-idempotent-job-processing-in-go published: 2026-04-17T12:05:42+00:00 updated: 2026-04-25T22:00:13+00:00 canonical: https://www.gofaq.org/en/how-to-implement-idempotent-job-processing-in-go/ --- # How to Implement Idempotent Job Processing in Go ## The retry that broke the bank You send a payment request to your backend. The network hiccups. The client doesn't get a response. The client retries. Your server receives the same request twice. Without protection, you charge the user twice. The user complains. The support ticket arrives. You realize the job processor isn't idempotent. Idempotency means doing the same thing multiple times has the same effect as doing it once. In job processing, this usually means tracking job IDs and refusing duplicates. If a job ID comes in, you process it. If that same ID comes in again, you return the previous result or skip it. The state of the world doesn't change on the second attempt. This protects against network retries, duplicate messages from a queue, or a crashed worker that the scheduler thinks failed. Idempotency is the shield between your logic and the chaos of distributed systems. ## What idempotency actually means Think of idempotency like a light switch. You can flip the switch on ten times in a row. The result is always the same: the light is on. You don't get brighter light or a second bulb. Job processing needs that property. In Go, idempotency usually relies on an idempotency key. This is a unique identifier attached to the request. The system stores this key. When a request arrives, the system checks the key. If the key exists, the system knows the work is done. It returns the stored result or skips execution. If the key is new, the system stores it and runs the work. The key must be unique per logical operation. A payment ID works. A user ID plus a timestamp might not, if the user can make multiple payments. The key represents the intent, not the attempt. Idempotency is a contract. Enforce it at the boundary, not in the middle. ## In-memory guard with sync.Map Here's the simplest in-memory guard: use a `sync.Map` to track IDs and reject duplicates instantly. ```go package main import ( "fmt" "sync" ) // jobTracker holds completed job IDs to prevent re-processing within this process. var jobTracker sync.Map // ProcessJob runs the function only if the ID hasn't been seen. func ProcessJob(jobID string, fn func()) { // LoadOrStore is atomic. It checks and sets in one step. // It returns the existing value if present, or stores the new one. // The loaded bool tells us if the value was already there. // struct{}{} is an empty struct with zero size. It saves memory compared to storing a bool. if _, loaded := jobTracker.LoadOrStore(jobID, struct{}{}); loaded { fmt.Println("Skipping duplicate job:", jobID) return } // First time seeing this ID. Run the work. fn() } ``` `sync.Map` is a concurrent-safe map optimized for cases where keys are written once and read many times, or where different goroutines write to distinct keys. `LoadOrStore` is atomic. It checks and sets in one step. If the key exists, it returns the value and `true`. If not, it stores the value and returns the zero value and `false`. The `struct{}{}` is an empty struct. It takes zero bytes of memory. Using it as the value saves space compared to storing a boolean or a pointer. With millions of job IDs, that memory savings adds up. The `loaded` variable captures the boolean return. If `loaded` is true, the job was already in the map. You skip execution. If false, you stored the ID and proceed. This pattern is fast and safe for a single process. It vanishes when the process restarts. If the server crashes and restarts, the map is empty. Old job IDs are forgotten. New retries will run again. This is acceptable for transient data or when the cost of re-running is low. Goroutines are cheap. Channels are not magic. `sync.Map` is a tool, not a silver bullet. ## How sync.Map beats a mutex map Developers often reach for a plain `map` wrapped in a `sync.Mutex`. That works. `sync.Map` is different. It uses internal sharding and read-write locks to allow concurrent reads without blocking. When keys are distinct per goroutine, `sync.Map` avoids contention entirely. Each goroutine writes to its own shard. If many goroutines write to the same keys, `sync.Map` falls back to a global lock. In that case, a `map` with a `sync.Mutex` might be faster. Profile your workload. The Go documentation warns about this trade-off. `sync.Map` shines when you have high read concurrency or distinct writes. Job tracking often fits this pattern. Each job ID is unique. Workers process different IDs. Reads happen to check existence. Writes happen to record completion. Run `gofmt` on this code. It aligns the struct fields and fixes indentation. Don't argue about braces. The tool decides. Most editors run it on save. Trust `gofmt`. Argue logic, not formatting. ## Durable tracking with a database In production, jobs survive restarts. You need a durable store. Here's a pattern using a database to track completion across process boundaries. ```go package main import ( "context" "database/sql" "fmt" ) // JobProcessor manages durable idempotency using a database. type JobProcessor struct { db *sql.DB } // NewJobProcessor creates a processor bound to a database connection. func NewJobProcessor(db *sql.DB) *JobProcessor { return &JobProcessor{db: db} } // Process attempts to insert the job ID. If the unique constraint fails, the job is skipped. func (p *JobProcessor) Process(ctx context.Context, jobID string, fn func(context.Context) error) error { // Context is always the first parameter. It carries cancellation and deadlines. // Try to insert the job ID. The unique index on job_id prevents duplicates. // ON CONFLICT DO NOTHING ensures the insert fails silently if the ID exists. result, err := p.db.ExecContext(ctx, ` INSERT INTO processed_jobs (job_id) VALUES ($1) ON CONFLICT (job_id) DO NOTHING `, jobID) if err != nil { return fmt.Errorf("insert job tracking: %w", err) } // RowsAffected returns 1 if inserted, 0 if conflict. rows, err := result.RowsAffected() if err != nil { return fmt.Errorf("check rows affected: %w", err) } if rows == 0 { // ID already exists. Job was processed before. return nil } // New ID. Execute the work. if err := fn(ctx); err != nil { // If the job fails, you might want to delete the tracking row // to allow retries, or mark it as failed depending on your retry policy. return fmt.Errorf("execute job %s: %w", jobID, err) } return nil } ``` The database unique constraint is the safety net. Without it, the check-then-act pattern breaks under concurrency. Two goroutines can check for the ID simultaneously, both see it's missing, and both insert. The unique constraint forces the database to reject the second insert. The application handles the rejection. The receiver name is usually one or two letters matching the type: `(p *JobProcessor)`, not `(this *JobProcessor)`. This is Go convention. Keep it short. `if err != nil { return err }` is verbose by design. The community accepts the boilerplate because it makes the unhappy path visible. Check the error immediately. Return it. Don't hide it. The `Process` function takes `ctx` first. This is the Go convention. Pass it down. Respect cancellation. Functions that take a context should respect deadlines and cancellation signals. The database unique constraint is the only truth. Application logic is a suggestion. ## The zombie job problem The simple insert pattern has a flaw. If the process inserts the ID and then crashes before running the work, the ID is stuck in the database. The work never happened. The queue might retry. The retry sees the ID and skips. The job is dead. This is the zombie job problem. The fix depends on your tolerance for duplicates versus lost jobs. One approach is to insert with a `status='pending'`. If the work succeeds, update to `status='done'`. If the work fails, delete the row or mark it `failed`. On retry, check the status. If `pending`, you can re-run the work. If `done`, skip. If `failed`, decide whether to retry or alert. This adds complexity. You need transactions. You need to handle the gap between insert and update. You might need a background cleanup job to expire old `pending` rows. Idempotency is easy when the work is instant. It gets hard when the work takes time and crashes happen. Another approach is to accept the risk. If the job is cheap to re-run, let it run twice. The idempotency key prevents infinite loops. Two runs are better than zero runs. If the job is expensive or destructive, you need the status tracking. Track the ID before the work. Clean up the ID after the work. Manage the gap carefully. ## Memory leaks and eviction `sync.Map` grows forever. If you process millions of jobs, the map consumes memory. You need eviction. `sync.Map` doesn't have built-in TTL. You need a background goroutine to clean up old IDs. One pattern is to store a timestamp instead of `struct{}{}`. Use `LoadOrStore` to set the timestamp. A background goroutine iterates the map and deletes old entries. `sync.Map` supports `Range` and `Delete`. You can sweep the map periodically. Alternatively, use a cache library like `ristretto` or `bigcache`. These provide TTL and size limits out of the box. They are optimized for high throughput. If you need eviction, don't roll your own. Use a library. The worst goroutine bug is the one that never logs. If your cleanup goroutine panics, your memory leaks silently. Add recovery and logging to background workers. ## Pitfalls and compiler errors Race conditions are the enemy. If you check for existence and then insert, two goroutines can pass the check simultaneously. The database unique constraint prevents this. Without it, you get duplicates. The compiler won't catch this. You'll get data corruption in production. If you use `sync.Map` in a distributed system, you get false negatives. Each node has its own map. Node A processes ID 123. Node B doesn't know. Node B processes ID 123. The job runs twice. Use a shared store for distributed systems. If you forget to capture the loop variable, the compiler rejects the program with `loop variable i captured by func literal` (which became a hard error in Go 1.22+). This applies if you spawn goroutines in a loop. Always capture the variable. If you try to use `sync.Map` with non-pointer types incorrectly, you might get data races. The race detector catches this. Run `go run -race` during development. It flags concurrent access to shared memory. Public names start with a capital letter. Private start lowercase. No keywords like `public` or `private`. `JobProcessor` is public. `db` is private. This controls visibility. Interfaces are accepted, structs are returned. "Accept interfaces, return structs" is the most common Go style mantra. Your processor returns errors and results, not interfaces. Idempotency keys are cheap. Duplicates are expensive. ## When to use what Use `sync.Map` when you need fast in-memory deduplication within a single process and losing state on restart is acceptable. Use a database with a unique constraint when jobs must survive restarts and multiple workers share the same job queue. Use a distributed cache like Redis with `SET NX` when you need cross-process deduplication with lower latency than a relational database. Use a message queue with exactly-once semantics if your broker supports it and you want to offload idempotency to the infrastructure. Use a local mutex-protected map when keys are written frequently by multiple goroutines, as `sync.Map` optimizes for read-heavy or distinct-key workloads. Use a cache with TTL when you need automatic eviction of old job IDs to prevent memory leaks. Use a status-tracking pattern when jobs are expensive and you cannot afford zombie jobs that block retries. Use plain sequential code when you don't need concurrency: the simplest thing that works is usually the right thing. ## Where to go next - [How to Use Machinery for Distributed Task Processing in Go](/en/how-to-use-machinery-for-distributed-task-processing-in-go/) - [How to Set a Timeout with time.After and context](/en/how-to-set-a-timeout-with-timeafter-and-context/) - [How to Implement an Actor Model in Go](/en/how-to-implement-an-actor-model-in-go/)