You need a buffer that survives crashes
You are building a signup flow. A user clicks a button, and your service needs to send a welcome email, update an analytics dashboard, and trigger a onboarding sequence. If you call these services sequentially, the signup request hangs for seconds while each downstream call completes. If you fire off goroutines for each task, you risk dropping messages when the server restarts or the process crashes. You need a buffer that persists data, scales across multiple instances, and guarantees delivery even when consumers are slow.
Kafka solves this problem. It is a distributed commit log that stores streams of records. Producers write to topics, and consumers read from them. The log survives restarts, and multiple consumers can share the load through consumer groups. In Go, you have two primary clients. segmentio/kafka-go is a pure-Go implementation. It is lightweight, idiomatic, and easy to debug. confluent-kafka-go is a wrapper around librdkafka, the official C library. It offers maximum performance and advanced features like exactly-once semantics, but it requires CGO and complicates cross-compilation.
Kafka is a log. Treat it like one.
How Kafka clients work in Go
Kafka organizes data into topics. Each topic splits into partitions, which are ordered, immutable sequences of records. Every record has an offset, a unique position within the partition. Producers append records to partitions. Consumers read records by offset and commit their progress so they can resume after a restart.
The Go client handles the protocol details. It manages connections to brokers, discovers partition leaders, serializes messages, and handles retries. You configure the client with broker addresses, topic names, and consumer group IDs. The client then exposes a simple API to produce and consume messages.
segmentio/kafka-go uses standard Go patterns. It returns structs, accepts interfaces, and integrates with context.Context. confluent-kafka-go mirrors the C library API. It uses configuration maps and event loops. Both clients are production-ready, but they serve different needs.
Run gofmt on every file. The community expects consistent formatting. Most editors run it automatically on save.
Minimal producer example
Here is the simplest producer using the pure-Go client. It connects to a broker, writes a single message, and closes the connection.
package main
import (
"context"
"log"
"time"
"github.com/segmentio/kafka-go"
)
func main() {
// Writer handles the network connection and serialization.
w := &kafka.Writer{
Addr: kafka.TCP("localhost:9092"),
Topic: "orders",
Balancer: &kafka.Hash{}, // Hash balancer routes messages by key to preserve order.
}
// Message requires key and value as byte slices.
msg := kafka.Message{
Key: []byte("order-123"),
Value: []byte(`{"item": "laptop", "qty": 1}`),
Time: time.Now(),
}
// WriteMessages blocks until the broker acknowledges or context cancels.
err := w.WriteMessages(context.Background(), msg)
if err != nil {
log.Fatal(err)
}
// Close flushes buffers and releases the connection.
w.Close()
}
The Writer struct holds the configuration. Addr points to the broker. Topic specifies where messages go. The Balancer decides which partition receives each message. The Hash balancer computes a hash of the key and maps it to a partition. This ensures all messages for the same key land in the same partition, preserving order for that key.
WriteMessages takes a context.Context as the first argument. This follows Go convention: functions that might block should accept a context to support cancellation and deadlines. The context lets you timeout the write if the broker is unresponsive.
The Message struct requires Key and Value as []byte. If you pass a string directly, the compiler rejects this with cannot use "value" (untyped string constant) as []byte value in struct literal. You must convert explicitly using []byte("value").
Always close the writer. The client buffers messages to improve throughput. If you exit without closing, buffered messages may never send.
Close your writers. Commit your offsets. Respect the context.
Realistic consumer with graceful shutdown
Real consumers run continuously. They join a consumer group so multiple instances can share the load. When you add or remove instances, Kafka rebalances partitions among the group members. Here is a robust consumer loop that handles shutdown signals and respects context cancellation.
// Reader configures the consumer. GroupID enables load balancing across instances.
r := kafka.NewReader(kafka.ReaderConfig{
Brokers: []string{"localhost:9092"},
Topic: "orders",
GroupID: "order-processor",
MinBytes: 10e3, // Wait for a batch of 10KB to reduce network overhead.
MaxBytes: 10e6, // Cap the batch size to 10MB to prevent memory spikes.
})
// Graceful shutdown on SIGINT or SIGTERM.
ctx, cancel := context.WithCancel(context.Background())
go func() {
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
<-sig
cancel()
}()
The ReaderConfig sets up the consumer. GroupID is critical. It tells Kafka which group this consumer belongs to. Kafka assigns partitions to group members so each message is processed exactly once by the group. If you omit GroupID, the reader acts as a standalone consumer and reads from the beginning or last committed offset.
MinBytes and MaxBytes control batching. The reader waits until it accumulates at least MinBytes before returning a batch. This reduces network round-trips and improves throughput. MaxBytes caps the batch size to prevent memory issues. Tune these values based on your message size and latency requirements.
The signal handler listens for SIGINT and SIGTERM. When the process receives a shutdown signal, it cancels the context. This triggers the consumer loop to exit cleanly.
// Loop reads messages until context is cancelled or an error occurs.
for {
msg, err := r.ReadMessage(ctx)
if err != nil {
// ReadMessage returns context.Canceled when shutdown triggers.
log.Printf("consumer stopped: %v", err)
break
}
// Process the message. Commit happens automatically after successful read.
log.Printf("partition=%d offset=%d key=%s value=%s",
msg.Partition, msg.Offset, string(msg.Key), string(msg.Value))
}
// Close releases resources and commits any pending offsets.
r.Close()
The ReadMessage call blocks until a message is available or the context cancels. It returns the message and an error. If the context cancels, the error is context.Canceled. The loop checks the error and breaks out on cancellation.
The reader commits offsets automatically after each successful read. This means if the process crashes, it may reprocess the last message. This is at-least-once delivery. If you need exactly-once semantics, you must use transactions, which segmentio/kafka-go supports but requires careful handling.
The if err != nil check is verbose by design. The Go community accepts the boilerplate because it makes error handling explicit. Never ignore errors in production code. If you must discard a return value, use _ to signal intent. For example, result, _ := someFunc() says you considered the second return and chose to drop it. Use this sparingly with errors.
The worst Kafka bug is the one that silently drops messages.
Pitfalls and common errors
Type mismatches are the most common compile-time error. Kafka messages use byte slices for keys and values. If you pass a string, the compiler rejects this with cannot use "value" (untyped string constant) as []byte value in struct literal. Convert explicitly. If you forget to import the package, the compiler complains with undefined: kafka. If you import but don't use it, you get imported and not used.
Resource leaks are a runtime risk. If you create a writer or reader and forget to close it, the connection stays open. The client may buffer messages indefinitely. Always defer close or handle it in a finally block. Goroutine leaks are equally dangerous. If you spawn a goroutine to process a message and that goroutine blocks, the consumer loop stalls. The reader stops pulling messages, and lag grows until the system chokes. Ensure every goroutine has a cancellation path.
confluent-kafka-go introduces CGO complexity. CGO requires a C compiler and links against librdkafka. This complicates cross-compilation and static linking. If you build for a different architecture, you need the cross-compilation toolchain installed. segmentio/kafka-go avoids this entirely. It compiles to a single binary with no external dependencies.
Memory usage differs between clients. segmentio/kafka-go allocates Go objects for messages. High-throughput workloads may generate more garbage collection pressure. confluent-kafka-go uses the C library's memory management, which can be more efficient for massive volumes. Profile your application to see which client fits your constraints.
The compiler catches syntax errors. It does not catch logic errors. Test your consumer with realistic message volumes and failure scenarios. Verify that offsets commit correctly and that rebalancing works as expected.
Don't fight the type system. Wrap the value or change the design.
Decision matrix
Use segmentio/kafka-go when you want a pure-Go client that integrates smoothly with the standard library and supports cross-compilation without CGO. Use segmentio/kafka-go when your workload is moderate and you value simplicity, idiomatic APIs, and easy debugging. Use confluent-kafka-go when you need maximum throughput and low latency for high-volume ingestion pipelines. Use confluent-kafka-go when you require advanced features like exactly-once semantics, transactional producers, or schema registry integration. Use a simpler queue like Redis Streams or NATS when Kafka's operational complexity outweighs the benefits for your use case.
Pick the tool that matches your volume. Don't over-engineer.