When JSON gets too heavy
You're building a service that needs to talk to another service across the network. You start with JSON because it's easy to debug and every language supports it. Then the payload grows. You add nested objects, large arrays, and binary blobs. The network latency spikes. The client app complains about download size. You need something smaller and faster, but you still want type safety and a contract that both sides agree on. Protocol Buffers solve this.
Protocol Buffers, or Protobuf, is a way to define data structures in a language-neutral file called a .proto file. You write the schema once, and a tool generates the code for Go, Python, Java, or whatever else you need. Think of it like a standardized form at a government office. The form defines exactly what fields exist, what types they are, and which ones are required. You fill out the form, and the clerk processes it. In code, you define the message, and the generated code handles packing the data into bytes and unpacking it back. The bytes are compact. The structure is strict.
The schema is the source of truth
Here's the core loop: define a message in a .proto file, generate Go code, then marshal to bytes and unmarshal back.
// user.proto
syntax = "proto3";
package example;
// Message defines the structure. Fields get numbers, not names, for wire format.
message User {
string name = 1;
int32 age = 2;
}
You run the protoc compiler with the Go plugin to generate the source files. The command looks like this:
# Generate Go code from all proto files in the current directory.
# paths=source_relative keeps the output file structure matching the input.
protoc --go_out=. --go_opt=paths=source_relative *.proto
This creates a user.pb.go file containing the Go structs and methods. You import the generated package and use the standard proto library to serialize and deserialize.
// main.go
package main
import (
"fmt"
"log"
// Import the generated package. The path matches your module and proto package.
"example.com/myapp/gen"
"google.golang.org/protobuf/proto"
)
func main() {
// Create a message struct. Zero values are defaults, so empty fields are fine.
user := &gen.User{
Name: "Alice",
Age: 30,
}
// Marshal converts the struct to a compact byte slice for storage or transmission.
data, err := proto.Marshal(user)
if err != nil {
log.Fatal(err)
}
// Unmarshal reconstructs the struct from bytes. Pass a pointer to the destination.
var decoded gen.User
err = proto.Unmarshal(data, &decoded)
if err != nil {
log.Fatal(err)
}
fmt.Println(decoded.Name, decoded.Age)
}
The generated code is idiomatic Go. The structs use standard Go types. The methods follow Go naming conventions. You treat the generated types like any other Go types, except you don't edit them by hand.
Field numbers drive the wire format
The numbers in the proto file are not arbitrary. They are tags used in the binary encoding. The binary format stores tag: value pairs. The tag encodes the field number and the wire type. This means the field name doesn't matter on the wire. You can rename name to full_name in the proto, regenerate code, and old binaries will still decode it correctly because the number is the same. This is the secret to backward compatibility.
Field numbers are cheap for small values. Numbers 1 through 15 take only one byte in the wire format. Numbers 16 through 2047 take two bytes. The proto compiler reserves numbers 19000 through 19999 for internal use. You should start your field numbers at 1 and increment. Never reuse a field number for a different type. If you delete a field, reserve the number so you don't accidentally reuse it later and break compatibility.
Field numbers are the contract. Names are for humans. Numbers are for machines.
Versioning without breaking changes
Protobuf handles versioning gracefully. You can add new fields to a message without breaking old clients. Old clients ignore unknown fields during unmarshaling. New clients see default values for missing fields. This allows you to evolve your API over time.
If you need to remove a field, you must reserve the number. The reserved keyword prevents accidental reuse.
// user.proto
syntax = "proto3";
package example;
message User {
string name = 1;
// age was removed. Reserve the number to prevent reuse.
reserved 2;
string email = 3;
}
You can also use optional fields in proto3 to distinguish between a missing value and a zero value. By default, proto3 treats all fields as optional and uses zero values for missing data. If you need to know whether a field was explicitly set, mark it as optional. The generated code will include a presence check method.
Add fields freely. Delete with caution. Reserve the number.
Realistic usage: HTTP handler with context
Real code usually involves sending data over the network. Here's an HTTP handler that returns Protobuf. The handler respects context for cancellation and sets the correct content type.
// handler.go
package main
import (
"context"
"net/http"
"strconv"
"example.com/myapp/gen"
"google.golang.org/protobuf/proto"
)
// GetUserData returns a user message as application/x-protobuf.
func GetUserData(w http.ResponseWriter, r *http.Request) {
// Context flows through. Respect deadlines if the client cancels.
ctx := r.Context()
// Extract ID from query param.
id := r.URL.Query().Get("id")
if id == "" {
http.Error(w, "missing id", http.StatusBadRequest)
return
}
// Simulate fetching data. In real code, check ctx.Done() during long operations.
user := &gen.User{
Name: "Bob",
Age: 25,
}
// Marshal the response.
data, err := proto.Marshal(user)
if err != nil {
http.Error(w, "marshal error", http.StatusInternalServerError)
return
}
// Set content type so the client knows how to parse the body.
w.Header().Set("Content-Type", "application/x-protobuf")
w.Write(data)
}
The Content-Type header is critical. Clients need to know they are receiving binary Protobuf data, not JSON or HTML. The standard media type is application/x-protobuf. Some systems use application/octet-stream, but application/x-protobuf is more explicit.
Context is plumbing. Run it through every long-lived call site.
Handling choices with oneof and enums
Protobuf supports enums and oneof fields for structured choices. Enums define a fixed set of values. The first value is the default and must be zero. oneof allows exactly one field to be set within a group. The generated Go code uses an interface for oneof fields, which you type switch on.
// task.proto
syntax = "proto3";
package example;
// Enum defines a fixed set of values. The first value is the default (zero).
enum Status {
UNKNOWN = 0;
ACTIVE = 1;
ARCHIVED = 2;
}
// Oneof allows exactly one field to be set. The generated code uses a wrapper struct.
message Task {
string title = 1;
Status status = 2;
oneof payload {
string text = 3;
bytes data = 4;
}
}
The generated Go code provides a getter that returns an interface. You use a type switch to handle the different cases.
// task.go
package main
import (
"fmt"
"example.com/myapp/gen"
)
// processTask handles a task message and inspects the oneof payload.
func processTask(t *gen.Task) {
// Check the oneof field. The generated code provides a type switch or interface check.
switch p := t.GetPayload().(type) {
case *gen.Task_Text:
fmt.Println("Text payload:", p.Text)
case *gen.Task_Data:
fmt.Println("Binary payload length:", len(p.Data))
default:
fmt.Println("No payload set")
}
}
The oneof pattern is useful when a field can be one of several types, but never more than one at a time. It keeps the message compact and the API clear. The generated wrapper structs like Task_Text and Task_Data ensure type safety.
Pitfalls and compiler errors
Protobuf has a few gotchas. If you pass the wrong type to proto.Unmarshal, the function returns an error. The compiler won't catch this at compile time because the generated types are just structs. You get a runtime error like proto: message gen.User is not a pointer if you forget to pass a pointer.
If you try to unmarshal invalid data, you get proto: cannot parse invalid wire-format data. This happens when the byte slice is corrupted or comes from a different schema. Always validate your data sources.
If you forget to generate the code or import the wrong package, the compiler rejects the build with undefined: gen. Make sure your protoc command runs successfully and your go.mod includes the generated package path.
Convention aside: the protoc plugin for Go is installed via go install google.golang.org/protobuf/cmd/protoc-gen-go@latest. Most teams use a tool like buf to manage proto files and generation, but the raw protoc command works fine for small projects.
Protobuf is a contract. Break the contract, and the wire breaks.
Decision: Protobuf vs alternatives
Use Protocol Buffers when you need compact binary encoding for high-throughput services or mobile clients where bandwidth matters. Use Protocol Buffers when you want strict schema enforcement and automatic code generation across multiple languages. Use JSON when you need human-readable payloads for debugging, configuration files, or public APIs where flexibility outweighs size. Use Go's encoding/binary when you are writing a custom binary protocol and don't need the overhead of a schema definition. Use encoding/gob when you are serializing Go-specific data structures for internal caching and don't care about cross-language compatibility.