The problem with sending raw JSON over the wire
You build a service that tracks inventory. Another service needs to read those counts. You settle on JSON because it is easy to debug and every language understands it. Six months later, you rename a field from item_count to quantity. The downstream service crashes because it still expects the old key. You add a new field, but the old client ignores it. You remove a field, and the old client panics. You start writing validation middleware, custom version headers, and manual serialization helpers. The network payload is bloated with repeated field names. You are spending more time maintaining the contract than building features.
Protocol Buffers solve this by making the contract explicit, compiled, and version-safe. You define the shape of your data once. The compiler generates the serialization code. Both sides agree on the structure before a single byte crosses the network.
What protobuf actually does
Protocol Buffers is a schema-first serialization format. You write a .proto file that describes messages, fields, and their types. The protoc compiler reads that file and generates language-specific code. In Go, that means structs, marshaling functions, and interface implementations that handle the binary encoding.
Think of it like a stamped shipping manifest. The manifest defines exactly what goes into the box, where each item sits, and how to unpack it later. The box itself contains only the items and minimal labels. No repeated field names. No human-readable overhead. Just compact, predictable bytes.
The generated Go code implements the proto.Message interface. That interface is small: it requires Reset(), String(), and ProtoReflect(). The compiler fills in the heavy lifting. You interact with the structs like normal Go types, then call proto.Marshal or proto.Unmarshal when you need to send or receive data.
Protobuf handles versioning through field numbers, not field names. Each field gets a unique integer tag. The wire format encodes tag: value pairs. If you add a new field, old clients simply ignore the unknown tag. If you remove a field, old clients drop it on the floor. The contract stays stable as long as you never reuse a deleted field number.
Protobuf is a contract compiler. Treat the .proto file as the source of truth, not the generated Go code.
The minimal setup
You need the protoc compiler and the Go plugin. The plugin translates the schema into idiomatic Go. Install the plugin through the module proxy, then run the compiler with the correct output flags.
# Install the official Go code generator
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
# Generate Go code from all .proto files in the current directory
# paths=source_relative keeps the output next to the .proto file
protoc --go_out=. --go_opt=paths=source_relative *.proto
This produces .pb.go files. Import them like any other package. The generated code lives in the same directory as the schema, which keeps your module layout clean.
Run gofmt on the generated files if your editor does not do it automatically. The protobuf generator follows Go formatting conventions, but some older plugin versions leave trailing whitespace. Trust the toolchain.
How the generated code works
The compiler turns each message definition into a Go struct. Every field becomes a struct field with a zero value. The compiler also adds methods for resetting the struct, printing it, and reflecting over its fields. You rarely call those methods directly. You use proto.Marshal and proto.Unmarshal.
package main
import (
"fmt"
"log"
"google.golang.org/protobuf/proto"
"yourmodule/gen" // import path to your generated .pb.go directory
)
func main() {
// Create a message instance and populate fields
msg := &gen.Item{
Id: 42,
Name: "Wrench",
Quantity: 150,
}
// Serialize to binary bytes
data, err := proto.Marshal(msg)
if err != nil {
log.Fatal(err)
}
// Deserialize back into a fresh struct
decoded := &gen.Item{}
if err := proto.Unmarshal(data, decoded); err != nil {
log.Fatal(err)
}
fmt.Println(decoded.Name, decoded.Quantity)
}
The proto.Marshal call walks the struct, encodes each non-zero field into the wire format, and returns a byte slice. The wire format uses variable-length integers for small numbers, length-delimited bytes for strings and nested messages, and packed arrays for repeated fields. You do not need to understand the encoding rules to use the library. You do need to know that the output is not human-readable. Do not log raw protobuf bytes in production. Decode them first or use a hex dump.
The generated struct fields are public by default because the serialization methods need to read and write them. Go convention says public names start with a capital letter. The protobuf compiler respects this. You will see Id instead of id in the generated code. That is intentional.
Protobuf generates code, not magic. Read the .pb.go file once to see how the compiler maps your schema to Go types.
A realistic service example
Real services receive protobuf payloads over HTTP or gRPC. The pattern is identical: read the body, unmarshal into a typed struct, validate, process, and return a response. Error handling follows the standard Go idiom. The unhappy path stays visible.
package handler
import (
"encoding/json"
"net/http"
"google.golang.org/protobuf/proto"
"yourmodule/gen"
)
// HandleInventoryUpdate processes an incoming protobuf payload
func HandleInventoryUpdate(w http.ResponseWriter, r *http.Request) {
// Read the raw request body
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "bad request", http.StatusBadRequest)
return
}
// Unmarshal into the generated struct
msg := &gen.InventoryUpdate{}
if err := proto.Unmarshal(body, msg); err != nil {
http.Error(w, "invalid protobuf", http.StatusBadRequest)
return
}
// Business logic would go here
// For demonstration, we just echo a confirmation
resp := &gen.Ack{
Status: "ok",
ItemId: msg.ItemId,
}
// Serialize the response
out, err := proto.Marshal(resp)
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
// Set the correct content type for protobuf
w.Header().Set("Content-Type", "application/x-protobuf")
w.Write(out)
}
The handler reads the body, unmarshals it, and returns a new protobuf message. The Content-Type header tells the client how to decode the response. If you forget to set it, clients may try to parse the binary payload as JSON or plain text and fail.
Notice the error handling. Each err != nil check returns immediately. The community accepts this boilerplate because it makes failure points explicit. Do not swallow errors with _ unless you have a documented reason. The underscore discards a value intentionally. Use it for return values you actively chose to ignore, not for errors you forgot to handle.
Protobuf over HTTP works fine for simple services. For long-running streams or bidirectional communication, move to gRPC. The serialization layer stays identical.
Where things break
Protobuf is forgiving with additions and strict with changes. The most common mistake is reusing a deleted field number. If you remove quantity and later add weight with the same tag, old clients will interpret weight as quantity. The data corrupts silently. Never reuse field numbers. Mark deleted fields as reserved in the schema.
message Item {
int32 id = 1;
string name = 2;
reserved 3; // quantity was removed
reserved "quantity";
}
If you forget to regenerate the Go code after editing the schema, the compiler rejects the program with undefined: MessageName or undefined: FieldName. The generated package does not know about your new types. Run protoc again.
Unmarshaling into the wrong type causes a runtime panic. The proto.Unmarshal function expects a pointer to a struct that implements proto.Message. If you pass a raw struct or a pointer to a non-protobuf type, the library panics with proto: message type mismatch. Always allocate a fresh pointer before unmarshaling.
Repeated fields behave like Go slices. The generated code initializes them to nil. Accessing a nil slice is safe in Go, but sending it over the wire omits the field entirely. If you need to distinguish between "empty" and "not set", use proto.HasField or switch to wrapper types in the schema.
Protobuf hides complexity until you break the contract. Guard your field numbers and regenerate often.
When to reach for protobuf
Use protobuf when you need compact, version-safe serialization across services that share a strict schema. Use JSON when you need human-readable payloads, rapid prototyping, or dynamic field shapes. Use MessagePack when you want binary efficiency without a schema compiler. Use flatbuffers when zero-copy deserialization is mandatory for performance-critical paths. Use plain text or CSV when the data is tabular and consumed by spreadsheets or simple parsers.
Protobuf is a compiled contract. JSON is a flexible document. Pick the tool that matches your data lifecycle.