MessagePack: JSON's binary shadow
You're building a high-throughput service. Bandwidth costs money. Latency kills user experience. You're sending JSON, but half the payload is curly braces, quotes, and repeated keys. You need a format that keeps the structure of JSON but strips the text overhead. MessagePack is a binary serialization format that encodes data efficiently while staying compatible with JSON's data model. It gives you smaller payloads and faster parsing without forcing you to generate code or lock into a strict schema.
MessagePack treats data as a stream of types and values. It uses a type tag to indicate what follows, then writes the raw bytes. A string is a length prefix followed by UTF-8 bytes. An integer is encoded based on its magnitude. Small integers fit in one byte. Large integers use more. The result is a compact binary blob. You can decode it back into Go structs, or into generic maps and slices. The library handles the translation.
MessagePack is JSON's binary shadow. Same shape, less weight.
Basic round-trip
Here's the simplest usage: define a struct, marshal to bytes, unmarshal back. The msgpack/v5 package provides Marshal and Unmarshal functions that mirror the JSON package API.
package main
import (
"fmt"
"github.com/vmihailenco/msgpack/v5"
)
type User struct {
ID int
Name string
}
func main() {
// Create a struct with concrete values to serialize.
user := User{ID: 1, Name: "Alice"}
// Marshal converts the struct to a []byte slice in MessagePack format.
data, err := msgpack.Marshal(user)
if err != nil {
panic(err)
}
// Unmarshal reads the bytes back into a new struct instance.
var decoded User
err = msgpack.Unmarshal(data, &decoded)
if err != nil {
panic(err)
}
// Verify the round-trip preserved the data.
fmt.Printf("Decoded: %+v\n", decoded)
}
# prints:
Decoded: {ID:1 Name:Alice}
Marshal writes bytes. Unmarshal reads them. Tags control the mapping.
How encoding works
When you call Marshal, the library inspects your struct using reflection. It looks at each exported field, checks for struct tags, and writes a compact binary stream. The v5 version of the library caches reflection metadata, so repeated marshals of the same type are significantly faster than v4.
Unmarshal does the reverse. It reads the stream, sees a map type, allocates a struct, and fills fields by matching keys. If the binary stream has a key your struct doesn't have, the library skips it. If your struct has a field the stream doesn't, the field stays at its zero value. This lenient behavior makes MessagePack safe for evolving schemas. New fields on the receiver won't break old data. Old fields in the data won't break new receivers.
The if err != nil pattern is verbose by design. It forces you to acknowledge failure paths. In examples, panic keeps the focus on the mechanism, but production code should return errors. Never swallow errors in serialization code; a corrupt payload usually indicates a deeper issue.
Controlling the wire format
Real code needs control over field names and optional fields. MessagePack supports struct tags just like the JSON package. Use msgpack:"key" to rename a field in the wire format. Use omitempty to skip zero values. This keeps your internal Go naming clean while matching external API contracts.
package main
import (
"fmt"
"github.com/vmihailenco/msgpack/v5"
)
// Event represents a domain object with wire-format control via tags.
type Event struct {
EventType string `msgpack:"type"`
Payload string `msgpack:"data,omitempty"`
Timestamp int64 `msgpack:"ts"`
}
func main() {
// Omit the Payload field since it is empty and tagged omitempty.
event := Event{EventType: "click", Timestamp: 1678886400}
// Marshal produces a compact binary representation.
data, _ := msgpack.Marshal(event)
// Print the size to demonstrate compactness compared to JSON.
fmt.Printf("Size: %d bytes\n", len(data))
// Decode back to verify tag mapping works correctly.
var decoded Event
_ = msgpack.Unmarshal(data, &decoded)
fmt.Printf("Type: %s, TS: %d\n", decoded.EventType, decoded.Timestamp)
}
# prints:
Size: 15 bytes
Type: click, TS: 1678886400
Tags are your contract. Keep internal names Go-friendly, wire names API-friendly.
Use _ to discard values you've considered but don't need. In the example above, _ discards the error from Marshal and Unmarshal to keep the output clean. In production, handle those errors. Don't use _ for errors in critical paths.
Custom types and interfaces
Sometimes the default behavior isn't enough. You can implement msgpack.Marshaler and msgpack.Unmarshaler interfaces to control encoding. This is useful for types like time.Time, custom enums, or values that need special compression.
package main
import (
"fmt"
"github.com/vmihailenco/msgpack/v5"
)
// Status implements custom encoding for a domain type.
type Status int
const (
StatusActive Status = 1
StatusPaused Status = 2
)
// MarshalMsgpack writes the status as a single byte integer.
func (s Status) MarshalMsgpack() ([]byte, error) {
// Encode as a fixint to save space.
return []byte{byte(s)}, nil
}
// UnmarshalMsgpack reads a single byte back into the status.
func (s *Status) UnmarshalMsgpack(b []byte) error {
// Validate the input length before reading.
if len(b) != 1 {
return fmt.Errorf("invalid status length")
}
*s = Status(b[0])
return nil
}
func main() {
// Marshal the custom type using the interface method.
data, _ := msgpack.Marshal(StatusActive)
fmt.Printf("Size: %d bytes\n", len(data))
// Unmarshal back to verify the custom logic.
var decoded Status
_ = msgpack.Unmarshal(data, &decoded)
fmt.Printf("Status: %d\n", decoded)
}
# prints:
Size: 1 bytes
Status: 1
Receiver names should be short, usually one or two letters matching the type. Use (s Status), not (this Status) or (self Status). This is a Go community convention that keeps code readable.
Custom marshalers give you control. Implement the interface when the default falls short.
Streaming large data
For large datasets, buffering everything in memory is wasteful. Use the encoder and decoder wrappers to stream data. This works with io.Writer and io.Reader. You can encode multiple items sequentially without building a slice.
package main
import (
"bytes"
"github.com/vmihailenco/msgpack/v5"
)
type Item struct {
ID int
Name string
}
func main() {
// Create a buffer to simulate an io.Writer.
var buf bytes.Buffer
// Create an encoder bound to the writer.
enc := msgpack.NewEncoder(&buf)
// Encode items sequentially without building a slice.
enc.Encode(Item{ID: 1, Name: "Alpha"})
enc.Encode(Item{ID: 2, Name: "Beta"})
// Create a decoder bound to the reader.
dec := msgpack.NewDecoder(&buf)
// Decode items back in order.
var item Item
dec.Decode(&item)
fmt.Printf("First: %+v\n", item)
}
# prints:
First: {ID:1 Name:Alpha}
Streaming keeps memory flat. Encode to writers, decode from readers.
Pitfalls and errors
Unexported fields are invisible to the encoder. If you have a lowercase field, it won't appear in the binary output. The compiler won't catch this; the data just disappears silently. Always check your tags and field visibility.
Circular references cause a runtime panic. If struct A contains struct B, and struct B contains struct A, the marshaler loops forever until the stack overflows. The library detects this and panics with a message like msgpack: circular reference detected. Break cycles with pointers and careful design, or use a custom marshaler.
Decoding into an empty interface requires a concrete type. You can't unmarshal into var v interface{} and expect magic. The library needs to know what to allocate. Unmarshal into a struct or a map[string]any. If you pass the wrong type, you get a runtime error like msgpack: cannot unmarshal array into Go value of type string.
Reflection hides bugs. Check your tags. Watch for cycles.
When to use MessagePack
Use MessagePack when you need a drop-in replacement for JSON with smaller payloads and faster parsing, and your schema is flexible. Use JSON when humans need to read the data directly, or you are interacting with systems that only accept text. Use Protocol Buffers when you have a strict schema, need backward compatibility guarantees, and are willing to generate code from a .proto file. Use plain binary formats when you control both ends of the wire and need maximum performance with zero reflection overhead.
Pick the format that matches your constraints. Don't optimize prematurely.