The drone is screaming, and JSON is too slow
You are building a telemetry system for a fleet of drones. Each drone sends position, velocity, and battery status every 10 milliseconds. The network is a noisy 4G link. You need every byte to count. JSON adds overhead you cannot afford. The string "latitude": 37.7749 takes 22 bytes. The raw float64 takes 8 bytes. Multiply that by a thousand drones, and the bandwidth bill explodes.
You also need to handle fragmentation. TCP streams bytes, not messages. A single Read call might return half a header and part of the payload. Or it might return three full messages at once. You need a parser that understands the boundaries of your protocol, reads exactly what you asked for, and turns raw bytes into structured data without allocating garbage on every packet.
That is where a custom binary protocol parser comes in. You define the layout, you read the bytes, you get the data. No text encoding, no escaping, no bloat.
Binary protocols are just contracts for bytes
A binary protocol is a contract that maps byte positions to meaning. Think of it like a shipping container manifest. The first few inches of the container label always hold the destination code. The next few inches hold the weight. The rest of the container holds the cargo. If you know the layout, you can load and unload quickly. You do not need labels on every box inside. You just count the inches.
In code, this means reading a fixed number of bytes for an ID, then a fixed number for a length field, then that many bytes for the payload. The encoding/binary package in Go handles the conversion between byte slices and integers. The io package handles the stream mechanics. Together they let you build a parser that is fast, predictable, and easy to test.
Minimal parser
Here is the struct definition for a simple message format. The header contains a 2-byte ID and a 4-byte length. The payload is a variable-length byte slice.
// Header contains fixed-size fields.
type Header struct {
ID uint16
Len uint32
}
// Message embeds Header and holds the payload.
type Message struct {
Header
Data []byte
}
The parser reads the header, allocates a buffer for the payload, and reads the payload. It uses io.Reader so it works with files, network connections, or in-memory buffers.
package main
import (
"encoding/binary"
"io"
)
// ParseMessage reads a header and payload from the stream.
func ParseMessage(r io.Reader) (*Message, error) {
var m Message
// Read header fields in one call.
if err := binary.Read(r, binary.BigEndian, &m.Header); err != nil {
return nil, err
}
// Pre-allocate payload slice.
m.Data = make([]byte, m.Len)
// ReadFull blocks until payload is complete.
if _, err := io.ReadFull(r, m.Data); err != nil {
return nil, err
}
return &m, nil
}
binary.Read takes the reader, the byte order, and a pointer to the destination. It uses reflection to inspect the struct fields, determines their size, and reads the bytes sequentially. BigEndian is the standard for network protocols because the most significant byte comes first, matching the historical network byte order.
io.ReadFull is the workhorse for the payload. A raw Read call can return fewer bytes than requested. ReadFull loops internally until it fills the buffer or encounters an error. This prevents partial reads from corrupting your data.
Goroutines are cheap. Channels are not magic.
Walkthrough: what happens under the hood
When you call binary.Read(r, binary.BigEndian, &m.Header), the function looks at m.Header. It sees ID is a uint16, so it needs 2 bytes. It sees Len is a uint32, so it needs 4 bytes. It calls r.Read to get 6 bytes. If the reader returns fewer bytes, binary.Read keeps calling until it has enough or hits an error.
The bytes are converted to integers based on the byte order. In BigEndian, the first byte is the high byte. If the stream contains 0x00 0x01, the result is 1. If you used LittleEndian, the result would be 256. Mismatched endianness is a common source of bugs. Always match the byte order to the sender.
binary.Read ignores struct padding. Go may add padding between fields for alignment, but binary.Read reads fields sequentially without skipping bytes. This matches most wire protocols, which do not include padding. If your protocol requires padding, you must add explicit fields to absorb the bytes.
The if err != nil check is verbose by design. The Go community accepts the boilerplate because it makes the unhappy path visible. You cannot accidentally ignore an error. If the stream ends early, the parser returns an error immediately. This keeps the code safe and debuggable.
Trust the type system. Wrap the value or change the design.
Realistic usage with buffering
In production, you rarely read from a raw io.Reader directly. Each Read call on a network connection triggers a system call. System calls are expensive. You should wrap the reader in a bufio.Reader to reduce overhead. bufio.Reader reads a large chunk of data into memory and serves subsequent reads from the buffer.
binary.Read works seamlessly with bufio.Reader because the buffer implements io.Reader. The parser code stays the same. You just wrap the input.
package main
import (
"bufio"
"bytes"
"encoding/binary"
"fmt"
"io"
)
// ParseMessage reads a header and payload from the stream.
func ParseMessage(r io.Reader) (*Message, error) {
var m Message
// Read header fields in one call.
if err := binary.Read(r, binary.BigEndian, &m.Header); err != nil {
return nil, err
}
// Pre-allocate payload slice.
m.Data = make([]byte, m.Len)
// ReadFull blocks until payload is complete.
if _, err := io.ReadFull(r, m.Data); err != nil {
return nil, err
}
return &m, nil
}
func main() {
// Simulate a stream with a buffer.
raw := []byte{0x00, 0x01, 0x00, 0x00, 0x00, 0x03, 'H', 'i', '!'}
buf := bufio.NewReader(bytes.NewReader(raw))
msg, err := ParseMessage(buf)
if err != nil {
fmt.Println("Error:", err)
return
}
fmt.Printf("ID: %d, Data: %s\n", msg.ID, msg.Data)
}
The bufio.NewReader wraps the bytes.Reader. The parser calls Read on the buffer. The buffer serves data from memory. No system calls occur during the parse. This pattern is essential for high-throughput servers.
If you wrap the parser in a struct, name the receiver with one or two letters matching the type. Use (p *Parser) ParseMessage, not (this *Parser) or (self *Parser). The convention keeps the code concise and idiomatic.
Buffering reduces syscalls. Wrap your readers.
Pitfalls and runtime errors
Binary parsers face three main risks: endianness mismatches, length validation failures, and partial reads.
Endianness mismatches produce garbage data. If the sender uses LittleEndian and the parser uses BigEndian, every integer is wrong. The compiler cannot catch this. You must document the byte order and test with known values.
Length validation failures cause panics. The Len field is a uint32. If a malicious sender sets Len to a huge value, make([]byte, m.Len) panics with panic: runtime error: makeslice: len out of range. You must validate the length before allocating. Check against a maximum allowed size.
const maxPayload = 1 << 20 // 1 MB
if m.Len > maxPayload {
return nil, fmt.Errorf("payload too large: %d", m.Len)
}
Partial reads return errors. If the stream ends before the header is complete, binary.Read returns io.EOF. If the stream ends before the payload is complete, io.ReadFull returns io.ErrUnexpectedEOF. These errors tell you exactly where the parse failed. Handle them appropriately. Do not panic on io.EOF in a long-running stream; it just means the connection closed.
The worst goroutine bug is the one that never logs.
Decision matrix
Use encoding/binary when you have fixed-size integer fields and need a quick parser without manual byte manipulation.
Use a manual byte-slice approach when you need zero-allocation parsing for high-throughput systems and are willing to write more code.
Use gob or protobuf when you need schema evolution, cross-language compatibility, and are willing to accept the dependency overhead.
Use JSON when human readability matters more than bandwidth and performance is not critical.
Use unsafe when you are absolutely certain about memory layout and need raw speed, though this breaks portability and should be avoided in most cases.
Validate lengths before allocating. Trust nothing from the network.