How to Use encoding/binary for Byte-Level Operations
You are building a multiplayer game server. A client sends a packet over the network. The packet is a raw stream of bytes. Your protocol specification says the first four bytes represent the player ID, the next two bytes are the health value, and the following eight bytes are the X coordinate as a floating-point number. You cannot use JSON here. The data is packed tightly to save bandwidth and reduce latency. You need to extract those numbers from the byte slice and put them back when sending a response. This is where encoding/binary lives.
Byte order and the binary contract
Numbers in Go are values in memory. Bytes are values in a slice. Converting between them requires agreement on how the bits map. The biggest agreement is byte order. Some systems write the most significant byte first. Others write the least significant byte first. This is called endianness. Big-endian writes the big end first. Little-endian writes the little end first.
encoding/binary provides functions to read and write these conversions explicitly. You choose the byte order, and the package handles the bit-shifting and masking. The package exports BigEndian and LittleEndian as variables, not functions. These variables implement the ByteOrder interface. You pass them to functions like binary.Read. This is a common Go pattern: use a value to represent a strategy or configuration. The interface defines methods for every primitive type, allowing functions to accept any byte order implementation.
Minimal example: round-trip conversion
Here's how byte order changes the result. Write a value with one endianness and read it with the other to see the mismatch.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
// Buffer holds 4 bytes for a 32-bit value.
buf := make([]byte, 4)
// Write 0x12345678 with BigEndian.
// Index 0 gets the most significant byte.
binary.BigEndian.PutUint32(buf, 0x12345678)
fmt.Printf("BigEndian write: % x\n", buf)
// Read the same buffer with LittleEndian.
// The bytes are interpreted in reverse order.
val := binary.LittleEndian.Uint32(buf)
fmt.Printf("LittleEndian read: %x\n", val)
}
# output:
BigEndian write: 12 34 56 78
LittleEndian read: 78563412
The write places 0x12 at index 0. The read with LittleEndian treats index 0 as the least significant byte. The result is 0x78563412. The data is intact, but the interpretation is wrong. Endianness is a contract. Mismatch the order and the numbers become noise.
What happens under the hood
The Put functions mutate the slice you pass in. They do not allocate a new slice. You must provide a slice of the correct length. The Get functions read from the slice and return the value. They also do not allocate. The operations are pure bit manipulation. The compiler generates efficient machine code for these shifts and masks. There is no reflection overhead. The type is known at compile time.
The binary.Size function returns the number of bytes required to encode a value. It works for basic types and structs. For structs, it sums the sizes of the fields. It does not account for padding. Use binary.Size to allocate buffers before calling Put functions.
// Size returns the byte size of a uint64.
size := binary.Size(uint64(0))
// size is 8.
Realistic example: parsing a protocol header
Here's a realistic parser for a fixed-header protocol. Check bounds, extract fields, validate magic.
import (
"encoding/binary"
"fmt"
)
// ParsePacket extracts fields from a raw binary packet.
// It expects at least 8 bytes for the header.
func ParsePacket(data []byte) (magic uint16, version uint16, length uint32, err error) {
// Check minimum size to prevent panic on short reads.
if len(data) < 8 {
return 0, 0, 0, fmt.Errorf("packet too short: got %d bytes", len(data))
}
// Extract header fields using LittleEndian.
// Many network protocols use LittleEndian for compatibility.
magic = binary.LittleEndian.Uint16(data[0:2])
version = binary.LittleEndian.Uint16(data[2:4])
length = binary.LittleEndian.Uint32(data[4:8])
// Validate magic number matches expected protocol signature.
if magic != 0xABCD {
return 0, 0, 0, fmt.Errorf("invalid magic number: %x", magic)
}
return magic, version, length, nil
}
The encoding/binary functions do not return errors for out-of-bounds access. They panic. You must check slice lengths before calling them. This is a common source of runtime panics in binary parsers. Always validate len(data) against the expected size. The error handling follows the standard pattern. Return the error immediately. The caller decides how to handle it.
Pitfalls and runtime panics
If you call binary.BigEndian.Uint32 on a slice with only 3 bytes, the program crashes with runtime error: slice bounds out of range. The compiler cannot check slice length at compile time. The check must happen at runtime.
If you use binary.Read with a reader that provides fewer bytes than needed, the function returns an error. The error is usually io.ErrUnexpectedEOF. You must handle this error. Ignoring it leads to partial data or zero values.
The binary.Varint functions return a decoded value and a size. If the input does not contain a valid varint, the size is zero. You must check the size to detect truncation. A size of zero means the varint was incomplete or invalid.
Variable-length integers
Variable-length integers encode small numbers in fewer bytes. binary.PutVarint writes a signed integer. binary.PutUvarint writes an unsigned integer. They use a continuation bit. The last byte has the high bit clear. This saves space for small values. Protobuf uses this encoding.
Here's how variable-length integers work. Small numbers shrink to one byte, large numbers expand.
package main
import (
"encoding/binary"
"fmt"
)
func main() {
// Buffer large enough for the maximum varint size.
buf := make([]byte, binary.MaxVarintLen64)
// Encode a small number using variable-length encoding.
// Small values take fewer bytes than fixed-width integers.
n := binary.PutVarint(buf, 100)
fmt.Printf("Encoded 100 in %d bytes: % x\n", n, buf[:n])
// Encode a large number.
// Larger values require more bytes.
n = binary.PutVarint(buf, 1<<30)
fmt.Printf("Encoded 2^30 in %d bytes: % x\n", n, buf[:n])
// Decode the first value back.
val, size := binary.Varint(buf)
fmt.Printf("Decoded: %d, size: %d\n", val, size)
}
# output:
Encoded 100 in 2 bytes: 64 01
Encoded 2^30 in 5 bytes: 80 80 80 80 08
Decoded: 100, size: 2
The value 100 takes two bytes. The value 1<<30 takes five bytes. A fixed uint64 would take eight bytes for both. Varints save space for small values. Fixed-width wins for predictable layout.
Append versus Put
The binary.Append functions append encoded bytes to a slice and return the new slice. binary.Put functions mutate a pre-allocated slice. Use Append when you are constructing a message dynamically and want to avoid manual index tracking. Use Put when you have a pre-allocated buffer and need zero-allocation conversion.
When using binary.Append, the function checks the capacity of the slice. If the slice has enough capacity, it extends the length and writes the bytes. If not, it allocates a new underlying array, copies the existing data, and appends the new bytes. Repeated calls to Append without pre-allocation can trigger multiple allocations. Pre-allocate the slice with make([]byte, 0, expectedSize) to avoid this.
// Build a message using Append.
// Pre-allocate capacity to avoid reallocation.
buf := make([]byte, 0, 12)
buf = binary.AppendUint32(buf, 0x12345678)
buf = binary.AppendUint32(buf, 0xDEADBEEF)
// buf now contains 8 bytes.
Append for building. Put for filling. Choose based on allocation strategy.
Decision matrix
Use binary.Put and binary.Get when you have a pre-allocated buffer and need zero-allocation conversion. Use binary.Append when you are constructing a message dynamically and want to avoid manual index tracking. Use binary.Read and binary.Write when you are parsing a struct and performance is secondary to code clarity. Use binary.Varint functions when encoding integers where small values are common and bandwidth matters. Use encoding/gob when serializing Go types for internal use without defining a binary protocol. Use Protocol Buffers or FlatBuffers when you need cross-language compatibility and schema evolution.