The kitchen drawer you're ignoring
You are building a CLI tool to back up logs. You need to bundle files into an archive. You start searching GitHub for a tar library. You stop. The standard library already has archive/tar. You need to count how many times a specific debug flag triggered. You look for a metrics library. runtime/metrics is already there. You need to concatenate binary data without allocating a new string every time. bytes.Buffer is waiting.
The Go standard library is a treasure chest that most people walk past because they only look at fmt and net/http. The packages inside are stable, fast, and require no go mod tidy for new dependencies. They follow the same conventions as the rest of the ecosystem. Using them keeps your dependency tree flat and your code portable.
This article covers three packages that solve real problems without external imports: archive/tar for file archiving, bytes for efficient byte manipulation, and runtime/metrics for observing runtime behavior.
Archive files with archive/tar
The archive/tar package reads and writes tar archives. A tar file is a container format that bundles multiple files into a single stream. It does not compress data. Compression is a separate step handled by packages like compress/gzip.
Tar archives are the backbone of container images and system backups. Every Docker image layer is a tar archive. Every tar.gz file you download starts as a tar stream.
Here's the simplest way to create a tar archive in memory.
package main
import (
"archive/tar"
"bytes"
"fmt"
)
func main() {
// Buffer stores the archive in memory.
var buf bytes.Buffer
// Writer formats the output as a tar stream.
tw := tar.NewWriter(&buf)
// WriteHeader adds file metadata directly.
tw.WriteHeader(&tar.Header{Name: "data.txt", Size: 7})
// Write adds the file content.
tw.Write([]byte("payload"))
// Close flushes padding and finalizes.
tw.Close()
fmt.Println("Size:", buf.Len())
}
The tar.Header struct defines the metadata for each entry. The Name field holds the file path. The Size field holds the length of the file content in bytes. The Mode field holds Unix permissions. If you omit Size for a regular file, the archive might be malformed.
The tar.NewWriter function takes an io.Writer. bytes.Buffer implements io.Writer, so the tar data flows into the buffer. This is the "accept interfaces" pattern in action. The tar writer doesn't care if the destination is a buffer, a file, or a network connection.
Convention aside: tar.Header fields are public because they start with capital letters. You set them directly. The package doesn't use setters. Go structs are often used as data bags where fields are public and validation happens on write or read.
Walking through the archive format
A tar archive is a sequence of 512-byte blocks. Each file entry starts with a 512-byte header block. The header contains the filename, size, mode, modification time, and other metadata. The header is followed by the file content, padded to a 512-byte boundary. The archive ends with two 512-byte blocks of zeros.
When you call tw.WriteHeader, the package encodes the header struct into the 512-byte format. When you call tw.Write, it streams the content. When you call tw.Close, it writes the zero blocks to signal the end.
If you forget to close the writer, the archive is incomplete. The compiler won't catch this. The code runs, but the resulting tar file is truncated. Always close the writer.
Realistic example: compressed archives
Real archives usually compress data. You combine archive/tar with compress/gzip to create a .tar.gz file. The gzip writer compresses the stream as it flows through. The tar writer formats the files inside the gzip stream.
Here's a helper function that builds a compressed archive.
// CreateTarGzip builds a compressed archive from a single file.
func CreateTarGzip(name string, content []byte) ([]byte, error) {
// Buffer collects the output.
var buf bytes.Buffer
// Gzip compresses the stream.
gw := gzip.NewWriter(&buf)
// Tar formats the file structure.
tw := tar.NewWriter(gw)
// Header defines entry metadata.
hdr := &tar.Header{Name: name, Size: int64(len(content))}
// WriteHeader adds metadata.
if err := tw.WriteHeader(hdr); err != nil {
return nil, fmt.Errorf("header: %w", err)
}
// Write adds content.
if _, err := tw.Write(content); err != nil {
return nil, fmt.Errorf("write: %w", err)
}
The close calls matter. You must close the tar writer before the gzip writer. The tar writer adds padding blocks that need to be compressed. If you close gzip first, the padding is lost.
// Close tar to flush padding.
if err := tw.Close(); err != nil {
return nil, fmt.Errorf("tar: %w", err)
}
// Close gzip to finalize compression.
if err := gw.Close(); err != nil {
return nil, fmt.Errorf("gzip: %w", err)
}
return buf.Bytes(), nil
}
Convention aside: Error wrapping with fmt.Errorf and %w is the standard way to propagate errors. The wrapper adds context without hiding the original cause. Callers can use errors.Is or errors.As to inspect the chain.
Pitfalls and errors
The tar package doesn't validate filenames strictly. If you write a header with a relative path like ../secret.txt, the archive contains that path. Extracting the archive could overwrite files outside the target directory. Always sanitize filenames before writing.
The tar.Header.Size field is an int64. If you pass a negative size, the compiler accepts it, but the archive is invalid. The compiler complains with cannot use -1 (untyped int constant) as int64 value in struct literal if you try to assign a negative untyped constant to a field that expects a specific type, but int64(-1) compiles fine. Runtime validation is your responsibility.
Tar is a container, not a compressor. Pair it with gzip or zstd when size matters.
Efficient byte manipulation with bytes
The bytes package provides functions for manipulating byte slices. It mirrors the strings package but works with []byte instead of string. Use bytes when you are dealing with binary data, network protocols, or file formats. Use strings when you are dealing with text.
Here's how to build binary data efficiently.
package main
import (
"bytes"
"fmt"
)
func main() {
// Buffer grows dynamically as you write.
var buf bytes.Buffer
// WriteString appends text to the buffer.
buf.WriteString("Hello, ")
// WriteByte appends a single byte.
buf.WriteByte(33)
// String returns the buffer content as a string.
fmt.Println(buf.String())
}
The bytes.Buffer type is a growable byte slice. It avoids allocations by reusing internal storage. When you write data, the buffer expands only when necessary. This is much faster than concatenating strings or byte slices with +.
Convention aside: bytes.Buffer implements io.Writer and io.Reader. You can pass it to any function that expects an io.Writer. This makes it useful for capturing output from functions that write to streams.
Reading with bytes.Reader
Sometimes you have a byte slice and need an io.Reader. bytes.NewReader creates a reader that wraps the slice without copying it.
package main
import (
"bytes"
"fmt"
"io"
)
func main() {
// Data is the source byte slice.
data := []byte("stream data")
// Reader wraps the slice without copying.
r := bytes.NewReader(data)
// Read reads up to 5 bytes.
buf := make([]byte, 5)
n, _ := r.Read(buf)
fmt.Printf("Read %d bytes: %s\n", n, buf)
}
The bytes.Reader tracks the current position. Each Read call advances the position. You can reset the position with Seek. This is useful for parsing binary formats where you need to read headers, then seek back to read content.
Pitfalls and errors
bytes.Buffer is not thread-safe. If multiple goroutines write to the same buffer, you get data races. The compiler won't catch this. Use a mutex or separate buffers per goroutine.
The bytes.Buffer.Write method expects a []byte. If you pass a string, the compiler rejects it with cannot use s (variable of type string) as []byte value in argument. Convert the string with []byte(s) or use WriteString.
bytes.Buffer has a Truncate method that reduces the size. It doesn't free memory immediately. The internal buffer stays allocated for future writes. This is an optimization. If you need to free memory, create a new buffer.
Use bytes.Buffer for building data. Use string for the result.
Observability with runtime/metrics
The runtime/metrics package exposes runtime metrics as counters and gauges. You can read garbage collection stats, scheduler info, and memory usage without external libraries. The metrics are low-overhead and designed for production use.
Here's how to read runtime metrics.
package main
import (
"fmt"
"runtime/metrics"
)
func main() {
// Sample defines which metric to read.
samples := []metrics.Sample{
{Name: "/gc/gogc:percent"},
}
// Read populates the samples with current values.
metrics.Read(samples)
// Value holds the metric data.
fmt.Println("GOGC:", samples[0].Value.Float64())
}
The metrics.Sample struct holds the metric name and the value. The Name field is a string that identifies the metric. The Value field is a union that can hold different types. You access the value with methods like Uint64(), Float64(), or Bool().
Convention aside: Metric names are strings. Use constants or variables to avoid typos. A typo in the name causes metrics.Read to return a zero value. The compiler won't catch this. Define metric names in a package-level variable.
Metric types and kinds
Metrics have different kinds. Counters only increase. Gauges can go up or down. The metrics.Sample struct has a Kind field that tells you the type. Use the correct accessor method based on the kind.
package main
import (
"fmt"
"runtime/metrics"
)
func main() {
// Sample requests a counter metric.
samples := []metrics.Sample{
{Name: "/sched/gomaxprocs:threads"},
}
metrics.Read(samples)
// Check the kind before accessing the value.
switch samples[0].Kind {
case metrics.KindUint64:
fmt.Println("GOMAXPROCS:", samples[0].Value.Uint64())
case metrics.KindFloat64:
fmt.Println("GOMAXPROCS:", samples[0].Value.Float64())
default:
fmt.Println("Unknown kind")
}
}
The /sched/gomaxprocs:threads metric reports the value of GOMAXPROCS. It's a counter that represents a configuration value. The /gc/gogc:percent metric reports the GC trigger percentage. It's a gauge.
Pitfalls and errors
metrics.Read allocates memory for the samples slice. If you call it in a tight loop, the allocations add up. Reuse the samples slice across calls. Allocate once, read many times.
Some metrics are only available in newer Go versions. If you request a metric that doesn't exist, the value is zero. The compiler won't warn you. Check the Go release notes for metric availability.
The runtime/metrics package doesn't support custom metrics. It only exposes runtime internals. If you need application metrics, use a third-party library or build your own counters.
Metrics are free observability. Read them before adding a telemetry agent.
When to use these packages
The standard library covers many common tasks. Knowing when to reach for built-in packages versus third-party libraries saves dependencies and reduces maintenance.
Use archive/tar when you need to create or extract tar archives for backups, container layers, or file bundling. Use compress/gzip alongside it when you need compression. Use bytes.Buffer when you need to build binary data efficiently or capture output from an io.Writer. Use bytes.Reader when you need an io.Reader from a byte slice without copying. Use runtime/metrics when you need to observe garbage collection, scheduler, or memory stats without external dependencies. Use third-party libraries when you need features the standard library doesn't provide, like advanced compression algorithms, custom metric exporters, or complex archive formats.