When a separate database server gets in the way
You are building a CLI tool that tracks user progress. You want the data to survive a restart, but spinning up PostgreSQL feels like overkill. You do not want to manage a separate database process, handle network latency, or write migration scripts. You just need a reliable place to store key-value pairs that lives inside your own binary. That is exactly where Badger fits.
Badger is an embedded key-value store. Embedded means the database runs inside your Go process as a library, not as a separate server. Think of it like a personal notebook you keep on your desk instead of a shared office filing cabinet. You open it, write in it, and close it. The data stays on disk, survives crashes, and handles concurrent reads and writes without you writing a single lock. Under the hood, Badger uses an LSM-tree architecture. That stands for Log-Structured Merge-tree. Instead of updating records in place like a traditional database, Badger appends writes to a log, periodically sorts them into sorted string tables, and merges them in the background. This design makes writes incredibly fast and keeps read performance predictable.
Badger trades complex query language for raw speed and simplicity. Keep your keys predictable and your values small.
How Badger actually works
When you call badger.Open, the library checks the target directory. If it does not exist, Badger creates it and initializes the value log and level-zero SST files. The value log is where raw writes land first. It is a simple append-only file. Writing to an append-only file is fast because the disk head never needs to seek backward. Once the value log grows large enough, Badger triggers compaction. Compaction reads the unsorted log, merges it with existing data, sorts everything by key, and writes it into immutable SST files organized in levels. Reads check the in-memory memtable first, then scan the value log, and finally fall back to the SST files. This separation of concerns keeps write throughput high while maintaining fast lookups.
The db.Update and db.View methods wrap your code in transactions. Badger uses optimistic concurrency control. Multiple read transactions run simultaneously without blocking. Write transactions acquire a lock only when they are ready to commit. If two writers conflict, one rolls back and you get a retry error. You handle that by wrapping the call in a simple retry loop.
Transactions are short-lived. Commit fast, read often, and let compaction run in the background.
Minimal example: opening and writing
Here is the smallest working program that opens a database and stores a single key.
package main
import (
"log"
"github.com/dgraph-io/badger/v4"
)
func main() {
// Open creates the directory structure and initializes the LSM tree.
db, err := badger.Open(badger.DefaultOptions("/tmp/badger-example"))
if err != nil {
log.Fatal(err)
}
// Always close the database to flush pending writes and release file handles.
defer db.Close()
// Update starts a write transaction that batches changes before flushing.
err = db.Update(func(txn *badger.Txn) error {
// Keys and values must be byte slices. Badger handles the serialization.
return txn.Set([]byte("user:101"), []byte("active"))
})
if err != nil {
log.Fatal(err)
}
}
If you pass a string instead of a byte slice to txn.Set, the compiler rejects the program with cannot use "key" (untyped string constant) as []byte value in argument. Go's type system catches this at compile time so you never accidentally store malformed data.
Minimal example: reading back
Here is how you retrieve that value in a separate step.
func readStatus(db *badger.DB) {
// View starts a read-only transaction that guarantees a consistent snapshot.
err := db.View(func(txn *badger.Txn) error {
item, err := txn.Get([]byte("user:101"))
if err != nil {
return err
}
// ValueCopy reads the data into a new byte slice so it survives after the transaction ends.
val, err := item.ValueCopy(nil)
if err != nil {
return err
}
log.Printf("Status: %s", string(val))
return nil
})
if err != nil {
log.Fatal(err)
}
}
Go conventions favor explicit resource management. You will see defer db.Close() in almost every Badger example. The community treats it as standard practice because it guarantees cleanup even if a panic occurs. The same applies to iterators. Pair every allocation with a deferred cleanup, and the garbage collector handles the rest.
Realistic example: scanning a namespace
Real applications rarely store just one key. You usually group related data under a common prefix and iterate over it. Here is how you scan a namespace efficiently.
func listActiveUsers(db *badger.DB) error {
// View guarantees that the iteration sees a consistent snapshot of the data.
return db.View(func(txn *badger.Txn) error {
// PrefixIterator automatically filters keys that start with the given prefix.
opts := badger.DefaultIteratorOptions
opts.Prefix = []byte("user:")
it := txn.NewIterator(opts)
// Iterators hold file handles and memory buffers. Closing them prevents leaks.
defer it.Close()
for it.Rewind(); it.Valid(); it.Next() {
item := it.Item()
// Key returns the raw byte slice. It is only valid inside this loop.
k := string(item.Key())
v, err := item.ValueCopy(nil)
if err != nil {
return err
}
log.Printf("%s -> %s", k, string(v))
}
// Iterators can encounter I/O errors during traversal. Always check the final error.
return it.Error()
})
}
Iterators are stateful machines. Rewind them, check validity, and close them before the function returns.
Pitfalls and runtime boundaries
Badger is robust, but it enforces strict rules. If you try to read a key that does not exist, the runtime returns badger.ErrKeyNotFound. You must check it explicitly. The Go community embraces verbose error handling for this reason. The unhappy path stays visible in your code. Another common trap is holding a transaction open too long. Badger limits transaction lifetime to prevent memory bloat. If your read transaction runs for more than a few seconds, Badger rejects subsequent operations with a transaction timeout error. Keep transactions short. Do a lookup, extract the value, and return.
If you forget to close an iterator, you leak file descriptors. The operating system will eventually refuse to open new files, and your program crashes with a too many open files panic. Always pair txn.NewIterator with a defer it.Close(). Memory usage is another consideration. Badger loads level-zero SST files into memory for fast lookups. If you write millions of keys without tuning, the memory footprint grows. You control this by adjusting MaxTableSize and ValueLogFileSize in the options struct before calling Open. Smaller tables mean more files but lower memory usage. Larger tables mean fewer files but higher memory pressure. Pick a balance that matches your available RAM.
Disk space is finite. Monitor your value log size and tune table limits before production traffic hits.
Decision matrix
Use Badger when you need a persistent key-value store that runs inside your process without managing a separate database server. Use an in-memory map when your data is temporary and you do not care about surviving a restart. Use Redis when multiple processes or machines need to share the same cache and you can tolerate network latency. Use SQLite when you need relational queries, joins, and a mature SQL ecosystem. Use a flat JSON or YAML file when your dataset is tiny and you prefer human-readable backups.
Pick the tool that matches your access pattern. Embedded stores stay embedded for a reason.