The proto file trap
You define a message in a .proto file. You run protoc with a handful of flags to generate Go code. The build passes. A week later, a teammate adds a field to the message, forgets to update the import path, and the CI pipeline fails with a cryptic error about missing dependencies. Or worse, the build passes, but the client crashes because a field number was reused and the wire format broke. Managing protocol buffers manually feels like juggling chainsaws while riding a unicycle. The compiler works, but the workflow around it is fragile.
buf replaces that fragility with a structured toolchain. It treats proto files like a proper software module: dependencies are explicit, style is enforced, and breaking changes are caught before they merge. You still use the protoc compiler under the hood, but buf handles the orchestration, dependency resolution, and quality checks that protoc leaves to you.
buf is the workflow, protoc is the compiler
protoc is the compiler. It reads .proto files and emits code. It does one thing well. It also requires you to manage include paths, resolve imports, and chain plugins manually. If you have three proto files that import each other and depend on Google's common types, you need to construct a command line that tells protoc exactly where every file lives. One wrong path and the compiler gives up.
buf wraps that compiler with a module system and a set of quality gates. Think of protoc as a raw paintbrush. buf is the entire studio: the easel, the palette, the lighting, and the critic who tells you your perspective is off before you frame the painting. You configure buf once, and every developer and CI pipeline uses the same rules, the same plugin versions, and the same dependency graph.
The tool introduces a module concept similar to Go modules. A buf.yaml file declares the module name, its dependencies, and its linting rules. buf can fetch dependencies from a registry or local files, resolve the full dependency tree, and build a module image. That image is what gets passed to plugins for code generation. This eliminates the "works on my machine" problem where local include paths differ from the build environment.
Minimal setup
Start by initializing the configuration. The buf.yaml file lives at the root of your proto directory and tells buf how to treat your files. You don't need to specify every file manually; buf scans the directory structure based on the config.
Here's the configuration that replaces a dozen command-line flags.
# buf.yaml
# version v1 is the current stable config format
version: v1
# name points to the module identifier, usually a registry path
name: buf.build/my-org/my-service
# lint enforces style rules automatically
lint:
use:
- STANDARD
# breaking checks for API compatibility issues
breaking:
use:
- FILE
The version field locks the config format. buf supports multiple versions, so this prevents silent breakage when the tool updates. The name field identifies the module. Even if you never publish to a registry, giving the module a name helps buf generate accurate error messages and track breaking changes. The lint section enables style checks. STANDARD is a curated set of rules that catches common mistakes like missing field numbers or incorrect naming. The breaking section enables compatibility checks. FILE mode ensures that changes don't break the file-level API contract.
Run buf build to verify the module. This command resolves dependencies, checks syntax, and runs lint rules. If everything is clean, buf produces a module image in memory. You can then run buf generate to produce code. The generation step uses plugins to transform the module image into Go structs, gRPC stubs, or any other target language.
buf build validates. buf generate produces code. Run both.
Generation rules
Hardcoding plugin flags in shell scripts leads to drift. One developer generates with paths=source_relative, another uses the default, and the imports break. buf solves this by centralizing generation rules in a buf.gen.yaml file. This file defines which plugins to run, what options to pass, and where to write the output.
Generation rules live in a separate file so the team shares the exact same plugin versions.
# buf.gen.yaml
# version v1 matches the config format
version: v1
# plugins define how to transform proto files into code
plugins:
# go plugin generates Go structs and methods
- plugin: go
# opt sets plugin-specific options like paths
opt: paths=source_relative
# out specifies where generated files land
out: gen/go
# go-grpc plugin generates gRPC service stubs
- plugin: go-grpc
opt:
- paths=source_relative
- require_unimplemented_servers=false
out: gen/go
The plugin field names the generator. buf looks for the plugin in the local PATH or downloads it from a registry if configured. The opt field passes options to the plugin. For Go, paths=source_relative is the standard convention. It tells the generator to place output files relative to the source proto file, which keeps Go imports clean and matches the directory structure. The out field sets the base directory for generated files.
The go-grpc plugin adds the gRPC service methods. The require_unimplemented_servers=false option is a Go-specific convention. It tells the generator not to require implementing every method in the server interface, which allows for partial implementations and cleaner mock setups in tests.
Run buf generate --template buf.gen.yaml to produce code. The command reads the template, resolves the plugins, and runs them against the module image. Every developer gets the same output because the template is version-controlled.
Lock the generation config. Share the pain.
Dependencies and the module system
Proto files often import common types from Google or other libraries. With protoc, you need to download those files, manage their versions, and pass the correct include paths. buf treats these as dependencies. You declare them in buf.yaml, and buf fetches them automatically.
Here's how to add external dependencies to your module.
# buf.yaml
version: v1
name: buf.build/my-org/my-service
# deps lists external modules this module imports
deps:
# googleapis provides common types like Duration and Timestamp
- buf.build/googleapis/googleapis
# envoyproxy provides gRPC gateway annotations
- buf.build/envoyproxy/envoy
lint:
use:
- STANDARD
breaking:
use:
- FILE
The deps field lists module identifiers. buf resolves these identifiers to specific versions. If you use a registry like buf.build, the tool fetches the exact commit or version pinned in your lock file. This ensures that everyone builds against the same dependency tree. If a dependency updates, you control when to upgrade by updating the lock file, not by hoping someone remembers to pull the latest files.
When you run buf build, the tool downloads the dependencies, checks their integrity, and merges them into the module image. Plugins see the full graph, so they can generate code that references types from dependencies without you manually managing include paths.
If a dependency is missing or unreachable, buf fails fast. The tool reports dependency "buf.build/googleapis/googleapis" not found if the registry is down or the identifier is wrong. If a dependency has a breaking change that affects your code, buf can catch that during the build or in CI.
Dependencies are first-class citizens. Pin them. Version them. Trust the lock file.
Linting and breaking changes
buf excels at catching errors that protoc ignores. The compiler checks syntax, but it doesn't care if you reuse a field number or name a field incorrectly. buf lint enforces style and correctness rules. buf breaking checks for API compatibility issues.
Run buf lint to check style. The tool scans your proto files against the rules defined in buf.yaml. If you use STANDARD, you get checks for field ordering, reserved numbers, and naming conventions. If a rule fails, buf reports the violation with a clear message. The tool might complain with field "id" should be named "id" (FILE_LOWER_SNAKE_CASE) if your naming doesn't match the convention.
Run buf breaking to check for API changes. This command compares your current proto files against a previous version. You can compare against a branch, a commit, or a remote image. The tool analyzes the changes and reports any breaking modifications. If you rename a field without keeping the old name in reserved, buf flags it. The tool reports Field number 1 is reused by field "new_name" if you accidentally reuse a number. If you change a field type in a way that breaks wire compatibility, buf catches that too.
Breaking changes are bugs. Catch them early.
Pitfalls and errors
buf is strict by design. It rejects configurations that would lead to inconsistent builds. You'll encounter errors if you drift from the conventions.
If you forget to specify a plugin version in the template, buf might use a cached version that differs from your teammate's. The tool warns with plugin "go" version not specified if you rely on the default. Pin your plugin versions in the template to avoid drift.
If you have circular dependencies between modules, buf fails during the build. The tool reports circular dependency detected and lists the modules involved. Proto files shouldn't have circular imports. Restructure the code to break the cycle.
If you run buf generate without a template and without plugins installed, buf doesn't know what to do. The tool reports no plugins found and suggests creating a template. Always use a template for reproducible builds.
buf errors are actionable. Read them. Fix the config. Move on.
When to use buf versus protoc
Use buf when your project has multiple proto files or imports external dependencies. Use buf when you need to enforce style guides or catch breaking changes in CI. Use buf when generating code for multiple languages from the same proto set. Use buf when you want a single source of truth for plugin versions and generation rules. Use protoc directly only when you have a single proto file with no imports and no need for linting or version management. Use protoc directly when you are debugging a plugin issue and need to isolate the compiler from the toolchain.
buf is the gofmt of the proto world. Trust the toolchain.