The plugin problem
You built a CLI tool that users love. Now they want custom formatters, or a way to export data to a proprietary format, or a hook that runs after every command. You can't ask every user to fork your repository, add their code, and rebuild the binary. You want them to drop a file in a plugins/ directory and have it just work.
That is the plugin problem. You need extensibility without rebuilding the core application. You also need safety: a buggy plugin should not crash the host. And you need flexibility: the plugin might need to call external APIs or access files the host doesn't touch.
hashicorp/go-plugin solves this by treating plugins as separate processes. The host spawns the plugin binary. They communicate over RPC using gRPC under the hood. The plugin runs in its own address space, so a panic in the plugin kills only the plugin. The host survives. The framework handles the handshake, connection setup, serialization, and error propagation. You define an interface, and the rest becomes plumbing.
Process isolation
Think of the host as a restaurant and the plugin as an external caterer. The restaurant takes orders from customers. When an order requires a special dish, the restaurant sends the request to the caterer via a secure line. The caterer prepares the dish and sends it back. The restaurant serves it. If the caterer's kitchen catches fire, the restaurant stays open. The caterer can use different ingredients or equipment than the restaurant. The only requirement is that both sides agree on the order format and the delivery protocol.
In Go terms, the host and plugin are two separate binaries. They share a common package that defines the interface and the handshake configuration. The host uses plugin.Client to start the plugin process. The plugin uses plugin.Serve to listen for connections. The framework negotiates a connection over a Unix socket or TCP, then exposes the interface as a local object. Calls to the interface travel over the wire, get serialized, execute in the plugin, and return the result.
Plugins are processes. Treat them like children: spawn them, feed them, and kill them when done.
Minimal example
Start with a shared interface. Both the host and the plugin must import this package. The interface defines the contract. The framework generates the RPC stubs automatically based on this interface.
Here is the shared interface with the handshake configuration. The handshake ensures the host and plugin are compatible before any calls happen.
// shared/plugin.go
package shared
import (
"context"
"github.com/hashicorp/go-plugin"
)
// ProtocolVersion must match between host and plugin.
const ProtocolVersion = 1
// Handshake identifies this plugin family and prevents loading incompatible binaries.
var Handshake = plugin.HandshakeConfig{
ProtocolVersion: ProtocolVersion,
MagicCookieKey: "BASIC_PLUGIN",
MagicCookieValue: "hello",
}
// Greeter defines the contract between host and plugin.
type Greeter interface {
// Greet returns a personalized message. Context must be the first parameter.
Greet(ctx context.Context, name string) (string, error)
}
// GreeterPlugin implements plugin.Plugin to register the Greeter interface.
type GreeterPlugin struct {
plugin.Plugin
Impl Greeter
}
// Plugin returns the interface implementation for RPC generation.
func (p *GreeterPlugin) Plugin(broker *plugin.MuxBroker) interface{} {
return p.Impl
}
The plugin binary implements the interface and calls plugin.Serve. This function prints handshake information to stdout, waits for the host to connect, and serves RPC requests.
// plugin/main.go
package main
import (
"context"
"fmt"
"log"
"github.com/hashicorp/go-plugin"
"myapp/shared"
)
// GreeterImpl implements shared.Greeter.
type GreeterImpl struct{}
// Greet returns the greeting. The framework serializes the return value.
func (g *GreeterImpl) Greet(ctx context.Context, name string) (string, error) {
return fmt.Sprintf("Hello, %s!", name), nil
}
func main() {
// ServeConfig registers the plugin and starts the RPC server.
plugin.Serve(&plugin.ServeConfig{
HandshakeConfig: shared.Handshake,
Plugins: map[string]plugin.Plugin{
"greeter": &shared.GreeterPlugin{
Impl: &GreeterImpl{},
},
},
})
}
The host binary uses plugin.Client to spawn the plugin. It waits for the handshake, retrieves the typed interface, and calls methods as if they were local.
// host/main.go
package main
import (
"context"
"fmt"
"log"
"os/exec"
"github.com/hashicorp/go-plugin"
"myapp/shared"
)
func main() {
// ClientConfig points to the plugin binary and defines the handshake.
client := plugin.NewClient(&plugin.ClientConfig{
HandshakeConfig: shared.Handshake,
Plugins: map[string]plugin.Plugin{
"greeter": &shared.GreeterPlugin{},
},
Cmd: exec.Command("./greeter-plugin"),
})
// Client() starts the process and waits for the RPC connection.
rpcClient, err := client.Client()
if err != nil {
log.Fatal(err)
}
// Kill ensures the child process terminates when the host exits.
defer client.Kill()
// Dispense retrieves the typed interface from the remote process.
raw, err := rpcClient.Dispense("greeter")
if err != nil {
log.Fatal(err)
}
greeter := raw.(shared.Greeter)
msg, err := greeter.Greet(context.Background(), "World")
if err != nil {
log.Fatal(err)
}
fmt.Println(msg)
}
Compile the plugin first, then run the host. The output is Hello, World!. The host spawned ./greeter-plugin, connected via RPC, called Greet, and received the result.
How the handshake works
When the plugin starts, plugin.Serve writes a JSON payload to stdout containing the protocol version, magic cookie, and available plugins. The host runs the command, captures stdout, and parses this payload. If the magic cookie or protocol version does not match the host's HandshakeConfig, the connection is rejected immediately.
This handshake prevents the host from accidentally loading a random binary as a plugin. It also catches version mismatches early. If you bump the ProtocolVersion in the shared package, the host and plugin must both be recompiled with the new version. Otherwise, the host refuses to connect.
The framework then negotiates a transport. By default, it uses a Unix domain socket on Linux and macOS, or a named pipe on Windows. The socket path is communicated during the handshake. Once connected, gRPC handles the RPC calls. The framework generates server and client stubs for your interface, so you don't write any serialization code.
The shared package is the contract. Break the contract and the plugin won't load.
The shared package trap
The most common mistake is assuming the interface is enforced at compile time across binaries. It is not. The host and plugin are separate programs. They compile independently. The only link is the shared package they both import.
If the host imports shared@v1.2 and the plugin imports shared@v1.1, and v1.2 adds a method to the interface, the host will try to call that method at runtime. The plugin's RPC server does not implement it. The call fails with a runtime error. The compiler cannot catch this because the plugin binary is already built.
Pin the shared package version in both go.mod files. Use the same commit or tag. When you change the interface, bump the ProtocolVersion and update both binaries. Treat the shared package like a library that requires a coordinated release.
Version drift kills plugins. Pin your shared dependencies.
Pitfalls and errors
Orphaned processes are the biggest runtime risk. If the host exits without calling client.Kill(), the plugin process continues running. It holds open the RPC connection and consumes resources. Use defer client.Kill() immediately after client.Client() succeeds. If the host crashes, the OS may reap the child, but relying on that is unsafe. Explicit cleanup is required.
The compiler rejects the program with undefined: shared if the import path is wrong in either binary. Double-check the module path. The compiler complains with cannot use raw (type interface{}) as shared.Greeter in assignment if you forget the type assertion after Dispense. The assertion is safe because Dispense returns the interface type registered in the plugin, but the compiler only sees interface{}.
Runtime errors appear as RPC errors. If the plugin panics, the host receives an error like rpc error: code = Unknown desc = plugin crashed. If the handshake fails, you get handshake error: magic cookie mismatch. If the protocol versions differ, the error is protocol version mismatch. These errors are verbose and actionable. Read them. They tell you exactly what went wrong.
Context propagation matters. The framework passes context.Context through RPC calls. If the host cancels the context, the plugin receives the cancellation and can stop work. Always put context.Context as the first parameter of interface methods. The framework expects it. If you put it second, the context won't propagate, and cancellation won't work.
Plugins are processes. Kill them or they linger.
When to use go-plugin
Use hashicorp/go-plugin when you need a robust plugin system with process isolation and automatic RPC generation. Use os/exec with JSON or text output when you only need to invoke an external tool and parse its result. Use a single binary with command-line flags when performance is critical and you don't need dynamic loading. Use raw gRPC when you need a full service mesh with load balancing and cross-language support beyond Go.
RPC is not local. Latency and serialization matter.