The integration problem
You have a Go microservice handling requests with low latency. The data team hands you a Python script that runs a machine learning model, or a legacy data pipeline written in Python five years ago. You need to bridge the gap. Go handles the network and concurrency; Python handles the math and the legacy code. How do you make them talk without rewriting everything?
There are two distinct ways to connect Go and Python. The first is the subprocess approach. You treat Python as an external program. Go spawns a new process, waits for it to finish, and reads the output. This is like sending an email to a colleague: you write the message, hit send, and wait for a reply. The processes are isolated. If one crashes, the other survives.
The second approach is embedding Python inside Go using cgo. This loads the Python interpreter into the same memory space as your Go program. You can call Python functions directly from Go code. This is like moving in with that colleague. You share resources and talk instantly, but you also share the risk. If the Python interpreter segfaults, your Go program goes down with it.
Subprocess: the safe boundary
The subprocess approach is the standard way to integrate Python into Go. You use the os/exec package to run the Python executable as a child process. This method is robust, easy to debug, and keeps the Python runtime isolated from your Go service.
Minimal example
Here's the simplest way to run a Python script from Go. You create a command, execute it, and capture the result.
package main
import (
"fmt"
"os/exec"
)
func main() {
// Run python3 with the script argument.
// CombinedOutput captures stdout and stderr together.
cmd := exec.Command("python3", "script.py")
output, err := cmd.CombinedOutput()
// Check for errors. The process might exit with non-zero status.
if err != nil {
// Print the error and the output for debugging.
fmt.Printf("Error: %v\nOutput: %s\n", err, output)
return
}
// Print the result.
fmt.Println(string(output))
}
The exec.Command function prepares the execution but doesn't run it yet. You get a *exec.Cmd struct that you can configure. Calling CombinedOutput starts the process, waits for it to terminate, and returns both standard output and standard error as a single byte slice. If the Python script exits with a non-zero status code, err will be non-nil. The error message usually includes the exit status. You can also use Output if you only want stdout, or Start and Wait if you need to stream output in real-time.
Go developers write if err != nil explicitly. It's verbose, but it forces you to handle the failure case. In the example above, we check the error immediately. This pattern makes the unhappy path visible. You can't accidentally ignore a failure.
Realistic example with JSON and Context
Real-world integration rarely involves just printing text. You usually need to pass arguments and parse structured data. JSON is the standard bridge. You also need to handle timeouts so a slow Python script doesn't block your Go service forever.
Here's a realistic pattern: marshal data to JSON, pass it as an argument, run with a context timeout, and unmarshal the response.
package main
import (
"context"
"encoding/json"
"fmt"
"os/exec"
"time"
)
// Payload represents the data sent to Python.
type Payload struct {
Name string `json:"name"`
Age int `json:"age"`
}
func main() {
// Marshal the struct to JSON bytes.
data, _ := json.Marshal(Payload{Name: "Alice", Age: 30})
// Create a context with a 5-second timeout.
// This prevents the Go program from hanging if Python is slow.
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
// Use CommandContext to link the command to the context.
// The command will be killed if the context expires.
cmd := exec.CommandContext(ctx, "python3", "process.py", string(data))
output, err := cmd.Output()
if err != nil {
fmt.Println("Script failed:", err)
return
}
// Unmarshal the JSON response from Python.
var result map[string]interface{}
json.Unmarshal(output, &result)
fmt.Println(result)
}
The context package is plumbing. Run it through every long-lived call site. exec.CommandContext attaches the context to the command. If the timeout fires, the context is cancelled, and the child process receives a signal to terminate. This is how you prevent goroutine leaks caused by blocked subprocess calls.
The context.Context always goes as the first parameter in Go functions, conventionally named ctx. While exec.CommandContext takes the context as the first argument, the convention holds for your own wrapper functions too. Functions that take a context should respect cancellation and deadlines.
Python calling Go
Python calling Go works the same way. You compile your Go code into a binary, and Python uses the subprocess module to run it. Go binaries are statically linked by default, so they run anywhere without dependencies. This makes them excellent tools to call from Python scripts.
import subprocess
# Run the Go binary and capture output.
# capture_output=True collects stdout and stderr.
result = subprocess.run(["./my-go-binary"], capture_output=True, text=True)
# Check the return code.
if result.returncode != 0:
print("Go binary failed:", result.stderr)
else:
print(result.stdout)
Go produces a single executable file. You don't need to manage a virtual environment or install packages on the host. Just copy the binary and run it. This is a major advantage when distributing Go utilities to Python users.
Cgo: embedding the interpreter
Embedding Python puts the interpreter inside your Go process. You use cgo to call the Python C API. This gives you direct function calls without process overhead, but it introduces complexity. You must manage the Python Global Interpreter Lock (GIL) and handle memory carefully.
Here's the basic structure for embedding Python. You initialize the interpreter, run code, and finalize.
package main
/*
#include <Python.h>
*/
import "C"
import (
"fmt"
"unsafe"
)
func main() {
// Initialize the Python interpreter.
// This must be called before any other Python APIs.
C.Py_Initialize()
// Ensure cleanup happens when main returns.
defer C.Py_Finalize()
// Create a C string for the Python code.
code := C.CString("print('Hello from embedded Python')")
// Free the C string memory when done.
defer C.free(unsafe.Pointer(code))
// Execute the string. Returns 0 on success.
if C.PyRun_SimpleString(code) != 0 {
// Handle Python errors via C API if needed.
fmt.Println("Python execution failed")
}
}
The cgo directive /* #include <Python.h> */ tells the C compiler to include the Python header file. The import "C" line binds the C symbols to Go. You can then call C.Py_Initialize and other Python C API functions.
This approach requires the Python development headers installed on the system. If they are missing, the compiler rejects the program with undefined: C.Py_Initialize. You need python3-dev on Debian/Ubuntu or python3-devel on RHEL/CentOS.
The GIL and goroutines
The Python Global Interpreter Lock (GIL) allows only one thread to execute Python bytecode at a time. Go's scheduler runs goroutines on multiple OS threads. If you call Python from a goroutine, you must acquire the GIL before calling Python APIs and release it afterward. Failing to do so causes deadlocks or crashes.
Use PyGILState_Ensure and PyGILState_Release when calling Python from Go goroutines. These functions handle the GIL state safely. Every call to the Python C API must happen while the GIL is held. This adds overhead and complexity. You must ensure the GIL is released before returning to Go code that might block or schedule other goroutines.
Cgo calls cross the Go/C boundary, which has overhead. Every call pauses the Go scheduler. This approach is best for long-running Python sessions where you call many functions, not for one-off scripts. The process spawn overhead of os/exec is often acceptable for simple tasks, and it avoids the GIL entirely.
Pitfalls and errors
Integration across languages introduces failure modes that don't exist in pure Go.
If Python isn't in your system path, the compiler won't catch it. The runtime panics with exec: "python3": executable file not found in $PATH. You must ensure the environment where Go runs has Python installed. Use exec.LookPath to check for the executable before running the command.
Another common issue is working directories. exec.Command inherits the working directory of the Go process. If your Python script uses relative paths, it might fail. Use cmd.Dir to set the working directory explicitly. This ensures the Python script runs in the expected location.
When using cgo, memory management is shared. If Python allocates memory and Go frees it, or vice versa, you get undefined behavior. Stick to the Python C API for memory management when embedding. Don't mix C.malloc and C.PyMem_Malloc unless you know exactly what you are doing.
Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Always have a cancellation path. When using os/exec, ensure you call Wait or use CombinedOutput to reap the child process. If you use Start without Wait, the process becomes a zombie.
Decision matrix
Use os/exec when you need isolation. The Python script runs in a separate process, so a crash in Python doesn't kill your Go service. Use os/exec when the task is short-lived or stateless. Spawning a process has overhead, but for simple tasks, the safety is worth it. Use os/exec when you want to leverage Python's virtual environments. Each subprocess can run with its own dependencies. Use cgo embedding when you need high-frequency calls. If you call Python thousands of times per second, the process spawn overhead becomes a bottleneck. Use cgo embedding when you need to share memory or state directly between Go and Python. Use Python calling Go via subprocess when Go provides a utility function. Compile Go to a binary and call it from Python for performance-critical subroutines. Use plain Go when you can rewrite the logic. The best integration is often no integration. If the Python code is simple, port it to Go to eliminate the boundary entirely.
Subprocess keeps the blast radius contained. If Python segfaults, Go lives. Cgo merges the memory spaces. A Python crash takes Go down with it. Context is the kill switch. Always pass it to long-running commands.