Your disk is full
Your Go service has been running for six months. It handles requests, processes data, and logs every error. One Tuesday morning, the monitoring alert screams: disk usage at 98%. You SSH in and find a single log file that's 40 gigabytes. tail -f takes ten seconds to load. vim crashes trying to open it. The application is still running, but every new log write is slow, and the disk is about to fill completely. When the disk fills, the application starts failing with no space left on device errors. Requests drop. The service is effectively dead, even though the process is still alive.
Log rotation prevents this. It splits a growing log file into manageable chunks based on size or time. Old logs get archived or deleted. The current log file stays small. Your disk stays healthy. Your tools stay fast.
The inode trap
Log rotation sounds simple: rename the file, start a new one. In Go, this approach fails silently.
The operating system manages files using inodes. An inode is a data structure that tracks the actual bytes on disk. A filename is just a directory entry pointing to an inode. When you call os.Open, the kernel allocates a file descriptor and links it to an inode. The process writes to the file descriptor. The kernel writes to the inode.
If you rename the file externally, you change the directory entry. The inode remains unchanged. The file descriptor still points to the same inode. The process continues writing to the data blocks associated with that inode. The file grows, but the new filename points to a different inode, which is empty.
This is the inode trap. You rename the log file, but Go keeps writing to the renamed file. The new file stays empty. The disk fills up anyway.
To fix this, the process must close the old file descriptor and open a new one. This forces the process to look up the filename again and get the new inode. Log rotation requires the application to reopen the file after rotation.
Rotation inside Go
The easiest way to handle rotation in Go is to use a library that manages the file reopening for you. The lumberjack package is the standard choice. It wraps a file writer and checks the file size on every write. When the size exceeds a limit, it rotates the file automatically.
Here's the simplest way to add rotation inside your Go code using lumberjack.
package main
import (
"log"
"os"
"gopkg.in/natefinch/lumberjack.v2"
)
func main() {
// Lumberjack handles rotation, compression, and cleanup.
// MaxSize triggers rotation at 100 megabytes.
// MaxBackups keeps 4 old files before deleting the oldest.
// Compress gzips old logs to save disk space.
l := &lumberjack.Logger{
Filename: "app.log",
MaxSize: 100,
MaxBackups: 4,
Compress: true,
}
// Wrap the lumberjack logger in a standard log.Logger.
// This lets you use log.Println while rotation happens behind the scenes.
logger := log.New(l, "", log.LstdFlags)
logger.Println("Application started")
}
When you write to the lumberjack logger, it intercepts the write call. It checks the current file size against MaxSize. If the file is under the limit, it writes the bytes normally. If the file hits the limit, lumberjack closes the current file, renames it to app.log.1, shifts existing backups down, and opens a fresh app.log. The application never stops writing. The file handle updates seamlessly.
If Compress is true, the renamed file gets gzipped in the background. This keeps disk usage bounded. The oldest backup gets deleted when the count exceeds MaxBackups. You can also set MaxAge to delete files older than a certain number of days, and LocalTime to use local time for file suffixes instead of UTC.
Lumberjack handles rotation. You handle the business logic.
Production with logrotate
In production, many teams prefer logrotate over in-process rotation. logrotate is a system tool that runs as a cron job. It manages log files for all services on the machine. It handles compression efficiently. It doesn't add dependencies to your binary. It provides a central place to configure retention policies.
Using logrotate requires your Go application to listen for a signal. logrotate sends a SIGHUP signal after it rotates a file. Your application catches the signal and reopens the log file.
Here's a production-ready logger that works with logrotate.
// Logger wraps a file handle and a slog logger.
// We store the file so we can close and reopen it later.
type Logger struct {
file *os.File
logger *slog.Logger
}
func NewLogger(filename string) (*Logger, func() error {
f, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
panic(err)
}
l := &Logger{file: f, logger: slog.New(slog.NewTextHandler(f, nil))}
// Reopen swaps the file handle.
// logrotate moves the file; this makes the process write to the new file.
reopen := func() error {
f.Close()
newFile, err := os.OpenFile(filename, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
return err
}
f = newFile
l.logger = slog.New(slog.NewTextHandler(f, nil))
return nil
}
return l, reopen
}
Go 1.21 introduced log/slog as the standard structured logging package. New projects should use slog instead of the legacy log package. slog supports attributes, levels, and handlers, making it easier to integrate with rotation writers. The community convention is to set a default logger early in main using slog.SetDefault. This ensures all packages using slog.Info write to the same destination without passing the logger everywhere.
The NewLogger function opens the file and creates a slog handler. It returns a reopen function. The reopen function closes the old file and opens a new one. It updates the logger to use the new file. This is the critical step that avoids the inode trap.
Here's how to wire up the signal handler.
func main() {
l, reopen := NewLogger("app.log")
slog.SetDefault(l.logger)
// Set up a channel to receive SIGHUP.
// logrotate sends this signal after it moves the log file.
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGHUP)
// Run the signal handler in a goroutine.
// This keeps the main goroutine free for application logic.
go func() {
for range sigChan {
slog.Info("Rotating log file")
if err := reopen(); err != nil {
slog.Error("Reopen failed", "error", err)
}
}
}()
// Application logic runs here.
// The signal handler runs concurrently in the background.
select {}
}
The signal handler runs in a separate goroutine. It listens for SIGHUP on a buffered channel. The buffer size of 1 prevents missed signals if the handler is busy. When a signal arrives, the handler calls reopen. If reopen fails, the error is logged. The application continues running.
You need a logrotate configuration file to trigger this. Create a file at /etc/logrotate.d/your-app with the following content.
/var/log/your-app/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
create 0640 your-user your-group
sharedscripts
postrotate
/usr/bin/killall -HUP your-app
endscript
}
This configuration rotates logs daily. It keeps 14 days of history. It compresses old logs. delaycompress delays compression of the most recent rotated file, giving you time to inspect it. missingok prevents errors if the log file doesn't exist. notifempty skips rotation if the file is empty. create sets the permissions and ownership of the new file. postrotate sends SIGHUP to your application after rotation.
Signals are safer than copytruncate. Trust the process to reopen.
Pitfalls
The most common bug is writing to a closed file. If your reopen logic has a race condition or fails, the next log call returns an error. The compiler won't catch this. At runtime, you'll see write /var/log/app.log: file already closed if the handle is invalid. Always check errors from reopen. If the file cannot be opened, log the error to stderr or a fallback writer so you know something is wrong.
Another trap is using copytruncate in logrotate. This directive copies the log file and then truncates the original. If your Go program writes a log line between the copy and the truncate, that line vanishes forever. The race window is small, but it exists. Signals are safer because the process controls the reopen. The file is moved, then the process reopens. No data is lost.
Permissions matter. If logrotate creates the new file with different permissions than your application expects, os.OpenFile might fail. Use create in the logrotate config to match your application's user and group. If your application runs as a non-root user, ensure the log directory is writable.
Goroutine leaks can happen if you spawn a goroutine for every log write or signal. The signal handler should be a single long-running goroutine. Don't create new goroutines inside the loop. Keep the handler simple.
The worst log bug is the one that disappears during rotation.
Decision
Use lumberjack when you want rotation logic inside your application and don't want to configure system tools. Use logrotate with signal handling when you run on Linux and prefer system-level management for all services. Use logrotate with copytruncate only when you cannot modify the application to handle signals, accepting the risk of lost log lines. Use manual rotation with os.Rename when you need custom rotation triggers based on application events rather than time or size.
Log rotation is plumbing. Configure it once, then forget it.