The S3 Client Pattern
You have a Go service that handles user uploads. The local disk fills up fast, backups are a nightmare, and scaling storage means buying more hardware. You need a reliable place to store objects that scales without you managing infrastructure. Amazon S3 is the standard for this. Go doesn't have a built-in S3 client, so you reach for the AWS SDK. The SDK is modular, type-safe, and handles the messy HTTP details for you.
S3 is a key-value store for blobs. You give it a bucket name and a key, and it stores the bytes. The AWS SDK for Go v2 is a collection of modules. You don't install one giant package. You import only the services you need. The SDK uses Go interfaces and structs to represent API calls. It handles authentication, retries, and serialization. You write Go code that looks like function calls, and the SDK turns it into HTTPS requests.
Trust gofmt. The SDK code follows standard Go formatting. Most editors run gofmt on save, so your code will match the community style automatically. Don't argue about indentation; let the tool decide.
Minimal Upload
Here's the simplest way to upload a file to S3 using the SDK. This script reads from standard input and writes to a bucket.
package main
import (
"context"
"fmt"
"os"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func main() {
// Load config from environment, shared config files, and EC2 metadata.
cfg, err := config.LoadDefaultConfig(context.Background())
if err != nil {
panic(fmt.Sprintf("failed to load config: %v", err))
}
// Create the S3 client using the loaded configuration.
client := s3.NewFromConfig(cfg)
// Upload standard input to the bucket specified in the environment.
_, err = client.PutObject(context.Background(), &s3.PutObjectInput{
Bucket: aws.String(os.Getenv("BUCKET_NAME")),
Key: aws.String("upload.txt"),
Body: os.Stdin,
})
if err != nil {
panic(fmt.Sprintf("failed to upload: %v", err))
}
fmt.Println("Upload complete")
}
The config.LoadDefaultConfig call scans for credentials. It checks environment variables like AWS_ACCESS_KEY_ID, then looks for a shared credentials file at ~/.aws/credentials, and finally checks instance metadata if running on EC2. This chain is the default behavior. You don't need to hardcode keys. The s3.NewFromConfig call builds a client that knows how to talk to the S3 endpoint. PutObject serializes the input struct into a request, signs it with your credentials, and sends it. The response comes back, and the SDK deserializes it. If the bucket doesn't exist or permissions fail, you get an error.
S3 clients are safe to share. Create one, keep it alive, and reuse it across requests. Creating a new client per request kills performance because the underlying HTTP transport cannot reuse connections.
Realistic Service Layer
Real applications need more than a script. You'll likely wrap S3 calls in a service layer, handle errors gracefully, and pass context for timeouts. This structure makes testing easier and keeps HTTP concerns separate from storage logic.
Here's the struct and constructor for an uploader service.
// S3Uploader encapsulates the S3 client and default bucket.
type S3Uploader struct {
client *s3.Client
bucket string
}
// NewS3Uploader initializes the uploader with a pre-configured AWS config.
func NewS3Uploader(cfg aws.Config, bucket string) *S3Uploader {
return &S3Uploader{
client: s3.NewFromConfig(cfg),
bucket: bucket,
}
}
The receiver name is usually one or two letters matching the type. (u *S3Uploader) is the convention. Avoid (this *S3Uploader) or (self *S3Uploader).
Here's the upload method that streams data to S3.
// Upload streams data to S3, respecting context cancellation.
func (u *S3Uploader) Upload(ctx context.Context, key string, reader io.Reader) error {
_, err := u.client.PutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String(u.bucket),
Key: aws.String(key),
Body: reader,
})
if err != nil {
// Wrap the error to add context about the key and operation.
return fmt.Errorf("upload to %s: %w", key, err)
}
return nil
}
The context.Context always goes as the first parameter, conventionally named ctx. Functions that take a context should respect cancellation and deadlines. The SDK checks the context before sending the request and cancels the operation if the context expires. Error wrapping with %w preserves the original error type so callers can use errors.Is or errors.As to inspect it.
Here's the HTTP handler that ties it together.
// HandleUpload attaches the upload logic to an HTTP endpoint.
func (u *S3Uploader) HandleUpload(w http.ResponseWriter, r *http.Request) {
// Pass the request context so the upload cancels if the client disconnects.
ctx := r.Context()
err := u.Upload(ctx, "uploads/data.bin", r.Body)
if err != nil {
http.Error(w, "upload failed", http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
Context is plumbing. Run it through every long-lived call site. If you drop the context, the upload might continue running after the client gives up, wasting resources.
Large Files and Multipart Uploads
The PutObject call works fine for small files. It fails or hangs for large files because the entire payload must be sent in a single request. Network interruptions cause the whole upload to restart. The SDK provides UploadManager to handle this automatically. It splits the file into parts, uploads them in parallel, and assembles the object on the server.
Here's how to use the upload manager for large files.
// UploadLarge uses the multipart upload manager for files over 16MB.
func (u *S3Uploader) UploadLarge(ctx context.Context, key string, reader io.Reader) error {
// UploadManager handles splitting, parallel uploads, and retries automatically.
manager := s3.NewUploadManager(u.client)
_, err := manager.Upload(ctx, &s3.PutObjectInput{
Bucket: aws.String(u.bucket),
Key: aws.String(key),
Body: reader,
})
if err != nil {
return fmt.Errorf("multipart upload to %s: %w", key, err)
}
return nil
}
The UploadManager detects the size and switches strategies. For small files, it falls back to a single PutObject. For large files, it uses multipart uploads. You don't need to calculate part sizes or manage part numbers. The manager handles the complexity.
Pitfalls and Conventions
The SDK has a few quirks that trip up new users. Understanding these saves debugging time.
The SDK uses pointers for optional fields. Go passes strings by value, but the SDK needs to distinguish between "not set" and "empty string". If you pass a string where a *string is expected, the compiler rejects this with cannot use string variable as *string value. Use the aws.String helper to wrap values. This is a common pattern in AWS SDKs. Don't pass a *string in your own code unless you need that distinction; strings are cheap to pass by value.
Credentials should never live in source code. Use environment variables or IAM roles. If you forget to set credentials, the SDK returns an error like no resolved credentials. The compiler won't catch missing environment variables at build time. The error appears at runtime when the first request is made.
The Body field expects an io.Reader. If you try to pass a string directly, the compiler complains with cannot use string variable as io.Reader value. Wrap strings in strings.NewReader. For bytes, use bytes.NewReader.
The SDK returns typed errors for specific failures. You can check for types.NoSuchBucket or types.NotFound using errors.As. This lets you handle missing buckets differently from permission errors.
var nf *types.NoSuchBucket
if errors.As(err, &nf) {
// Handle missing bucket specifically
}
Goroutine leaks happen when the goroutine waits on a channel that never gets closed. Always have a cancellation path. The S3 client itself doesn't leak goroutines, but if you wrap it in a custom pipeline, ensure channels are closed or contexts are cancelled.
The worst goroutine bug is the one that never logs. If you spawn a goroutine to upload asynchronously, capture the error and log it. Silent failures are harder to debug than loud ones.
Decision Matrix
Use s3.NewFromConfig when you need a standard S3 client with default settings and credential loading. Use s3.New with explicit options when you need to override the endpoint for testing or use a custom HTTP client. Use config.LoadDefaultConfig when your application runs in environments like EC2 or ECS where credentials are injected automatically. Use aws.Config passed explicitly when you are building a library that shouldn't assume how the caller manages credentials. Use context.Background for top-level scripts and context.WithTimeout for long-running operations to prevent hanging. Use UploadManager when files exceed 16MB or network conditions are unstable. Use PutObject directly when you need fine-grained control over headers or metadata for small files. Use io.NopCloser when wrapping a string or bytes slice to satisfy an io.ReadCloser requirement, though PutObject accepts io.Reader.
Credentials belong in the environment, not the repo. Wrap errors. Pass context. Reuse clients.