The gatekeeper pattern
You have a cluster running dozens of microservices. Someone deploys a new pod without the required cost-center label. The billing system breaks. You could write a controller that watches every pod, finds the missing label, and deletes the pod. That works, but it wastes API calls and creates a race condition where the pod runs for a few seconds before getting torn down. Kubernetes gives you a better tool: admission webhooks. You intercept the CREATE or UPDATE request at the API server level, check the payload, and reject it before it ever touches etcd.
Think of the Kubernetes API server as a busy train station. Every resource request is a passenger trying to board. Admission webhooks are the ticket inspectors standing at the gate. They do not move the trains. They just check the ticket. If the ticket is valid, the passenger boards. If the ticket is missing a stamp, the inspector turns them away. You can also use a mutating webhook to hand the passenger a corrected ticket before they step through the turnstile. The API server pauses its normal flow, calls your HTTPS endpoint, waits for your verdict, and then proceeds.
Webhooks are just HTTP servers that speak a specific Kubernetes protocol. You do not need a special framework. You need a standard net/http handler, a JSON decoder, and a valid TLS certificate. The rest is business logic.
How the protocol actually works
Kubernetes wraps every webhook interaction in an AdmissionReview object. The API server sends a POST request containing a JSON payload. Inside that payload sits a Request field. The Request holds the operation type (CREATE, UPDATE, DELETE), the resource kind, the namespace, and the raw JSON of the object being submitted. Your handler reads the Request, evaluates it against your rules, and writes a verdict into the Response field. You then send the entire AdmissionReview back. The API server extracts Response.Allowed and either commits the object to etcd or returns an error to the user.
TLS is mandatory. Kubernetes will not talk to your webhook over plain HTTP. The API server verifies your server certificate against a CA bundle you provide during deployment. If the certificate chain is broken, the API server drops the request and logs a TLS handshake failure. You must also handle timeouts. Kubernetes expects webhooks to respond quickly. The default timeout is ten seconds. If your handler blocks longer than that, the API server kills the connection and rejects the user request.
Minimal webhook server
Start with the HTTP contract. Kubernetes sends a POST request containing a JSON payload. Your job is to read it, decode it into a known structure, and send back a JSON response. The protocol wraps everything in an AdmissionReview object. The request lives inside review.Request. Your answer goes into review.Response.
Here is the simplest working handler that accepts every request and returns an allowed verdict:
package main
import (
"encoding/json"
"io"
"net/http"
"k8s.io/api/admission/v1"
"k8s.io/apimachinery/pkg/runtime"
)
// decoder translates raw JSON into Kubernetes Go types
var decoder = admission.NewDecoder(runtime.NewScheme())
// HandleAdmission processes incoming webhook requests from the API server
func HandleAdmission(w http.ResponseWriter, r *http.Request) {
// Kubernetes only sends POST requests to webhooks
if r.Method != http.MethodPost {
http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
return
}
// Read the raw payload sent by the API server
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, "failed to read body", http.StatusBadRequest)
return
}
// Unmarshal the JSON into the AdmissionReview struct
var review v1.AdmissionReview
if err := decoder.Decode(r, body, &review); err != nil {
http.Error(w, "decode failed", http.StatusBadRequest)
return
}
// Create a default allowed response
response := v1.AdmissionResponse{Allowed: true}
review.Response = &response
// Send the review back with the response attached
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(review)
}
func main() {
// Listen on port 8443 with TLS enabled
http.HandleFunc("/validate", HandleAdmission)
http.ListenAndServeTLS(":8443", "tls.crt", "tls.key", nil)
}
The handler reads the body, decodes it, attaches a response, and encodes it back. Notice the explicit error checks. The Go community accepts the if err != nil boilerplate because it forces you to handle the unhappy path. Skipping error checks in a webhook means the API server receives a malformed response, which triggers a cluster-wide rejection. Trust the compiler. Argue logic, not formatting.
Walking through the request lifecycle
When a user runs kubectl apply -f pod.yaml, the API server intercepts the request. It checks its internal admission controllers first. If those pass, it looks at your ValidatingWebhookConfiguration. The configuration tells the API server which URL to call, which resources to watch, and which CA to trust. The API server constructs an AdmissionReview, serializes it to JSON, and sends it over HTTPS.
Your handler receives the request. The decoder.Decode call uses the Kubernetes runtime scheme to map JSON fields to Go struct fields. The review.Request.Object.Raw byte slice contains the actual pod or deployment JSON. You unmarshal that slice into a typed struct like corev1.Pod. You run your validation rules. You set review.Response.Allowed to true or false. If you reject the request, you populate review.Response.Result with a metav1.Status object containing a message and a reason code. You attach the response to the review and encode it.
Context handling matters here. The API server does not pass a Go context.Context in the HTTP request headers. You create one yourself if you need cancellation or deadlines. Always pass context.Context as the first parameter to any function that performs I/O or calls downstream services. Name it ctx. Respect deadlines. If your validation logic calls an external database, attach a timeout to the context so a slow query does not block the webhook handler.
Realistic validation logic
A webhook that always returns Allowed: true is useless. You need to inspect the resource and enforce rules. Here is a handler that rejects pods missing a required team label:
import (
"encoding/json"
"k8s.io/api/admission/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// ValidatePod checks that every new pod has a required team label
func ValidatePod(review *v1.AdmissionReview) *v1.AdmissionResponse {
// Extract the raw object from the request
obj := review.Request.Object.Raw
var pod corev1.Pod
if err := json.Unmarshal(obj, &pod); err != nil {
return &v1.AdmissionResponse{
Allowed: false,
Result: &metav1.Status{
Message: "failed to unmarshal pod",
Reason: metav1.StatusReasonBadRequest,
},
}
}
// Check for the required label
if _, exists := pod.Labels["team"]; !exists {
return &v1.AdmissionResponse{
Allowed: false,
Result: &metav1.Status{
Message: "pod missing required 'team' label",
Reason: metav1.StatusReasonForbidden,
},
}
}
// Label exists, allow the request
return &v1.AdmissionResponse{Allowed: true}
}
You call ValidatePod from your HTTP handler, attach the returned response to review.Response, and encode it. The metav1.Status struct gives the API server structured error information. The Reason field maps to Kubernetes error codes. Using StatusReasonForbidden tells kubectl to display a clean access denied message instead of a generic server error.
Deploy the webhook with a ValidatingWebhookConfiguration. The resource tells the API server where to send requests and which CA to trust.
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
name: pod-team-label-webhook
webhooks:
- name: validate-team-label.example.com
clientConfig:
service:
namespace: default
name: webhook-service
path: /validate
caBundle: LS0tLS1CRUdJTi... # base64 encoded CA cert
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
admissionReviewVersions: ["v1"]
sideEffects: None
The sideEffects field is required. Set it to None if your webhook only reads data. Set it to NoneOnDryRun if it might write to external storage but should skip writes during kubectl --dry-run. The API server uses this field to optimize request routing.
Common failure modes
Webhooks fail in predictable ways. The API server logs are your first debugging tool. If you forget to mount the CA bundle correctly, the API server rejects the request with x509: certificate signed by unknown authority. The webhook never receives the call. Fix the caBundle field in your configuration.
If your handler panics, the HTTP server returns a 500 status. The API server treats any non-200 response as a rejection. You will see admission webhook "validate-team-label.example.com" denied the request in the user output. Wrap your handler logic in a recover block if you suspect third-party libraries might panic, but prefer explicit error returns. The compiler complains with assignment to entry in nil map if you forget to initialize pod.Labels before reading it. Always check for nil maps.
Timeouts are the most expensive bug. If your validation logic queries an external database without a deadline, the request hangs. Kubernetes kills the connection after ten seconds. The user sees a timeout error. The API server marks your webhook as unhealthy and may stop calling it until the configuration is reloaded. Attach a context deadline to every external call. Return a clear error message if the deadline passes.
Forgetting to set review.Response causes a silent failure. The API server receives a review with a null response field. It rejects the request and logs admission response is nil. Always assign the response before encoding. The compiler will not catch this mistake at build time. Runtime checks or unit tests are required.
When to use a webhook
Use a validating webhook when you need to enforce a policy before a resource is created or updated. Use a mutating webhook when you need to inject defaults, sidecars, or annotations into a resource automatically. Use a Kubernetes controller when you need to react to state changes after the resource already exists in etcd. Use OPA Gatekeeper or Kyverno when you want declarative policy-as-code without writing custom Go handlers. Use plain sequential code when you do not need concurrency: the simplest thing that works is usually the right thing.
Webhooks run on the critical path of every API request. They must be fast, reliable, and idempotent. Do not use them for heavy computation. Do not use them for long-running background jobs. Keep the handler thin. Delegate complex logic to separate services if necessary. The worst webhook bug is the one that silently times out and blocks deployments.