Controller runtime — handle resource deletion with predicate

When managing Kubernetes resources, it’s essential to handle resource deletion gracefully. For example, you may want to clean up some data before your resource is deleted. The data can be Kubernetes resources or data in external systems. If the cleanup is not done properly, you may have stale data in your system which will never be cleaned up.

If you write a controller using controller-runtime, you can filter events for your resource with predicate. Then in your reconcile function, you can reconcile the resource based on the event type. For example, you may have predicate like this:

import (
  ctrl "sigs.k8s.io/controller-runtime"
)

func (r *MyReconciler) Setup(mgr ctrl.Manager) error {
  return ctrl.NewControllerManagedBy(mgr).
    For(&corev1.Pod{}).
    WithEventFilter(predicate.Funcs{
     UpdateFunc: func(e event.UpdateEvent) bool {
      return false
     },
     DeleteFunc: func(e event.DeleteEvent) bool {
      return true
     },
     CreateFunc: func(e event.CreateEvent) bool {
      return false
     },
     GenericFunc: func(e event.GenericEvent) bool {
      return false
     },
    }).
    Complete(r)
}

In the above code, ctrl is the controller-runtime library. WithEventFilter filters event based on types so that thecontroller only handles the events it needs. In the above example, the controller only handles pod deletion since DeleteFunc returns true in predicate . The DeleteEvent has the deleted object:

DeleteFunc: func(e event.DeleteEvent) bool {
    log.V(1).Info("Delete event received for pod", "pod", e.Object)
    return true
},

In the reconcile function, you can do cleanup needed for the deleted pod.

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
  // do some cleanup
}

One caveat is, we cannot get the deleted object in the reconcile function because the object has been deleted before entering the reconcile function. It is by design because controller is supposed to reconcile resources based on the current state of the world, not by the event that triggers the reconciliation.

But what if you need the pod information to do some cleanup, e.g., you need pod information to query external resources and clean them up?

There are two solutions:

1. use finalizer

From the Kubernetes documentation:

Finalizers are namespaced keys that tell Kubernetes to wait until specific conditions are met before it fully deletes resources marked for deletion. Finalizers alert controllers to clean up resources the deleted object owned.

When you tell Kubernetes to delete an object that has finalizers specified for it, the Kubernetes API marks the object for deletion by populating .metadata.deletionTimestamp, and returns a 202 status code (HTTP "Accepted"). The target object remains in a terminating state while the control plane, or other components, take the actions defined by the finalizers. After these actions are complete, the controller removes the relevant finalizers from the target object. When the metadata.finalizers field is empty, Kubernetes considers the deletion complete and deletes the object.

Example code of removing finalizer:

import (
    "sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

func (r *MyReconciler) removeFinalizer(ctx context.Context, obj package.MyObject) error {
    // remove finalizer
    if controllerutil.ContainsFinalizer(&obj, name) {
        ok := controllerutil.RemoveFinalizer(&obj, name)
        return r.Update(ctx, &obj)
    }

    return nil
}

pros: use finalizer guarantees that we can finish cleanup before pod gets deleted.

cons: if our controller is down or the cleanup fails for some reason, it will block resource deletion. It worths to think about if the cleanup is crucial to risk blocking the resource deletion.

We also need to think about when and who should add the finalizer, and who should remove the finalizer. The complete life cycle of the resource should be well-defined to avoid race condition between multiple components if they need to handle events for the resource.

2. Use update event and DeletionTimestamp

We can watch on update event and do reconciliation based on that. The update event will be triggered before the delete event because Kubernetes will add DeletionTimestamp before deleting a pod.

func (r *MyReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
 var pod corev1.Pod
 if err := r.Client.Get(ctx, req.NamespacedName, &pod); err != nil {
  return ctrl.Result{}, client.IgnoreNotFound(err)
 }

if !pod.DeletionTimestamp.IsZero() {
  // do some cleanup
  ...
  ...
}

pros: decouples the resource deletion and the cleanup. If the cleanup fails for some reason, the controller will retry it and there is no impact to the resource deletion.

cons: since the resource deletion and cleanup are decoupled, if the cleanup fails, it may create stale data which should be cleaned up along with the deleted resources. We may need extra garbage collection mechanism to clean up those stale resources.

Conclusion

In conclusion, each approach has advantages and disadvantages. One guiding principle is that, if it is mandatory to clean up all related resources in order to maintain the correctness of the functionality, we should use finalizer. Otherwise, use update event can be a reasonable workaround.

Happy coding with controller!