I've been running workloads on Kubernetes at work for years now, and probes are everywhere. Every Deployment has them (or should), every Helm chart ships its own opinion about them, and sooner or later every incident call has someone asking "wait, what's the liveness probe on this thing?". Add Kubestronaut prep on top of that, and lately I'm under the impression that I spend more time reading, tuning and troubleshooting probe configs than writing them.

So I wanted to find out whether I could build a neat, handy little tool for it. That tool is kubectl probes, and it's my first kubectl plugin.

This post isn't the pitch; the project page does that. This one is about how it got built: a week of reading someone else's code, vibe coding my own, a dashboard I got way too excited about and then deleted, and a few things about the Kubernetes API that look different once you have to write code against them.

A handful of numbers

A probe is a handler and five numbers. What those numbers add up to is never on your screen. periodSeconds: 30 with failureThreshold: 3 looks harmless. It's also up to 90 seconds of traffic going to a container that is already broken. Not that I'd know anything about that. 🙃

Here's what kubectl describe gives you for the Postgres pod in my demo cluster:

$ kubectl -n shop describe pod postgres-0 | grep -E "^ +(Liveness|Readiness):"
    Liveness:   exec [pg_isready -U shop] delay=45s timeout=30s period=20s successThreshold=1 failureThreshold=3
    Readiness:  exec [pg_isready -U shop -d shop] delay=0s timeout=3s period=5s successThreshold=1 failureThreshold=3

All the facts are there. What isn't there: that liveness waits 45 seconds before its first check, that its 30-second timeout is longer than its own 20-second period, and that once Postgres stops answering, nothing restarts it for a full minute. Now do that arithmetic for every container in a namespace, once per replica, with the Unhealthy events sitting in a separate section of every single pod.

It's not a hard problem, IMHO. It's a tedious one, which makes it a perfect job for a small tool.

The plugins I live in

I use a bunch of kubectl plugins every day, and the two at the top of the list are both by Ahmet Alp Balkan: kubectx (and its sibling kubens), which I've been using for 8 years now, and kubectl node-resource, for when I want to know how full my nodes actually are.

What I like about Ahmet's plugins is that they're clear, well scoped and efficient. They do one thing and they do it well. No config file, no TUI, no dashboard. You type a command, you get an answer, you move on. That was the bar I had in my head.

A week inside someone else's plugins

Before writing a single line of my own, I took a week to properly understand how these work. I used Matt Pocock's skills for Claude Code, especially /teach. If you haven't tried it: you give it a mission ("I want to understand how a kubectl plugin talks to the cluster, well enough to write one") and it turns a folder into a small course, with lessons, reference sheets, and learning records of what you actually understood, so every session starts where the last one left off.

I pointed it at kubectx and node-resource and went through both recursively, from main down to the last API call and back up to the output. Some of what I took away:

• Scope is a feature. kubectx is kubectx [name]: a few flags, one optional positional, nothing else. kubectl probes ended up shaped exactly like that: kubectl probes for a namespace, kubectl probes deploy/web for one workload. No subcommands to remember.
• Don't reinvent the flags. node-resource builds on k8s.io/cli-runtime, which gives you --context, --kubeconfig, -n, --as and friends for free, parsed exactly the way kubectl parses them. Nobody has to learn anything new to use your plugin.
• Borrow the good bits. node-resource can end its output with a summary, or print only that with --summary=only: a distribution with histograms, so you see the whole cluster at a glance. I liked it so much that kubectl probes has the same flag, the same six percentile rows, and the same kind of histogram. You'll see it further down.

Only once I felt like I could explain those two codebases to someone else did I start on mine.

Then I vibe coded it (with some paperwork)

Let's be honest: I vibe coded kubectl probes. Claude Code wrote most of the Go. ¯\_(ツ)_/¯

But I didn't want it to become the kind of vibe-coded project where nobody, including me, knows why anything is the way it is. So I leaned on Matt's skills again, and the repo has the paperwork to show for it:

• A glossary (CONTEXT.md) with the words the plugin uses and the ones it avoids. A workload is the top-most owner of a set of pods, a finding is an opinion and never a fact, and so on. Sounds pedantic, but it's why the output, the docs and the code all say the same thing.
• ADRs, 10 of them so far, for every decision I'd otherwise re-argue with myself a week later: the plugin is read-only, it trusts the running pod's spec over the template, a set of -f files is read as one cluster...
• Issues as tickets. Every piece of work started as a GitHub issue, small enough to review in one sitting.
• Golden-file tests that run the whole pipeline against fixtures, no cluster needed.
• A demo cluster in the repo, on kind or minikube, where every rule has something to fire on, plus a make chaos that breaks two probes on purpose. It was the release gate, and yes, it caught bugs before v0.1.0.

It was supposed to be a little tool. It didn't stay little for long, which brings me to...

The dashboard I built, and then deleted

I got ahead of myself. As I tend to do.

Sam Rose wrote a gorgeous interactive essay about probes for ngrok: a kubelet, pods, containers changing colour, every probe check pulsing across the screen. I read it and thought: what if kubectl probes had that, but for your real cluster? 🔥

So on a Sunday afternoon I vibed it. kubectl probes --serve started a local web server, streamed the report to the browser over Server-Sent Events, drew the pods and the kubelet as a diagram, and had a timeline you could scrub. It was fun to build. It even worked.

Then I ran it against the demo cluster mid-chaos, and it hit me: a real cluster doesn't give you those pulses. That essay runs on webernetes, a Kubernetes that lives in the browser, with a simulated cluster and a simulated clock. In a real one, a probe that passes leaves no trace in the API server at all, and a failing one shows up as a deduplicated, rate-limited event (more on that below). The only honest thing my dashboard could draw was what the terminal already printed, just slower and prettier. It showed exactly two things the CLI couldn't: which node the failures were on, and whether they bunched up in time.

Monday morning I took it out of the release and hid it behind a build tag, "just in case". Monday night I deleted it. About 6,000 lines, gone. There's even an ADR whose whole point is that there is no dashboard. 😅

I struggled with scoping on this one, and even more with sticking to the scope once I'd set it. What made me want a plugin in the first place was Ahmet's "one thing, done well", and I almost built the exact opposite. A browser surface is a second product. And hiding it behind a build tag "just in case" didn't make it free: it still cost CI time and attention on every single change, for something nobody would ever run.

What it looks like

Everything below is the demo cluster from the repo, running on minikube, right after make chaos pointed web's readiness probe and catalog's liveness probe at paths that answer 404.

Worst first

One row per container. Each probe column shows the handler and the one duration that probe is read for: the whole budget for startup, and for readiness and liveness the worst case between the container going bad and the probe acting on it. --sort severity puts what's failing at the top.

The row to look at is web. The Deployment asks for 2 replicas, and there are 3 pods: the 2 old ones, plus 1 new one the rollout added, which isn't ready. And there's a * on its readiness cell, which means the running pods no longer match their own template.

The rollout that stalled, quietly

This is my favourite one, because it's the sneakiest. Changing web's readiness path started a rollout. The new pod runs the new probe, fails it, and never becomes ready, so the rollout just... stops there. The two old pods keep serving on the old probe. Nothing restarts, nothing crashes. And kubectl says:

READY 2/2. The only hint is UP-TO-DATE 1.

It doesn't get better with time, either. Ten minutes in (progressDeadlineSeconds defaults to 600), the Deployment's Progressing condition flips to False with ProgressDeadlineExceeded, and that's it. Kubernetes takes no action on a stalled Deployment other than reporting that condition, so nothing rolls back, and get deploy keeps saying 2/2.

Naming the workload puts all the pieces side by side:

The * on the readiness handler and the drift table say what changed: /readyz in the running pods, /readyz-broken in the template. The runtime state names the one pod that isn't ready, and the failure evidence carries the kubelet's own message: status code 404, 124 times in 13 minutes.

Killed by liveness, "Completed, exit 0"

catalog's liveness probe fails 3 times in a row, the kubelet restarts the container, and the last termination reads... Completed, exit 0. That's because the kubelet stops a container that failed liveness the same way it stops any container: SIGTERM and a grace period. An app that shuts down cleanly exits 0. So nothing about how the container exited tells you why. The failure evidence right below it does: the liveness probe, and the 404 it got.

The other two findings were there before anything broke: without a readiness probe the container counts as ready the moment it starts, and its liveness probe gives it the default 1 second to answer.

The pod that looks perfectly healthy

postgres is ready and has never restarted. The findings are about its config, and I'll be honest: I've written this exact config before. More than once. Here are its probes from the manifest:

readinessProbe:
  exec:
    command: ["pg_isready", "-U", "shop", "-d", "shop"]
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 3
livenessProbe:
  exec:
    command: ["pg_isready", "-U", "shop"]
  initialDelaySeconds: 45
  periodSeconds: 20
  timeoutSeconds: 30
  failureThreshold: 3

Somebody measured that Postgres takes a while to come up and wrote that wait into the liveness probe: initialDelaySeconds: 45. That's the liveness probe doing the startup probe's job. It switches liveness off for the first 45 seconds of every start, even when Postgres comes up in a couple of seconds. And if a start ever runs long, say a slow recovery after a crash, liveness has its 3 failures within about 105 seconds (45 + 3 × 20) and restarts Postgres in the middle of it, which starts the slow recovery all over again.

The Kubernetes docs say it directly: if your container usually takes longer to start than initialDelaySeconds + failureThreshold × periodSeconds, add a startup probe that checks the same thing. Liveness and readiness don't run until the startup probe succeeds, so you can give startup a generous budget (failureThreshold × periodSeconds, say 5 minutes) and keep liveness tight for the rest of the container's life.

The second finding is the 30-second timeout on a 20-second period. The kubelet runs one check at a time and doesn't start the next one while the last is still going, so if a check ever does hang for the full 30 seconds, failure takes longer to detect than the 1m the numbers suggest.

The summary, a.k.a. my node-resource homage

This is the part I borrowed from node-resource. Coverage first: how many containers have each kind of probe. Then failure detection as percentiles, each naming its container, so the outlier is one command away. In this cluster the slowest liveness detection belongs to the control plane's own kube-scheduler, at 1m20s.

What I learned

None of the following are secrets, and I knew all of them from running clusters. But writing code against something is a different game: every "yeah, I know" turned into "okay, but what exactly happens when...".

A passing probe leaves no trace

The kubelet runs the probes, and it only tells the API server when something happens. A probe that passes leaves nothing you could ever kubectl get. A probe that fails becomes an Unhealthy Event, and Events are lossy in ways that matter once you build on top of them. Here's that same stalled web pod, about an hour into the chaos:

$ kubectl -n shop get events --field-selector reason=Unhealthy,involvedObject.name=web-58b7b6f6f9-n25b2 -o custom-columns=FIRST:.firstTimestamp,LAST:.lastTimestamp,COUNT:.count
FIRST                  LAST                   COUNT
2026-09-29T13:37:04Z   2026-09-29T14:32:04Z   678

A readiness probe failing every 5 seconds for 55 minutes, stored as one Event object with a counter. It gets better: client-go's event recorder, which the kubelet uses, rate-limits per object. It lets 25 events through, then writes at most once every 5 minutes, carrying the running count with it. That's why the screenshot above said "last seen 3m7s ago" for a probe that fails every 5 seconds. And then Events expire, after 1 hour by default (the API server's --event-ttl).

Careful

So the FAILURES column is recent history, not all history. Zero means nothing has failed recently, not that nothing has ever failed. The plugin's README says so in bold, because I'd want someone to tell me.

This is also what killed the dashboard, as you saw. You can't animate data that was never recorded.

Controllers create down, owners point up

You know the hierarchy: a Deployment has ReplicaSets, a ReplicaSet has Pods. What's worth being precise about is which way the API records that relationship. The controllers work top-down: the Deployment controller creates a ReplicaSet, the ReplicaSet controller creates Pods. But the link is written on the child, pointing up:

$ kubectl -n shop get pod web-58b7b6f6f9-n25b2 -o json | jq -c '.metadata.ownerReferences[] | {kind, name, controller}'
{"kind":"ReplicaSet","name":"web-58b7b6f6f9","controller":true}
$ kubectl -n shop get rs web-58b7b6f6f9 -o json | jq -c '.metadata.ownerReferences[] | {kind, name, controller}'
{"kind":"Deployment","name":"web","controller":true}
$ kubectl -n shop get deploy web -o json | jq -c '.metadata.ownerReferences'
null

Nothing on the Deployment object lists its ReplicaSets or its Pods. Every link lives on the child as an ownerReference, with controller: true on the one that manages it. It's the same pointer the garbage collector follows for cascading deletes: delete the Deployment, and everything whose owner chain leads back to it goes too.

For a tool

You walk up, not down. kubectl probes starts from the pods, because that's where the probes are actually running, and follows ownerReferences until there's no owner left: Pod → ReplicaSet → Deployment, Pod → Job → CronJob. Whatever sits at the top is what the plugin calls a workload.

Then come the edge cases: a pod with no owner shows up as pod/<name>, a custom owner like an Argo Rollout is named by its resource, and static pods, whose mirror pods are owned by their Node, show up as node/<name>. That's all the node/probes-demo rows in the summary above.

Why start from the pods and not the templates? Because mid-rollout they disagree, and that disagreement is exactly the * from the stalled rollout.

The API server fills in the blanks

Here's the readiness probe from that Postgres manifest again, as it lives on the running pod:

$ kubectl -n shop get pod postgres-0 -o json | jq '.spec.containers[0].readinessProbe'
{
  "exec": {
    "command": [
      "pg_isready",
      "-U",
      "shop",
      "-d",
      "shop"
    ]
  },
  "failureThreshold": 3,
  "periodSeconds": 5,
  "successThreshold": 1,
  "timeoutSeconds": 3
}

The manifest never set successThreshold. The API server wrote the default in when the pod was created, and it does that for every probe field except initialDelaySeconds. So on a running pod, "left unset" and "explicitly set to the default" look exactly the same.

That one changed a design decision. I wanted the detailed view to mark defaults as defaults, and on a live pod there's almost nothing left to mark. So the plugin shows a default in gray, like the 0s delays in the screenshots, and the JSON report still tells the two apart wherever it can, for example in a manifest read with -f.

A plugin is just a binary on your PATH

Any executable called kubectl-<something> on your PATH is a kubectl plugin. That's the whole contract:

#!/usr/bin/env bash
echo "hello from a kubectl plugin, context: $(kubectl config current-context)"

$ kubectl hello
hello from a kubectl plugin, context: probes-demo

Underscores in the name become dashes, which is why Ahmet's binary is kubectl-node_resource and you type kubectl node-resource. Everything else is convention: cli-runtime for the flags, goreleaser to build archives for 5 platforms, and krew to find, install and upgrade plugins. The mechanism took 2 lines to understand. The conventions around it are what took a week of reading.

What's next

v0.1.0 is out. Grab an archive from the releases, or:

go install github.com/mertdotcc/kubectl-probes@latest

It's also submitted to krew-index and waiting for review, so kubectl krew install probes should work soon. 👀

If you want to poke at it without pointing it at a real cluster, the repo ships the same demo cluster I used for every screenshot in this post:

cd hack/demo-cluster
make up          # or: make up TOOL=minikube
make chaos
kubectl probes -n shop

And I already have a couple more ideas for plugins, all aimed at real pain points of mine. Next up is kubectl svc-why: why is this Service returning 503, or showing no endpoints? It walks the request path and tells you where it breaks first. After that, there are ideas around CronJobs, startup time and NetworkPolicies sitting in my notes. Same rule for all of them: one thing, done well.

Wrapping up

I enjoyed this a lot. More than I expected, honestly.

Reading someone else's code slowly, with a tutor that makes you explain it back, is a completely different experience from skimming a repo. Vibe coding on top of that understanding felt less like gambling and more like pairing. And deleting 6,000 lines taught me more about scope than I'd like to admit.

If you try kubectl probes and it tells you something wrong, or fails to tell you something it should, open an issue. There's literally one called "Dogfood log: what kubectl probes failed to tell me on a real cluster", and I'd love your entries in it.

Further Reading & References

1. kubectl probes on GitHub, with the ADRs under docs/adr/
2. kubectx and kubens (Ahmet Alp Balkan)
3. kubectl node-resource (Ahmet Alp Balkan)
4. Skills for Real Engineers (Matt Pocock)
5. 🔥 Probes, the interactive essay (Sam Rose, ngrok)
6. Kubernetes Docs: Configure Liveness, Readiness and Startup Probes
7. Kubernetes Docs: Owners and Dependents
8. Kubernetes Docs: Extend kubectl with plugins
9. krew, the kubectl plugin manager

Kubernetes 1.35 finally makes in-place Pod resize a stable feature. [1] That might not sound exciting at first, until you remember how often “just change the resources” used to mean restarting Pods, breaking caches, or disrupting stateful workloads. Ask me how I know. 🙃

In-place resize changes that model. It lets Kubernetes adjust CPU and memory on a running Pod, sometimes without even restarting the container. Getting there took years of design work and several API revisions. In this post, I want to walk through that journey, explain how the feature actually works under the hood, and talk about where it genuinely helps and where it still does not.

History and Evolution of In-Place Pod Resize

In-place Pod resize did not appear fully formed. It went through several iterations before reaching a shape that fits Kubernetes’ scheduling and runtime model.

The core problem was clear early on. CPU and memory are properties of a running workload, but Kubernetes treated them as immutable Pod attributes. Changing them required recreating the Pod, even when the workload itself could have continued running safely. The challenge was making those resources mutable without breaking scheduling guarantees, node stability, or workload expectations.

Kubernetes 1.27: Alpha and first usable implementation

Kubernetes 1.27 introduced in-place Pod resize as an alpha feature. [2] It was disabled by default and explicitly marked experimental.

At this stage, the feature focused on the minimum viable capability. CPU and memory requests and limits could be updated on a running Pod without deleting it. That alone required changes across the API, kubelet, and container runtime layers.

The constraints were strict. Memory limits could only be increased, not reduced. Support was limited to certain container types. The feature required enabling the InPlacePodVerticalScaling feature gate manually. None of this was accidental. The goal was to validate the mechanics before committing to a stable API.

Alpha answered a narrow question: can the kubelet and runtime safely apply new resource limits to an existing container while keeping the Pod intact.

Kubernetes 1.33: Beta and API stabilization

In Kubernetes 1.33, in-place Pod resize graduated to beta and was enabled by default. [3] This was the point where the feature stopped being a proof of concept and started to look like something that could be relied on operationally.

Most of the important work in this phase was not about adding new capabilities, but about tightening the API surface and making resize behavior explicit, observable, and harder to misuse.

The beta release introduced major API changes and improvements:

• Dedicated resize subresource

  Resource updates must be performed through a new Pod subresource, resize, for example using kubectl patch pod <name> --subresource=resize. In earlier versions, resource changes were applied by patching the Pod spec directly, which blurred the line between immutable and mutable fields. Making resize a distinct subresource clarified intent and avoided accidental live resizes as a side effect of generic Pod updates.

• Pod Conditions for resize status

  Kubernetes shifted from an ambiguous status.resize field (used in alpha) to using Pod Conditions to track resize progress.

  Two conditions were introduced:

  1. PodResizePending, used when a resize request cannot be satisfied immediately, with reasons such as Infeasible or Deferred.
  2. PodResizeInProgress, used while the kubelet is actively applying resource changes.

  This change made resize behavior visible in a way that could be monitored and automated reliably.

• Support for sidecar containers

  Early implementations focused primarily on main containers. Beta added support for in-place resizing of sidecar containers as well. This was necessary for real-world deployments where sidecars are long-running and often resource-sensitive.

• Stability and kubelet hardening

  Much of the beta work focused on correctness under failure. Kubelet state tracking around resource updates was reworked to survive restarts using checkpoint files, reducing the risk of inconsistent state after node disruption. The Container Runtime Interface (CRI) was extended with more robust calls for updating container resources, closing several edge cases seen in earlier iterations.

• Performance improvements

  The Pod Lifecycle Event Generator (PLEG) was optimized to detect and apply resize events more quickly. This reduced the delay between a resize request being issued and the kubelet acting on it.

• Feature gate enabled by default

  The InPlacePodVerticalScaling feature gate was switched on by default in 1.33. While the feature was still beta, this signaled confidence that the design was stable enough for broad testing and non-production use without special cluster configuration.

Taken together, these changes set the shape of the API that GA would later stabilize. Alpha proved the idea. Kubernetes 1.33 made the behavior explicit and observable enough to use with confidence.

Kubernetes 1.35: GA and removal of key restrictions

Kubernetes 1.35 promoted in-place Pod resize to GA. At this point, the feature stopped being experimental or provisional and became a supported part of the platform for production clusters.

Most of the underlying mechanics were already in place by beta. The GA release focused on removing long-standing restrictions, tightening behavior under pressure, and improving operational visibility.

The jump from beta to GA included some important final updates:

• Memory limit reduction

  Earlier versions only allowed memory limits to be increased. Reducing a container’s memory limit in place was disallowed to avoid evictions and immediate OOM kills. Kubernetes 1.35 lifts this restriction. Memory limits can now be decreased in place, subject to best-effort safety checks. The kubelet will only apply a downward resize if the container’s current memory usage is below the new requested limit at the time of the operation. This does not eliminate risk entirely. A usage spike after the resize can still trigger an OOM kill. It does, however, make memory rightsizing possible without forcing a Pod restart.

• Priority-based handling of pending resizes

  When a node cannot satisfy all resize requests immediately, the kubelet now applies a defined ordering instead of handling them opportunistically. Resize requests are prioritized based on Pod priority class, QoS class, and how long the request has been deferred, with older requests handled first. This matters in clusters where resizes are frequent and node capacity is heavily utilized, and it avoids low-priority workloads blocking more critical ones.

• Pod-level resource API (alpha)

  Kubernetes 1.35 also introduced Pod-level resources as a new alpha feature. This is a separate, opt-in capability that allows resource limits to be specified at the Pod level, aggregating all containers, via spec.resources. Pod-level resources are gated behind the feature gates of PodLevelResources and InPlacePodLevelResourcesVerticalScaling and remain alpha in 1.35. While related, this is distinct from container-level resizing. Pod-level resources complement in-place resize by bounding total Pod resource usage, but they are still early. The rest of this article focuses on container-level resizing, which is stable and production-ready.

• Observability improvements

  The GA release added new events and metrics related to in-place resize operations. The kubelet now emits events when a resize is attempted, deferred, or fails, and exposes metrics that reflect resize activity. This closes an important gap for operators who need to understand why a resize did not happen, especially under node pressure.

Taken together, the GA release does not introduce a new model. It removes the last major constraints and makes the existing one safe to depend on. With that context in place, I will now try to show how in-place resizing actually works inside the control plane and on the node.

How In-Place Pod Resize Works Under the Hood

Historically, Kubernetes treated a Pod’s resource requests and limits as immutable once the Pod was running. Any change required destroying and recreating the Pod, often through a Deployment rollout. In-place Pod resize changes this by allowing CPU and memory resources to be updated on an existing Pod.

Making that possible without breaking scheduling guarantees or node-level isolation required coordinated changes across several parts of the system. The kubelet, the scheduler, and the container runtime all play a role. I'll go through the control flow and the responsibilities of each component during an in-place resize.

Resize workflow overview

At a high level, an in-place resize follows a predictable sequence. The Pod remains bound to the same node throughout, and no rescheduling occurs.

1. Resize request is issued

   A resize starts when a user or controller updates a Pod’s resource requirements through the resize subresource. This is an explicit action and distinct from patching the Pod spec directly.

   For example, we can increase a container’s CPU allocation with a command like:

   kubectl patch pod mypod --subresource=resize -p '
   {
     "spec": {
       "containers": [
         {
           "name": "app",
           "resources": {
             "requests": {
               "cpu": "2"
             },
             "limits": {
               "cpu": "2"
             }
           }
         }
       ]
     }
   }'
   

   The API server validates the request in the same way it validates other resource changes. It checks namespace quotas, limit ranges, and basic correctness. If the request is valid, the updated Pod spec is persisted to etcd.

   No new Pod is created, and the Pod is not rescheduled.

2. API server updates the Pod object

   Patching the resize subresource results in a new Pod spec generation with updated resource values. From the control plane’s perspective, this is a normal object update.

   The scheduler does not attempt to place the Pod again, since it is already assigned to a node. However, the scheduler will later take the updated resource intent into account when evaluating node capacity for other Pods.

   The kubelet running on the node hosting the Pod is notified of the change through the standard watch mechanism.

3. Kubelet evaluates node feasibility

   Once the kubelet observes the updated Pod spec, it determines whether the node can satisfy the new resource request.

   If sufficient CPU and memory are available, the kubelet proceeds with the resize. If not, the kubelet does not evict other Pods or take corrective action on its own.

   Instead, the Pod is marked with a PodResizePending condition. The condition includes a reason:

   • Infeasible if the requested resources exceed what the node can ever provide
   • Deferred if the request might become feasible later, for example if other Pods terminate

   While a resize is pending, the Pod continues running with its existing resource allocation. The kubelet periodically retries deferred resizes when node resources change. Multiple pending resizes are ordered by Pod priority class, QoS class, and how long the request has been waiting.

   These conditions are visible in the Pod status, which makes pending or blocked resizes observable to users and automation.

4. Kubelet applies resource changes through the container runtime

   When a resize can proceed, the kubelet applies the new resource values using the Container Runtime Interface (CRI).

   On Linux, this means updating cgroup parameters such as CPU quotas and memory limits for the affected container. The kubelet issues a call like UpdateContainerResources to the runtime, passing the new values.

   The runtime, whether containerd or CRI-O, updates the kernel cgroup configuration for the container. For CPU increases and most memory increases, this happens without stopping the container process.

   From Kubernetes 1.35 onward, the CRI also supports updating Pod sandbox resources, which improves behavior on some platforms and prepares the ground for better Windows support.

5. Container restarts when required

   Not all resource changes can be safely applied without restarting a container. Kubernetes exposes this explicitly through the per-container resizePolicy.

   Each resource type can specify one of two policies:

   • NotRequired, which applies the new cgroup limits without restarting the container
   • RestartContainer, which restarts the container to ensure the new limits take effect

   CPU resizes typically use NotRequired. Memory resizes are more nuanced. Some runtimes and applications, such as the JVM, do not adjust internal memory limits dynamically. Increasing memory without a restart may not be effective, and decreasing memory may be unsafe.

   When RestartContainer is specified, the kubelet applies the new limits and then restarts only the affected container. Other containers in the same Pod are not restarted. The Pod itself remains intact, including its IP address and volumes.

6. Pod status and scheduler accounting are updated

   As the resize progresses, Kubernetes updates the Pod’s status to reflect both intent and reality.

   Each container status includes resource information that shows the desired, allocated, and actual resources. While a resize is in progress, the PodResizeInProgress condition is set. Once the resize completes, the condition is cleared.

   If a container was restarted as part of the resize, its restartCount is incremented, which can be observed through standard tooling.

   The scheduler uses this status information to maintain safe cluster-level accounting. During a resize, it considers the maximum of desired, allocated, and actual resources when deciding whether new Pods can fit on the node. This prevents overcommit during the window where a resize has been requested but not yet fully applied.

7. Resize completes

   At this point, the Pod is running with its new resource allocation. No rescheduling occurred, no Pod identity changed, and no volumes were reattached.

   From the application’s perspective, the resize may be completely transparent, unless a container restart was required. Kubernetes emits events throughout this process, which can be used for monitoring and debugging.

The kubelet’s role and guarantees

The kubelet is responsible for enforcing in-place resizes while preserving node stability.

If a resize is deferred, the kubelet retries it when node resources change. Retry ordering respects Pod priority and QoS, which ensures that higher-priority workloads are not waiting behind lower-priority ones.

Resize state is checkpointed to disk. If the kubelet restarts mid-resize, it can recover the last known state and continue. This avoids leaving Pods in partially applied configurations.

The kubelet never violates node capacity. If resources are not available, the resize remains pending indefinitely. Kubernetes does not currently preempt other Pods to satisfy a resize request, even for high-priority workloads.

QoS class is fixed at Pod creation and cannot change as a result of resizing. A Guaranteed Pod must remain Guaranteed. A BestEffort Pod cannot gain requests. Resize requests that would violate these invariants are rejected.

Scheduler and control plane considerations

The scheduler does not move Pods to new nodes as part of an in-place resize. The operation is strictly local to the node hosting the Pod.

If a resize cannot be satisfied, the Pod remains where it is with its original resources. It is up to the operator or higher-level tooling to decide whether to delete and recreate the Pod elsewhere.

The API server’s role is limited to validation and persistence. It does not orchestrate the resize. Controllers such as Deployments and StatefulSets do not currently perform in-place resizes automatically. Updating a Pod template still results in new Pods being created according to the rollout strategy.

In-place resize is therefore typically driven by direct user action or by specialized controllers such as vertical autoscalers.

Limitations and caveats

Even as a GA feature, in-place Pod resize has clear boundaries:

• Only CPU and memory can be resized. Other resources such as GPUs, hugepages, ephemeral storage, and extended resources remain immutable after Pod creation.

• In-place resizing is fully supported on Linux nodes. Windows nodes do not support in-place resource updates as of Kubernetes 1.35.

• Pods using static CPU or memory manager policies cannot be resized in place, since those policies rely on fixed allocations.

• If a resize triggers a container restart, that container experiences a full stop-start cycle. Memory state is lost, and applications must tolerate the restart. Other containers in the Pod are unaffected.

Careful

Some runtimes adapt poorly to live memory changes. For example, JVM-based applications fix their max heap size using the -Xmx parameter at startup (often based on initial Pod limits). Increasing a Java Pod’s memory limit won’t automatically enlarge the heap without additional tooling; decreasing the limit could even be dangerous if the JVM is using close to the old max heap.

Similarly, Node.js may have an internal max memory setting (--max-old-space-size) that doesn’t change at runtime. Python and Go are generally more flexible. They will happily use more memory or CPU if available, making them good candidates for in-place adjustments. It’s important to test how your application behaves when resources are changed on the fly. If it doesn’t react well, you might be forced to use the RestartContainer strategy for safe resizing (or forego in-place resizing for that app).

Note · Init and ephemeral containers

Not for Init/Ephemeral Containers: Only regular app containers and sidecars are resizeable. Init containers run to completion before the Pod is running, so they are not subject to resizing (they’ve already executed). Ephemeral containers (debug containers) also cannot be resized; they’re considered special-case and short-lived.

• Finally, in-place resize does not allow changing a Pod’s QoS class. All resize operations must remain within the bounds of the original spec.

In practice, these constraints define where in-place resize is a good fit. When used with those limits in mind, it avoids Pod recreation in cases where restarting a workload would previously have been the only option.

API Design and Usage

From a user’s perspective, in-place Pod resize introduces a small set of API changes rather than a completely new workflow. Most of the complexity lives in the kubelet and scheduler. The API surface is intentionally narrow.

In this section I'll try to show what actually matters when enabling and using the feature in Kubernetes 1.35.

Feature gates and client requirements

The core feature gate for in-place Pod resize is InPlacePodVerticalScaling. As of Kubernetes 1.33, it is enabled by default and considered stable. On a brand new 1.35 cluster, no additional configuration is required.

If you are working with older clusters where the feature was disabled by default, the gate must be enabled consistently across the API server, controller manager, and kubelets. For 1.35 and later, this is only relevant when upgrading legacy environments.

Working with in-place resize also requires a reasonably recent kubectl version. Explicit support for the resize subresource was added in kubectl 1.32. Clients older than that will not recognize the subresource and will fail with validation errors. In practice, keeping kubectl aligned with your cluster version avoids most issues here.

Pod-level resources, introduced as an alpha feature in 1.35, are controlled by a separate feature gate (namely, PodLevelResources) and are not covered further in this section.

Pod spec changes: resizePolicy and status fields

In-place resize adds a small number of fields to the Pod API that control behavior and expose state.

resizePolicy on containers

Each container can declare how it should behave when a specific resource is resized. This is done through the resizePolicy field on the container spec.

Policies are defined per resource name. At the moment, valid resource names are cpu and memory. Each resource can specify one of two policies:

• NotRequired, which applies the new resource limits without restarting the container
• RestartContainer, which restarts the container so the new limits take effect cleanly If resizePolicy is omitted, the default is NotRequired for both CPU and memory.

Most workloads leave CPU as NotRequired. Memory is more application-specific. Runtimes that cannot safely adapt to changing memory limits often require RestartContainer to avoid undefined behavior.

A minimal example looks like this:

spec:
  containers:
  - name: nginx
    image: nginx:1.25.1-alpine
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      requests:
        cpu: "500m"
        memory: "256Mi"
      limits:
        cpu: "500m"
        memory: "256Mi"

In this configuration, CPU changes will be applied in place, while memory changes will trigger a container restart.

If a Pod’s overall restart policy is Never, all container resize policies must be NotRequired. Kubernetes will reject any resize that would violate the no-restart guarantee.

Resource state in Pod status

Once a Pod is running with in-place resize enabled, resource information appears in the container status as well as in the spec.

• spec.containers[].resources represents the desired state.
• status.containerStatuses[].resources reflects the resources actually applied by the kubelet.
• status.containerStatuses[].allocatedResources is used internally for scheduler accounting during transitions.

After a successful resize, the values in status.containerStatuses[].resources converge to match the spec. Watching this field is the most reliable way to tell when a resize has taken effect.

Resize progress is also surfaced through Pod Conditions. Two conditions may appear:

• PodResizePending, when a resize cannot be satisfied immediately
• PodResizeInProgress, while the kubelet is actively applying the change

Both include a reason and message that explain what is happening. These conditions are cleared once the resize completes or is resolved.

Performing a live resize

In-place resize always targets the resize subresource. Attempting to patch resource fields on the main Pod spec will be rejected, since those fields are normally immutable.

Patching a running Pod

The most direct way to resize a Pod is with kubectl patch and the resize subresource.

For example, to raise a container’s CPU request and limit from 700m to 800m:

kubectl patch pod resize-demo --subresource=resize -p '{
  "spec": {
    "containers": [{
      "name": "pause",
      "resources": {
        "requests": {"cpu": "800m"},
        "limits": {"cpu": "800m"}
      }
    }]
  }
}'

After issuing the patch, the Pod spec will reflect the new values immediately. The status will update once the kubelet applies the change. If no restart is required, the container’s restartCount will remain unchanged.

Editing in place

For ad hoc adjustments, kubectl edit pod <name> --subresource=resize can be used. This opens an editor with only the fields that are allowed to change. Saving the file sends the resize request.

This is convenient for manual tuning, but easy to misuse if the --subresource=resize flag is not used. Without it, resource fields will be read-only.

Applying from a manifest

In GitOps or CI workflows, resizing can be done with server-side apply:

kubectl apply -f updated-pod.yaml --subresource=resize --server-side

Only the mutable fields are merged. The manifest must not attempt to change immutable parts of the Pod spec.

It is important to note that higher-level controllers do not use this path by default. Updating a Deployment or StatefulSet template with new resources will still result in Pod replacement. To perform true in-place resizes, Pods must be targeted directly, or controlled by components that are resize-aware.

One notable exception is the Vertical Pod Autoscaler (VPA). When configured with an in-place update mode, it will attempt to resize Pods directly and fall back to recreation only when necessary.

Example end-to-end flow

To tie it all together, consider this Pod spec with resizable resources (After creating the Pod, CPU can be increased in place without interruption. Memory increases will trigger a restart of the app container only.):

apiVersion: v1
kind: Pod
metadata:
  name: resize-demo
spec:
  containers:
  - name: app
    image: registry.k8s.io/pause:3.8
    # Allow CPU to change without restart, but memory requires a restart
    resizePolicy:
    - resourceName: cpu
      restartPolicy: NotRequired 
    - resourceName: memory
      restartPolicy: RestartContainer
    resources:
      requests:
        cpu: "500m"
        memory: "200Mi"
      limits:
        cpu: "500m"
        memory: "200Mi"

1. Deploy the Pod: kubectl apply -f pod.yaml. The Pod starts with 0.5 CPU and 200Mi memory. It's in Guaranteed QoS (requests == limits) in this example.

2. Increase CPU in-place:

kubectl patch pod resize-demo --subresource=resize -p '{
  "spec": {
    "containers": [
      {
        "name": "app",
        "resources": {
          "requests": {
            "cpu": "800m"
          },
          "limits": {
            "cpu": "800m"
          }
        }
      }
    ]
  }
}'

This raises CPU to 0.8 cores. Kubelet on the node updates cgroups without restarting the container. Checking kubectl get pod resize-demo -o yaml after a couple seconds should show the CPU updated in both spec and status, and restartCount still 0.

3. Increase Memory (requires restart):

kubectl patch pod resize-demo --subresource=resize -p '{
  "spec": {
    "containers": [
      {
        "name": "app",
        "resources": {
          "requests": {
            "memory": "300Mi"
          },
          "limits": {
            "memory": "300Mi"
          }
        }
      }
    ]
  }
}'

This raises memory to 300Mi. Because app container’s memory resizePolicy is RestartContainer, the kubelet will gracefully stop and restart the app container to apply the new memory limit. After patching, watch the Pod’s status: it may for a moment go through a ContainerCreating or Running (restarting) phase for that container. Once done, status.containerStatuses[0].resources will show 300Mi and restartCount will have incremented by 1.

4. Handle Impossible Cases:

Just to test it out, when I patch an absurd amount of CPU cores to the app container in my resize-demo Pod, such as 1000 cores, the Pod will get a PodResizePending with reason Infeasible.

kubectl patch pod resize-demo --subresource=resize -p '{
  "spec": {
    "containers": [
      {
        "name": "app",
        "resources": {
          "requests": {
            "cpu": "1000" # Notice 1000 instead of 1000m
          },
          "limits": {
            "cpu": "1000"
          }
        }
      }
    ]
  }
}'

Check it out via k get po resize-demo -oyaml.

The condition message will say something like "Node didn't have enough capacity".

This is how we know the request can’t be satisfied on the node. We could then decide to revert the request or delete the Pod to reschedule it manually on a bigger node.

Best Practices for Using In-Place Resize

1. Use gradual, step by step adjustments and not crazy jumps: Don't dramatically increase resources in one go beyond what the node can handle. If you need to double or triple resources, consider ensuring the node has buffer or using cluster autoscaler to pre-provision capacity, otherwise your pod will sit in Pending state.

2. Monitor Pod Conditions: A script or pipeline that patches Pods could also watch for the PodResizePending condition to clear or to log why it's deferred.

3. Leverage Autoscalers: When possible, delegate resizing decisions to an autoscaler (Horizontal or Vertical Pod Autoscaler). The VPA in particular is becoming resize-aware. With the "Auto" mode (InPlaceOrRecreate) it will attempt an in-place update first and only fall back to recreating the Pod if absolutely necessary. This basically gives us automatic vertical scaling with minimal disruption.

4. Plan for Restarts (if needed!): If your app is not 100% dynamic, that's fine. You can still benefit by using restart-on-resize. For instance, a memory-intensive app like a database might require a restart to allocate a larger buffer pool. Doing that with in-place resize (with restart) still keeps the Pod consistent! Same identity, and it avoids the overhead of re-scheduling and re-attaching storage to that Pod (unlike a full Pod replace would). This is still a HUGE WIN.

5. Resource Quotas and Limits: Check your namespace's ResourceQuotas and LimitRanges. A resize is still technically a spec change. If it violates a quota (example: we try to give a Pod more CPU than allowed per quota), t will be denied by the API server just like a new Pod creation would. Make sure that any such policies are updated to accomodate the maximum sizes you intend to allow.

6. Testing: Test in-place resize with your specific workload in a staging environment. Watch how your application behaves, especially for memory downscaling. Some apps might need a tweak (like making Java respect cgroup limits for heap sizing, or making sure a Python app frees memory back to the OS -- I am planning on writing a technical deep-dive on JVM-based applications and "gotcha"s, tweaking the heap size, or playing with the -Xmx settings. If you think you would be interested in such a post, make sure to follow! 👀)

Now that I've discussed how to use the feature, I want to give some example scenarios where in-place Pod resizing is particularly useful.

Real-World Use Cases and Examples

In-place Pod resize unlocks new possibilities for operational efficiency, cost optimization, and responsiveness. Below are, IMHO, some of the most important use cases showing how Kubernetes users can leverage this feature. I'll cover examples in Java, Go, and Python applications, as well as how autoscalers and CI/CD processes can utilize vertical scaling without restarts.

• Startup Boost for Java Applications: Java services often require a CPU surge during startup (think JIT compilation, class loading, etc) but need far less CPU in steady conditions. Previously, what we would do is over-allocate CPU to avoid slow starts or rely on the VPA to kill and restart Pods with new limits; both are suboptimal. With in-place resize, we can deploy a Java app with a high CPU request initially and then scale it down in-place once startup completes. (See) For example, a Java webserver might start with 2 vCPUs for a fast bootstrap, and after a few minutes, an automated job or the VPA can reduce it to 0.5 vCPU without interrupting the service. This pattern (sometimes called "CPU burst" or startup boost) is now achievable seamlessly.

• Dynamic Web Services in Go and Python: Many stateless services written in Go or Python have varying CPU/memory needs depending on load. HPA can add more replicas, but if your service scales better vertically (or if you hit replica limits), in-place resize is like a Christmas present. Go and Python apps typically handle additional CPU or memory with no special tuning, meaning you can add resources to a live container and it will immediately take advantage of them. For example, a Python Flask API under spike load could be given an extra 1 GB of memory to expand in-memory caches instead of spawning new replicas. Or a Go service doing image processing might be granted an extra core during a batch of heavy requests, then scaled down later. Since neither Go nor Python have a fixed heap or similar, the app just experiences less GC pressure or faster execution when more resources become available. This yields a more elastic response to traffic bursts without overshooting on replicas (which can add latency while new pods start).

• Stateful Services and Databases 🔥: In-place resizing truly changes the game for stateful workloads that are costly to restart. Databases, caches, and stateful streaming apps can rarely be simply killed and relaunched without impact. Suppose you have a PostgreSQL Pod that suddenly needs more memory to execute a complex query or to load a larger working set into cache. We can now bump up its memory limit on the fly and let the query finish faster, then potentially scale it back down later. All of this happens with zero connection drops. The DB keeps running during the resize. Same story with Redis.

• Batch Jobs and AI/ML Workloads: Batch processing jobs or ML model training often have variable resource needs during their lifecycle. For instance, a TensorFlow training Pod might not need much CPU during data loading but then require a lot of CPU and memory during the training phase with a larger model. If such jobs run as long-lived Pods, we can now write a simple script that handles resources increases right when the heavy lifting begins. The opposite of this angle is also true: if a job finishes early or enters a less intensive phase, we could simply dial down its resources to make room on the node for others.

• Sidecar Proxy Scaling 🔥: In microservice architectures with service mesh sidecars (ie Envoy in Istio), the sidecar container might at times become a bottleneck. Instead of evicting the Pod or over-provisioning every sidecar for worst-case load, we can now resize the sidecar container's resources in-place when needed.

• Autoscalers Leveraging Vertical Resizing: I already touched on this earlier, but I can't emphasize this enough: the Vertical Pod Autoscaler (VPA) can now operate in a mode that uses in-place resize by default. In Kubernetes 1.35, VPA's UpdateMode: InPlaceIfPossible (beta) will try to patch running pods with new recommendations instead of evicting them. This means you can get automated vertical scaling with much less disruption. For workloads where horizontal scaling is not applicable, VPA with in-place updates can continuously right-size resources (up to set limits) to optimize usage.

These examples are just a few. I can go on and on (cost optimization for night/day workloads, CI/CD pipeline resource management, etc) about how flexible and powerful in-place Pod resizing can be. Whether it’s avoiding downtime for critical stateful apps, improving efficiency of autoscaling, or saving money by cutting down excess resources, the feature opens up new optimization opportunities. It’s important to always weigh the complexity: automated resizing introduces another layer of resource management logic. But I truly believe that for many scenarios the benefits far outweigh the costs.

Conclusion

The graduation of In-Place Pod Resize to GA in Kubernetes 1.35 is, IMHO, one of the (if not the) most exciting developments. We can now basically treat CPU and memory as elastic properties of a running Pod, which opens up a realm of new optimization strategies for both performance tuning and cost savings.

To recap, I covered:

• History (the journey from 1.27 -> 1.33 -> 1.35)
• Technical Deep-Dive
• API Usage
• Use Cases
• Performance

Looking ahead

The Kubernetes community is already discussing the next steps such as broader resource support (aka GPU resizing), smarter controller integration, and handling of edge cases like node pressure and preemption during resizes. The ecosystem around this feature is growing, and cloud platforms are beginning to incorporate it.

For now, if you manage Kubernetes workloads, give in-place Pod resizing a try on some non-critical workloads or in staging. Play with scaling pods up and down and see how your applications behave. With proper safeguards, you can then roll it out to production and reap the benefits of a more flexible and efficient cluster.

Further Reading & References

1. 💡 Kubernetes 1.35: In-Place Pod Resize Graduates to Stable (Natasha Sarkar)
2. Kubernetes Docs: Resize CPU and Memory Resources assigned to Containers
3. Kubernetes v1.33: In-Place Pod Resize Graduated to Beta (Tim Allclair)
4. Kubernetes 1.27: In-place Resource Resize for Kubernetes Pods (alpha) (Vinay Kulkarni)
5. KEP-1287: In-place Update of Pod Resources - (for those who want to read the design rationale and future considerations)
6. AEP-4016: Support for in place updates in VPA
7. AEP-7862: Support CPU Startup Boost in VPA
8. 🔥 Video: In-Place Pod Resize in Kubernetes: Dynamic Resource Management Without Restarts (Tim Allclair & Mofi Rahman)
9. Video: Unleash the Power of Inplace Pod Resource Resizing for Startup and Cost Optimization (Zhang Zhen & Yuxing Yuan)
10. Video: To Infinity and Beyond: Seamless Autoscaling with in-Place Resource Resize for Kubernetes Pods (Aya Ozawa & Kohei Ota)
11. Video: Resize Your Pods In-Place With Deterministic eBPF Triggers (Pablo Chico de Guzman & Vinay Kulkarni)
12. Real-world case study: Kubernetes Optimization Using In-Place Pod Resizing – Halodoc engineering blog on cost savings and implementation details of their resizing scheduler
13. In-place Pod resizing in Kubernetes: How it works and how to use it - Palark Blog
14. DoiT Engineering Blog: No Restarts, No Disruptions: Seamless Pod Resource updates with In-Place Resizing

Hey there! So I finally got around to updating my website and adding this blog section. Been meaning to do this for ages, but you know how it goes ... always something else that needs attention.

One of my main goals for 2026 is to flip the script on how I spend my time. Instead of endlessly scrolling and consuming content, I want to start creating more. And that includes sharing my thoughts and learnings here, not just the usual tech stuff from my day job.

I'm planning to write about my journey into design and art (still very much a beginner), woodworking projects that may or may not turn out as planned, 3D printing adventures and inevitable failures, running my home server (and all the rabbit holes that come with it), and various DIY electronics projects with Raspberry Pi, ESP32, and Arduino boards.

I'm also deep into Brazilian Jiu Jitsu these days, which has been humbling in the best way possible. Plus, I love traveling to places that aren't on every Instagram feed, you know, the places your mum doesn't want you to go to, or places that are simply unrecognized.

So yeah, that's what's coming. No promises on posting schedules or fancy content calendars (my ADHD and also procrastinating ass could never). Just me sharing stuff as I learn and build and explore. Should be fun.

Here's to creating more in 2026.