Skip to main content

Known Issues

What the operator does not do for you, what is bounded, and what is not configurable. Every symptom has a runbook in Common Issues & Solutions.

No Domain Metrics​

The chart exposes controller-runtime metrics and nothing else: no gauge for "machines up", no counter for "scale-ups this hour", no pool panel. The pool's state is on the objects:

kubectl get groups -n homelab-autoscaler-system # SLOTS ON OFF LASTSCALE HEALTH
kubectl get nodes.infra.homecluster.dev -n homelab-autoscaler-system # POWERSTATE PHASE GROUP ZONE SINCE ON24H
kubectl get nodeclaim -n homelab-autoscaler-system -o wide # PHASE ATTEMPT REASON DEADLINE

prometheus.enable: true renders a ServiceMonitor over the manager's metrics endpoint (needs cert-manager, metrics.enable: true, and the Prometheus Operator CRDs). Alert on the objects: a NodeClaim in booting past spec.deadlineAt, an activationsLast24H that says the pool is churning, a Group whose status.health is offline while its status.slots is not 0.

Error Recovery Is Bounded, Not Manual​

Every failure path ends by itself. Nothing needs a restart, a re-arm, or a status field written back.

BoundValueWhat happens when it is hit
A claim's boot budget12m10s, floored by the Group's spec.maxNodeProvisionTimefailed, reason: DeadlineExceeded, and the machine is force-shut-down
Consecutive boot attempts, per slot3failed, reason: AttemptLimitExceeded: no machine is bought until an attempt lands
A stuck lifecycle transition15m (fsm.StuckTransitionThreshold)ForceCleanup; the claim ends failed, reason: StuckTransition
A shutdown that will not land3 Jobs, then a 30m pause (claim.ShutdownAttemptResetGrace)the controller stops firing the same script at the same machine, then fires it again
A ready claim whose machine is goneone boot window with the machine observed offfailed, reason: MachineGone, freeing the slot
A terminal claimspec.ttlSecondsAfterFinished, default 3600scollected; the story survives in Node.status.lastClaim

Manual action is what is left when the bound is not the fix: a spec.shutdownPodSpec whose script exits 0 without powering the machine off, a CNI that never clears the keys in the Group's spec.nodeReadinessTaints, and two slots declaring the same spec.kubernetesNodeName (admission refuses the second and names the holder).

What Is Not Configurable​

Constants, and the numbers the bounds above are built from: the idle resync of a settled slot (5m), the lifecycle Job monitor poll (30s), the readiness gate (4m), the registration/uncordon window (90s), the attempt bound (3), the shutdown streak (3 attempts, 30m reset grace), and the boot budget: 12m10s, which is the startup Job's own timeout (5m) plus the Job monitor poll (30s) plus the registration wait (90s) plus the readiness gate (4m) plus the uncordon retry ceiling (10s) plus a 1m margin. Raise the budget only through the Group's spec.maxNodeProvisionTime, which lifts the floor of every claim's spec.deadlineAt.

The knobs are the fields of the objects: the Group's seven spec fields (scaleDownUtilizationThreshold, scaleDownGpuUtilizationThreshold, scaleDownUnneededTime, scaleDownUnreadyTime, maxNodeProvisionTime, zeroOrMaxNodeScaling, ignoreDaemonSetsUtilization) plus minSize and nodeReadinessTaints; the slot's (startupPodSpec / shutdownPodSpec with their serviceAccount, priorityClassName and ttlSecondsAfterFinished, nodeCapacity, nodeLabels, nodeAnnotations, nodeTaints, taintsToRemove, kubernetesNodeName, pricing); and the claim's (requestedAt, deadlineAt, ttlSecondsAfterFinished, requestedBy, request). A scale-down window is a Group number: the pool is what cluster-autoscaler sizes, not the machine.

Symptoms To Runbooks​

SymptomRunbook
A claim is stuck pendingA Claim Is Stuck Pending. A hand-made claim with no spec.deadlineAt never expires: stamp the budget
A claim is failedA Claim Failed, and its reason table
"one live claim per slot" is rejectedOne Live Claim Per Slot Is Rejected: the refusal is the invariant working
A pod that selects a label never schedulesA Pod Needs A Slot With A Specific Label: slots that disagree on a key leave it out of the group's template, fail-closed by design
A shutdown that never landsThe Shutdown Job Exits 0 and the Machine Stays Up: three Jobs, then a 30m pause, and the shutdownAttempts patch as the escape hatch

Debugging Guide is the component-by-component method, State Management the phase table and the deadlines, Events every reason the claim and the slot publish, The NodeClaim model and NodeClaim Design why the bounds live on the request.