Architecture
Homelab Autoscaler is a Kubernetes operator with gRPC integration for physical node autoscaling.
Scale-up intent is an object: Node is what a slot is, NodeClaim is one request for one
machine, and Node.status is what the cluster observes about it.
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
│ Cluster │ │ Homelab │ │ Physical │
│ Autoscaler │◄──►│ Autoscaler │◄──►│ Nodes │
│ │gRPC│ │Jobs│ │
└─────────────────┘ └──────────────────┘ └─────────────────┘
│
▼
┌──────────────────┐
│ Kubernetes API │
│ (Groups, Nodes, │
│ NodeClaims) │
└──────────────────┘
Core Components
Custom Resources
Group CRD (infra.homecluster.dev/v2alpha1) - autoscaling policy for a pool:
- scaling thresholds and timing (
minSize,maxNodeProvisionTime, scale-down windows) - no node selection, and no
maxSize: a pool is the set ofNodeCRs labelledgroup: <name>, and the labels a machine carries are declared on the slot (spec.nodeLabels). The pool's ceiling is its slot count - the readiness taints a machine must be free of before it counts as capacity
- the pool's report:
slots,on,off,health, and the last scale
Node CRD (infra.homecluster.dev/v2alpha1) - a slot: the configuration of one machine, and
nothing about what anyone wants it to do next:
spec.startupPodSpec/spec.shutdownPodSpec,spec.kubernetesNodeName,spec.nodeTaints,spec.nodeCapacity,spec.pricingstatus.powerState(observed),status.phase(derived report),status.lastClaim(the digest of the last request)- Git is the only writer of the spec, so an apply can never contradict a scale-up
NodeClaim CRD (infra.homecluster.dev/v2alpha1) - one request for one machine:
spec.slotRef,spec.requestedBy,spec.request(call/delta/group/nodes),spec.deadlineAt,spec.maxNodeProvisionTime,spec.ttlSecondsAfterFinishedstatus.phase(pending/booting/ready/failed/completed),status.attempt,status.reason,status.finishedAt, conditions- created by the gRPC provider, owned by the slot, collected after its TTL
Controllers
Group Controller (internal/controller/infra/group_controller.go)
- Reads the policy object, publishes it to the provider, and keeps the
Loadedcondition - Counts the pool in one pass over its slots and writes the report
kubectl get groupsprints:slots,on,off, andhealthderived from the slots' reports (offlinewhen no slot has ever reported,healthyonce one has) - Materializes the last scale (
lastScaleAt/lastScaleAction) off each slot'sstatus.lastClaimdigest, so the pool's history outlives the Events and the claim's TTL - Decides nothing: no reconciler reads these counts, and the provider's arithmetic counts live claims and slots directly
Slot reconciler (internal/controller/infra/node_controller.go, lifecycle.go, claims.go)
- Reads the live claim of the slot, the observed power and the node's usability, and fires the one lifecycle transition that input asks for
- Finite State Machine architecture (
internal/fsm), seeded from that triple and never from a record - Coordination locking for race prevention (kept as a belt over the claim, see State Management)
- Owns the claim moves a transition is the evidence for (
pending → bootingwith the startup Job,booting → readywith the readiness gate,→ failedwith a Job failure, a timeout, a stuck transition) - ProviderID, declared taints, declared node labels and annotations (applied on join, re-asserted on drift, removed when the declaration drops the key), capacity drift: the slot-level plumbing
Claim controller (internal/controller/infra/nodeclaim_controller.go)
- Seeds
status.attemptfrom the slot's digest, and enforces the consecutive-attempt bound - Enforces one live claim per slot where admission cannot see it (
reason: Superseded) - Stamps the finalizer with the boot, runs the scale-down path, and ends the claim when the machine is observed off
- Fails an overdue claim (
reason: DeadlineExceeded) — the deadline is the heal the v1alpha1 watchdog performed - Stamps
Node.status.lastClaim, and collects a terminal claim after its TTL
Observation reconciler (internal/controller/core/node_controller.go)
- Bridges Kubernetes nodes with Node CRDs:
status.powerState, the derivedstatus.phase, health - Records what cluster-autoscaler decided about the machine (
internal/autoscaler): theDeletionCandidateOfClusterAutoscaler/ToBeDeletedByClusterAutoscalermarkers become anAutoscalerDeletionCandidate/AutoscalerDeletionRequestedevent on the Node CR, once per transition, and the node's taints andspec.unschedulableare logged so a cordon is attributable without raising the autoscaler's verbosity. The moment of the observation is in the condition's message (observed at ...), becauselastTransitionTimedoes not move on a candidate→requested transition (both areStatus: True) and would date the record early
Node slot controller (internal/controller/core/node_slot_controller.go)
- Reports the shape of slots that have never had a machine, so a pool's arithmetic and an
operator's
kubectl getdescribe them. "Is this slot free" is a claim question (internal/claim), so the v1alpha1 job of seeding a status so the target-size arithmetic excludes a never-provisioned slot is gone
Scale-up readiness
The claim reaches ready when the node is Ready and free of every readiness taint the
pool declares. Pool workloads therefore do not need to tolerate node.cilium.io/agent-not-ready:
the controller waits it out, because declaring capacity before its datapath works makes the
autoscaler look at a Ready node its pending pod cannot land on and buy another machine.
The key belongs to the CNI, so it is configuration: Group.spec.nodeReadinessTaints is the list the
gate waits for, defaulted by the CRD to Cilium's node.cilium.io/agent-not-ready (stamped by Cilium
since v1.9; this chart deploys 1.16.x). An empty list gates on Ready alone. The keys are recorded on
the slot as the NodeReadinessTaints condition.
The two deadlines are separate: 90s for the node object to register, then 4m for the gate, counted
from the moment the object appears. On a timeout the controller uncordons the node (a cordoned
machine is invisible to scale-down and would be paid for forever), records a ReadinessGateTimeout
Warning on the slot, and moves the claim to failed with that reason: the input law then powers
the machine off, the slot becomes free, and the next claim on it is attempt N+1. See
State Management.
gRPC Server (internal/grpcserver/server.go)
Implements the Cluster Autoscaler CloudProvider interface:
NodeGroups(),NodeGroupForNode(),NodeGroupNodes(),NodeGroupTemplateNodeInfo(),NodeGroupGetOptions(), pricingNodeGroupTargetSize()- the count of the group's live claimsNodeGroupIncreaseSize(delta)- createsdeltaclaims ondeltafree slots, each carrying the attribution:requestedBy,request.call,request.delta,request.group, and a deadlineNodeGroupDeleteNodes(nodes)- records the request on the claim of each named slot and cancels it (apendingclaim is deleted; abooting/readyclaim is deleted through its finalizer, which runs the shutdown)NodeGroupDecreaseTargetSize(delta)- cancelspendingclaims, never abootingone, and never below the group'sminSize
No handler writes a field of a Node spec. That is the point: spec.powerState was the request,
the lock, the arithmetic and the history at once, and a GitOps apply with selfHeal re-applied it
over the autoscaler's write on every sync.
Target size
targetSizeOf is the count of live claims: pending + booting + ready, excluding a claim a
scale-down has marked. A failed claim stops being counted the moment it is recorded, which
removes the state "the autoscaler is paying for a machine that never arrived, forever". No progress
value, and no slot spec field, appears in the arithmetic, and every minSize/maxSize check in the
provider is judged against this same number.
Instances
The NodeGroupNodes contract is unchanged: a slot with a corev1.Node is running while a live
claim wants it and deleting when nothing wants it any more, a slot a live claim names that has
not joined is creating, and a slot no claim names with no machine behind it is not reported. The
word "wanted" moved from a spec field to the claim. The node template
(NodeGroupTemplateNodeInfo) reads Node.spec only, which is what makes it correct in v2: the
config object is a config object.
Data Flow
- Declaration:
kubectl applyof aNode→ the slot reconciler stamps providerID, taints, the Group ownerRef, and checks the declared capacity - Scale up: Cluster Autoscaler →
NodeGroupIncreaseSize→ aNodeClaimon a free slot → the slot reconciler takes the lock, creates the startup Job, moves the claimpending → booting→ the machine boots, the node joins → the readiness gate moves the claimbooting → ready - Serving: a ready claim plus an observed-up, usable node is capacity; the slot reports
ready - Scale down: Cluster Autoscaler drains, then
NodeGroupDeleteNodes→ the request is recorded on the claim and the claim is cancelled → the slot reconciler cordons and runs the shutdown Job → the machine is observed off → the claim endscompletedand is collected, and the slot keeps the digest - A stalled attempt: the claim passes
spec.deadlineAt→failed(DeadlineExceeded) → the machine is force-shut-down → the slot is free, and the next claim is attempt N+1
Design Patterns
Controller Pattern
Each CRD has one owner:
- the claim controller owns the claim and its lifecycle Jobs' outcome
- the slot reconciler owns the lifecycle operation on the slot
- the observation reconciler owns the report of what the cluster sees
Job-Based Power Management
Power operations via Kubernetes Jobs:
- Startup jobs: Execute power-on commands, created by the
StartNodetransition - Shutdown jobs: Graceful power-off, created by the
ShutdownNodetransition - Isolation: Operations in containers
- Monitoring: the Job monitor of the operation fires the transition, and the claim's phase records its outcome
Health Monitoring
Node health determined by:
- observed power (
status.powerState, from thecorev1.NodeReady condition) - the derived report (
status.phase) - the last episode (
status.lastClaim) - periodic resyncs (5 minutes for a stable slot, at the deadline for a claim)
Integration Points
Cluster Autoscaler
Standard gRPC CloudProvider interface. The provider answers its four mutation calls with claims.
Kubernetes API
Controllers use controller-runtime for watching CRDs, writing status, creating and managing Jobs,
and monitoring node events. The claim controller's field index is spec.slotRef.name.
Webhook Validation
Node: thegrouplabel must reference a Group CR; a slot whosecorev1.Nodehas not joined is admitted with a warning, never deniedNodeClaim: the slot must exist, and one live claim per slot. The message names the slot and the claim that owns it. A terminal status write and a delete are never refused
Configuration
Namespace
- Defaults to
homelab-autoscaler-system, overridable withHOMELAB_AUTOSCALER_NAMESPACE
gRPC Server
- Enabled on port 50051 by default, configurable via command-line flags
Health Checks
- The manager's own liveness and readiness probes, on
:8081(/healthz,/readyz), defined in the chart (controllerManager.container.livenessProbe/.readinessProbe). There is no scheduled health job: nothing runs on a CronJob - A pool's health is a field on the objects, not a probe result:
kubectl get groupsanswers it from the slots' reports, and a slot's health is on the slot
Security
RBAC
Controllers need permissions for the Group, Node and NodeClaim CRDs (including
nodeclaims/status and nodeclaims/finalizers), Jobs, Kubernetes nodes, and events.
Job Security
Startup/shutdown jobs run with minimal permissions, Secrets/ConfigMaps for credentials, and network policy isolation.
Scalability
- controller-runtime leader election
- the gRPC server is stateless; the claims make the scale-up history an object, not a field
Related Documentation
- Installation - Helm deployment
- Quick Start - k3d testing
- State Management - the input law, the claim phases, the lock
- NodeClaim CRD - the request object
- NodeClaim Design - why this model
- Known Issues - Current limitations