Node CRD Reference
The Node Custom Resource Definition (CRD) is the declaration of one machine slot in the pool: what the slot is (its lifecycle pods, its shape, its taints, its price), and nothing about what anyone wants it to do next.
API Version
- Group:
infra.homecluster.dev - Version:
v2alpha1(served and storage;v1alpha1is removed in this release) - Kind:
Node
Specification
NodeSpec
The spec is Git-owned configuration. spec.powerState is removed in v2alpha1: there is no
field for a runtime intent, so an apply can never contradict a scale-up and a scale-up can never
overwrite a declaration. A machine is wanted exactly when a live
NodeClaim names this slot.
| Field | Type | Required | Description |
|---|---|---|---|
startupPodSpec | MinimalPodSpec | Yes | Pod specification for startup operations |
shutdownPodSpec | MinimalPodSpec | Yes | Pod specification for shutdown operations |
kubernetesNodeName | string | Yes | Name of the corresponding Kubernetes node |
pricing | PricingSpec | Yes | Cost information for the node |
nodeCapacity | corev1.ResourceList | No | The shape of the machine this slot boots, as the kubelet reports it. Applied to the node template resource by resource, which is what lets a pool at minSize: 0 scale back up; pods is defaulted for you |
nodeTaints | []corev1.Taint | No | Taints this machine must carry, applied to the Kubernetes node and included in the node template |
taintsToRemove | []corev1.Taint | No | Taints stripped from the node template so the autoscaler ignores them |
MinimalPodSpec
| Field | Type | Required | Description |
|---|---|---|---|
image | string | Yes | Container image for the operation |
command | []string | No | Command to execute |
args | []string | No | Arguments for the command |
volumes | []VolumeMountSpec | No | Volume mounts for secrets/configmaps |
serviceAccount | *string | No | ServiceAccount for the pod |
priorityClassName | *string | No | PriorityClass assigned to the job pod, e.g. system-cluster-critical |
VolumeMountSpec
| Field | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Volume name |
mountPath | string | Yes | Path to mount the volume |
secretName | string | No | Secret to mount (mutually exclusive with configMapName) |
configMapName | string | No | ConfigMap to mount (mutually exclusive with secretName) |
PricingSpec
| Field | Type | Required | Description |
|---|---|---|---|
hourlyRate | float64 | Yes | Cost per hour when running |
podRate | float64 | Yes | Cost per pod scheduled on the node |
NodeStatus
| Field | Type | Description |
|---|---|---|
powerState | PowerState | What the provider observes: on when the corev1.Node reports NodeReady=True, off otherwise. An observation, and one of the three inputs of every decision |
phase | NodePhase | The derived report of the slot: off, pending, booting, ready, draining. Computed from (live claim, observed power, node usability) and read by nothing that decides. status.progress is removed with the model that needed it |
lastClaim | LastClaimSummary | The digest of the slot's most recent claim: {name, phase, attempt, call, group, reason, requestedAt, deadlineAt, finishedAt}. It survives the claim's TTL collection, seeds the next claim's status.attempt, and answers "who asked, when, how many attempts, why did it end" from kubectl describe |
conditions | []metav1.Condition | Standard Kubernetes conditions: the autoscaler markers (AutoscalerDeletionCandidate, AutoscalerDeletionRequested), NodeReadinessTaints (the gate this slot is waiting on), lifecycle conditions written by the transitions |
health | string | Health status: healthy, offline, unknown |
lastStartupTime | *metav1.Time | Last successful startup time |
lastShutdownTime | *metav1.Time | Last successful shutdown time |
PowerState
type PowerState string
const (
PowerStateOn PowerState = "on" // the node object reports NodeReady=True
PowerStateOff PowerState = "off" // everything else, including a NotReady leftover
)
on/off accept true/false on the way in, for manifests written against the old field.
Phase (derived, reported)
type NodePhase string
const (
NodePhaseOff NodePhase = "off" // no live claim, no observed machine
NodePhasePending NodePhase = "pending" // a live pending claim, no lifecycle Job yet
NodePhaseBooting NodePhase = "booting" // a live claim whose boot is in flight
NodePhaseReady NodePhase = "ready" // a ready claim, node observed Ready and gate-clear
NodePhaseDraining NodePhase = "draining" // a machine the pool no longer wants, observed up
)
The table that computes it is in State Management.
It is a report: write a value that contradicts the claim and the observation and nothing about the
operator's behaviour changes — the next pass overwrites the value from its inputs. The phase the
model acts on is the claim's (NodeClaim.status.phase).
Node Taints
Declare the taints a machine must carry in spec.nodeTaints:
spec:
nodeTaints:
- key: dedicated
value: remote
effect: NoSchedule
What the field gives you:
- cluster-autoscaler simulates the machine your pool boots. The provider puts the declared taints
in the node template, so
dedicated=remote:NoScheduleis there before the machine registers, and a pod that doesn't tolerate it is never placed on this group. - The taints are in Git with the rest of the slot, and they survive node recreation. Nothing depends on a startup script running.
The controller applies them to the Kubernetes node as soon as the node object exists: the scale-up path stamps them in the hook that uncordons the node, so the machine carries them before it becomes schedulable, and every later reconcile re-applies them.
A tainting startup Job can't do that. It runs after the node joined, which is after the autoscaler simulated a placement on an untainted template and after the node was schedulable, so pods land on a machine that rejects them.
The window this closes is the one the controller drives. It can only patch a node that exists, so a machine you booted by hand, or one that registered while the controller was down, can be Ready and schedulable for a pass before the taint lands. Keep the tolerations of your own startup tooling correct.
The field only adds, and it keeps adding:
- A taint you don't declare is never removed.
- A taint you declare and then remove by hand with
kubectl taint <node> <key>-is put back on the next reconcile. The declaration is the intent, thekubectlrun isn't.
Keys the controller won't write, because a Kubernetes component owns them and re-applies them on its own sync loop:
| Keys | Owner |
|---|---|
node.kubernetes.io/not-ready, node.kubernetes.io/unreachable, node.kubernetes.io/unschedulable, node.kubernetes.io/out-of-service | node lifecycle controller |
node.kubernetes.io/memory-pressure, node.kubernetes.io/disk-pressure, node.kubernetes.io/pid-pressure, node.kubernetes.io/network-unavailable | kubelet |
ToBeDeletedByClusterAutoscaler, DeletionCandidateOfClusterAutoscaler | cluster-autoscaler |
node.cilium.io/agent-not-ready | Cilium (the bootstrap taint the scale-up readiness gate waits for - see the Group's nodeReadinessTaints) |
A declaration on one of those keys is dropped, and the drop is visible: a Warning event on the
Node CR (kubectl describe node.infra.homecluster.dev <name>, reason
SkippedProtectedTaintKey) and a V(1) log line. Nothing is dropped silently.
The node.kubernetes.io/ prefix is a namespace, not an ownership claim. The
node.kubernetes.io/role=agent taint your k3d agents carry is yours to declare, and the
controller applies it.
spec.taintsToRemove is the opposite field and is unrelated: it filters taints out of the node
template so the autoscaler ignores taints your startup process leaves behind. It never touches
the live node. A key listed in both fields ends up on the template, because nodeTaints is what
the machine boots with.
If you migrate a pool from a tainting startup Job, drop the kubectl taint step from the Job and
set nodeTaints on the Node CRs. The template carries the taint as soon as the CR is applied;
the node gets it the next time the controller touches it, which on a scale-up is before the node
is uncordoned.
Node Capacity
Declare the shape of the machine a slot boots in spec.nodeCapacity:
spec:
nodeCapacity:
cpu: 1950m
memory: 3.2Gi
What the field gives you is a pool that can come back up from zero. cluster-autoscaler decides
whether a pod fits on a machine by simulating it on the group's node template, and that template
is built from a corev1.Node. A pool whose every slot is powered off has no corev1.Node at all,
so the provider synthesizes a placeholder, and a placeholder it has no size for carries no cpu and
no memory. A pod with any resource request then fits nowhere, NodeGroupIncreaseSize is never
called, and the pool stays at zero. That is why a minSize: 0 pool needs an always-on machine
today.
Capacity is the one part of the machine's shape a Node CR can answer for, because you know the
machine you are going to boot. It is declared in the spec and not copied into the status for the
same reason: the template is needed before a slot has ever booted, so a status field fed from a
live node is empty in exactly the case it would be read for, and keeping it filled means a pipeline
that watches nodes and writes status. One field next to the nodeTaints of the same machine is
reviewable in Git, and it describes a machine that does not exist yet.
Declare what the kubelet reports, not what the box is sold as. The system reservation moves the
number: a 2 vCPU / 4 GiB machine, a Hetzner CX23 for instance, reports 1950m and 3.2Gi.
| Declared | Reported by the kubelet | Verdict |
|---|---|---|
1950m | 2 | 2.5% away, fine |
1950m | 1 | the machine is not the one you described |
The autoscaler judges the pod's requests against the template, so a declaration larger than reality places work the machine cannot take, and a smaller one leaves capacity unused.
Precedence in the template, resource by resource:
- Every resource you declare wins over the live node's
status.capacityandstatus.allocatable, on the path that copies a registered node as well as on the placeholder. The declaration is the shape of the machine the pool boots, and it is one value for every slot of the group. Reading the live node would make the group's template change with which slot happens to be Ready, and in a mixed pool it hands the estimator a different machine per call. - A resource you don't declare keeps the live node's number:
pods,ephemeral-storage, hugepages,nvidia.com/gpu, anything a device plugin adds. The estimator judges a pod against all of them, so declaring cpu and memory declares the shape of the box, not the list of resources it has. Replacing the whole list with a declaration is what makes a documented cpu-and-memory declaration unplaceable: the template losespods, and the scheduler answers every pod "Too many pods". podsis defaulted for you. A template whose lists carry none getspods: 110, the kubelet's default--max-podsand the number cluster-autoscaler's own template builders put in. Declare it only to move off that default, and a live node's own pod ceiling is never lowered to it. Nothing else is defaulted: cpu, memory and disk are facts about one machine, and the provider doesn't invent machines.- With the field unset, the live node's own numbers are used. A pool that declares nothing keeps working exactly as before, and a pool that declares nothing and is fully powered off gets a template with no cpu, no memory and no disk: the honest "this machine has no known size", never a guess.
The declaration goes into status.capacity as well as status.allocatable, because
cluster-autoscaler reads both. Allocatable is what a pod's requests are judged against; capacity
feeds the bookkeeping over the group's machines, the quota simulation and the resources it reports
for registered nodes. A template that fills only one of the two is judged by the other. A template
machine has no system daemons to subtract, so a placeholder's two lists end up identical; on a node
copied from a live slot each keeps its own numbers, overlaid resource by resource.
The controller checks the declaration against the machine once it registers:
- A
status.allocatablemore than five percent away from the declaration, on any declared resource, puts aWarningevent on the Node CR with reasonNodeCapacityMismatch, naming the resource and both values. Read it withkubectl describe node.infra.homecluster.dev <name> -n <namespace>. - Only the resources you declared are compared, so an incomplete declaration is not detected. Leave
memoryout and no warning ever mentions it, while a pod that asks for memory is judged against a template with no size for it. Declare every resource your workloads ask for;podsis the one resource the provider fills in for you. - It warns only. The node is never reconciled to the declared values: the kubelet owns
status.allocatable, and a machine that registered is the truth about what booted. Fix it by updatingspec.nodeCapacity, or by fixing the machine. - Nothing is compared until the node is Ready, because before the kubelet reports allocatable there is no measurement to compare. A pool that declares nothing is never warned about.
Node Template
The provider reports one machine shape per group, built from a slot of that group:
- A slot whose Kubernetes node object exists, a Ready one preferred. That node object is the
only source of
status.allocatable,spec.taintsand the labels a pod needs to schedule, so the autoscaler simulates a real machine. - If no slot has a node object (the whole pool is powered off), the provider synthesizes a
placeholder from the Node CR. It carries the group label, the declared
nodeTaintsand the declarednodeCapacity, on top of the default pod ceiling. No machine size is invented: a pool that declarednodeCapacityis answering the question from Git, and a pool that declared nothing gets a template with no cpu, no memory and no disk, because the size of a machine that never registered can't be read.
Because the template is copied from a live node, the autoscaler's own taints are filtered out of
it: ToBeDeletedByClusterAutoscaler and DeletionCandidateOfClusterAutoscaler on the
representative node describe that machine's scale-down state, not the shape of a new one, and
copying them into the group's template leaves the estimator with a taint no pod tolerates. The
transient node.kubernetes.io/unschedulable, not-ready and unreachable taints are filtered
out the same way. Keys the pool declares, node.kubernetes.io/role included, are not.
Example
apiVersion: infra.homecluster.dev/v2alpha1
kind: Node
metadata:
name: worker-01
namespace: homelab-autoscaler-system
labels:
group: worker-nodes
node-type: worker
zone: homelab
spec:
kubernetesNodeName: worker-01
startupPodSpec:
image: homelab/power-manager:latest
command: ["/bin/sh"]
args: ["-c", "wake-on-lan 00:11:22:33:44:55"]
volumes:
- name: credentials
mountPath: /etc/credentials
secretName: power-credentials
serviceAccount: power-manager
shutdownPodSpec:
image: homelab/power-manager:latest
command: ["/bin/sh"]
args: ["-c", "ssh root@worker-01 'shutdown -h now'"]
volumes:
- name: ssh-keys
mountPath: /root/.ssh
secretName: ssh-keys
serviceAccount: power-manager
pricing:
hourlyRate: 0.05
podRate: 0.001
nodeCapacity:
cpu: 1950m
memory: 3.2Gi
nodeTaints:
- key: dedicated
value: remote
effect: NoSchedule
status:
health: healthy
powerState: on # observed
phase: ready # derived from (live claim, observed power, node usability)
lastClaim: # the digest of the request that owns this machine
name: worker-01-7f4b9c
phase: ready
attempt: 2
call: IncreaseSize
group: worker-nodes
requestedAt: "2025-01-01T11:52:00Z"
deadlineAt: "2025-01-01T11:59:10Z"
lastStartupTime: "2025-01-01T12:00:00Z"
conditions:
- type: NodeReadinessTaints
status: "True"
reason: ReadinessTaints
message: "node.cilium.io/agent-not-ready"
lastTransitionTime: "2025-01-01T11:52:00Z"
The claim behind that status is the NodeClaim CRD.
Power Management
Startup Process
- Trigger: a live
NodeClaimnames the slot and the machine is off - Job Creation: the slot reconciler takes the coordination lock and creates the Job from
spec.startupPodSpec, in the same write that moves the claimpending → booting - Report: the slot reports
bootingwhile the startup is in flight - Completion: the node registers, passes the readiness gate, the claim moves to
ready, and the slot reportsready
Shutdown Process
- Trigger: no live claim names the slot while its machine is observed up — a scale-down cancelled the claim, or the claim hit its deadline
- Node Drain: cluster-autoscaler evicts the pods before it calls
DeleteNodes; the provider observes and records the marker, it does not evict - Job Creation: the slot reconciler cordons the node and creates the Job from
spec.shutdownPodSpec - Report: the slot reports
drainingwhile the machine is still observed up — a shutdown Job that exits 0 is not evidence that the machine is off - Completion: the machine is observed off, the scale-down claim ends
completed, and the slot reportsoff
Power Operation Examples
Wake-on-LAN Startup
startupPodSpec:
image: alpine/socat:latest
command: ["/bin/sh"]
args:
- "-c"
- |
echo -ne '\xFF\xFF\xFF\xFF\xFF\xFF\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55\x00\x11\x22\x33\x44\x55' | socat - UDP-DATAGRAM:255.255.255.255:9,broadcast
IPMI Power Control
startupPodSpec:
image: gemtest/ipmitool:latest
command: ["ipmitool"]
args: ["-I", "lanplus", "-H", "worker-01-ipmi", "-U", "admin", "-P", "password", "power", "on"]
volumes:
- name: ipmi-credentials
mountPath: /etc/ipmi
secretName: ipmi-credentials
shutdownPodSpec:
image: gemtest/ipmitool:latest
command: ["ipmitool"]
args: ["-I", "lanplus", "-H", "worker-01-ipmi", "-U", "admin", "-P", "password", "power", "soft"]
volumes:
- name: ipmi-credentials
mountPath: /etc/ipmi
secretName: ipmi-credentials
SSH-based Shutdown
shutdownPodSpec:
image: alpine/ssh:latest
command: ["ssh"]
args: ["-i", "/root/.ssh/id_rsa", "-o", "StrictHostKeyChecking=no", "root@worker-01", "shutdown -h now"]
volumes:
- name: ssh-keys
mountPath: /root/.ssh
secretName: ssh-keys
Usage
Creating a Node
kubectl apply -f node.yaml
Monitoring Node Status
# List all nodes
kubectl get nodes.infra.homecluster.dev -n homelab-autoscaler-system
# Get detailed node information
kubectl describe node.infra.homecluster.dev worker-01 -n homelab-autoscaler-system
# Watch node status changes
kubectl get nodes.infra.homecluster.dev -n homelab-autoscaler-system -w
Manual power operations
There is no field to declare. A manual scale-up is a claim, and a manual scale-down is the
cancellation of one — the same two objects the autoscaler's RPCs produce, with requestedBy
naming who did it:
# power on: ask for the machine
apiVersion: infra.homecluster.dev/v2alpha1
kind: NodeClaim
metadata:
name: worker-01-manual
namespace: homelab-autoscaler-system
labels:
group: worker-nodes
infra.homecluster.dev/node: worker-01
spec:
slotRef:
name: worker-01
requestedBy: filippo
request:
call: IncreaseSize # the closest label for a hand-made request
group: worker-nodes
# power on
kubectl apply -f worker-01-manual.yaml
kubectl get nodeclaim -n homelab-autoscaler-system -w
# power off: cancel it. A pending claim is deleted outright; a booting/ready claim runs the
# shutdown path through its finalizer, and ends when the machine is observed off.
kubectl delete nodeclaim worker-01-manual -n homelab-autoscaler-system
A second live claim on one slot is refused by the webhook, naming the slot. See the NodeClaim CRD.
Best Practices
Security
- Use ServiceAccounts with minimal required permissions
- Store credentials in Secrets, not in pod specs
- Use SSH keys instead of passwords where possible
- Implement network policies to restrict power management pod access
Reliability
- Test startup and shutdown procedures thoroughly
- Implement timeouts in power operation scripts
- Use idempotent operations (safe to retry)
- Monitor job completion and implement retry logic
Labeling
- Always include
grouplabel for autoscaling group association - Add descriptive labels for node capabilities and location
- Use consistent labeling across your infrastructure
Pricing
- Set realistic hourly rates based on actual power consumption
- Include infrastructure costs (cooling, networking) in hourly rate
- Use pod rate to encourage efficient resource utilization
Troubleshooting
Common Issues
-
A slot that never leaves
booting- The boot is the claim's:
kubectl get nodeclaim -n homelab-autoscaler-system -l infra.homecluster.dev/node=<slot>and readstatus.phase,status.reason,spec.deadlineAt - Check startup job logs:
kubectl logs -n homelab-autoscaler-system job/<startup-job> - Verify network connectivity to the target node, and the credentials it uses
- If the gate is what is slow,
kubectl describe node.infra.homecluster.dev <slot> -n homelab-autoscaler-systemshows theNodeReadinessTaintscondition: the key the slot is waiting for. Past the gate's 4m the claim fails withReadinessGateTimeout
- The boot is the claim's:
-
Node not joining cluster
- Verify
kubernetesNodeNamematches actual node name - Check node configuration and kubelet status
- Ensure network connectivity between node and cluster
- Verify
-
Power operations failing
- Check job logs for error messages
- Verify credentials and network access
- Test power commands manually
-
A pool at
minSize: 0never scales up- With no slot registered, the provider can only report the capacity the slots declare:
kubectl get node.infra.homecluster.dev <slot> -o jsonpath='{.spec.nodeCapacity}'returning nothing means the group's template has no cpu and no memory, and no pod with requests fits it. Declare the field, and declare every resource your workloads ask for kubectl describe node.infra.homecluster.dev <slot>shows aNodeCapacityMismatchwarning when the declaration and the booted machine disagree. That drift doesn't stop placement: it means the estimator is sizing the machine wrong, so you over-provision work the node can't take (the pod stays Pending after the machine boots) or leave a booted machine idle. The unplaceable case is the missing declaration above- A declaration that names no
podsis not the problem it looks like: the provider defaults the pod ceiling to the kubelet's 110, so a cpu-and-memory declaration scales from zero fine
- With no slot registered, the provider can only report the capacity the slots declare:
Debugging Commands
# Check node controller logs
kubectl logs -n homelab-autoscaler-system deployment/homelab-autoscaler-controller-manager
# List power operation jobs
kubectl get jobs -n homelab-autoscaler-system
# Check job logs
kubectl logs -n homelab-autoscaler-system job/<job-name>
# Describe node for detailed status
kubectl describe node.infra.homecluster.dev <node-name> -n homelab-autoscaler-system
Related Documentation
- Group CRD - Autoscaling group configuration
- NodeClaim CRD - the request for a machine: the field this Node CRD no longer has
- NodeClaim Design - why
spec.powerStateis gone - State Management - the input law, and the report
status.phaseis - Architecture Overview - System design