E2E Test Flow

make test-e2e runs the suite against a real k3d cluster: cluster-autoscaler in the cluster, the manager on your laptop, the pool's machines as k3d containers that only the suite's HTTP server can start and stop. It is the only test that drives the whole loop — a pending pod, the gRPC provider's claim, a lifecycle Job, a machine that joins, a scale-down that powers it off — and the only one that fails when the model and a real cluster stop agreeing.

Test Structure

The e2e suite is one Ginkgo Describe("K3d Integration", Ordered), and the specs are the four Contexts inside it, in the order the container runs them:

Setup & Environment Validation - Verifies prerequisites, prepares the environment, and claims the machine the scale-down spec runs on
Scale-Down Workflow - Verifies CA decides to remove a machine and the model powers it off
Scale-Up Workflow - Verifies CA scales up when pods need scheduling
Scheduling With Slot Labels - Verifies the labels a slot declares land on its node, are re-asserted against drift, come off when the declaration drops them, never overwrite a key Kubernetes owns, and bind a pod that selects the label to the machine that declares it

All tests share state via Ginkgo's BeforeSuite/AfterSuite and global variables.

The fourth spec is a Context inside the ordered container, not a container of its own, and that placement is a decision about cost, not style: Ginkgo randomizes the order of top-level containers, so a container of its own runs on whatever the suite left behind - the scale-down specs power the pool's machines down, so a spec that needs a Ready, uncordoned, served machine in its own container starts by booting one, and paying for a boot inside a second container does not fit the suite's go test budget. Inside the ordered container the machine the scale-up spec bought is Ready, uncordoned and served by a live claim, which is the state a labelled pool is in when a pod asks for it. The consequence is the coupling the spec documents: it reads shutdownNodeName, the machine the scale-up spec served.

The suite is the provider's counterpart: the pool's machines are k3d containers that only the VM control server powers, so a simulated machine that is in service has a live NodeClaim naming its slot, and a slot nobody claims is a machine the pool does not want. claimSlot and releaseSlot in test/e2e/k3d_integration_test.go are the harness playing that half of the model.

Architecture Components

1. VM Control Server (`examples/k3d/vm_control_server.py`)

Simple HTTP server running on port 9052
Exposes /start?vm=VM_NAME and /stop?vm=VM_NAME endpoints
Executes k3d node start/stop commands to power k3d nodes on/off
Simulates real VM provisioning/shutdown for k3d nodes
Logs all requests to /tmp/vm-control-requests.log for test verification

2. Homelab Autoscaler Controller (`cmd/main.go`)

Runs locally outside the cluster
Exposes gRPC server on port 50052
Implements the Cluster Autoscaler integrations API
Monitors node groups and communicates scaling decisions to CA

3. Kubernetes Cluster Autoscaler (Helm chart)

Runs in-cluster in homelab-autoscaler-system namespace
Connects to local gRPC server via service forwarder
Makes scaling decisions based on pending pods and node utilization
Communicates with homelab-autoscaler for node lifecycle management
Logs all gRPC interactions for test verification

4. Node Custom Resources (`examples/k3d/nodes1.yaml`)

Define individual nodes with startup/shutdown hooks
startupPodSpec: Executes curl to VM control server's /start endpoint when node is provisioned
shutdownPodSpec: Executes curl to VM control server's /stop endpoint when node is decommissioned
Organized into groups (e.g., group1) for collective scaling

5. Service Forwarders (created during test setup)

homelab-autoscaler-grpc-local: Routes cluster traffic on port 50052 to the local gRPC server
host-internal: Routes traffic on port 9052 to the VM control server running on the host

6. Test Utilities (`test/utils/utils.go`)

Node tainting utilities (TaintNode, UntaintNode, VerifyNodeTaint)
VM control server utilities (VerifyVMControlServer, VerifyVMControlServerRequest)
gRPC verification utilities (GetClusterAutoscalerLogs, VerifyGRPCCall)
Job verification utilities (VerifyJobHasServiceAccount, VerifyJobHasVolume, WaitForJobCompletion)
Slot and node verification utilities (WaitForSlotPhase, WaitForNodeReady, WaitForK3dNodeState)
Pod verification utilities (WaitForPodScheduled)

Test Execution Flow

Phase 1: Cleanup (the head of the `test-e2e` target)

lsof -i :8081 | awk '{print $$2}' | tail -n 1 | xargs kill -9 || echo "ok"
lsof -i :9052 | awk '{print $2}' | tail -n 1 | xargs kill -9 || echo "ok"
lsof -ti :50052 | xargs kill -9 2>/dev/null || true

Frees the ports the run owns: 8081 the manager's health probes bind, 9052 the VM control server, 50052 the local gRPC server
Ensures clean state before starting; make run-e2e only frees 50052, because it is the target you re-run against a cluster you already have

Phase 2: Environment Setup (`./development/install-dev.sh`)

./development/install-dev.sh

This script performs:

Prerequisites Check: Validates k3d, helm, kubectl are installed
Cluster Creation: Creates fresh k3d cluster named homelab-autoscaler
Kubeconfig Setup: Exports ./kubeconfig for cluster access
Namespace Creation: Creates homelab-autoscaler-system namespace
Helm Installation: Installs cluster autoscaler with development values
Service Forwarder: Applies local service forwarder for gRPC communication

Phase 3: Start VM Control Server

cd examples/k3d && nohup python3 vm_control_server.py > /tmp/vm-control.log 2>&1 &

Starts the HTTP server on port 9052
Listens for start/stop commands for k3d nodes
Stores the PID in /tmp/vm-control.pid for cleanup

Phase 4: Start Local Controller

KUBECONFIG=./kubeconfig ENABLE_WEBHOOKS=false nohup go run cmd/main.go --grpc-server-address :50052 > /tmp/homelab-autoscaler.log 2>&1 &

Launches the homelab-autoscaler controller locally
Binds the gRPC server to port 50052: the local service forwarder maps the in-cluster homelab-autoscaler-grpc-local:50051 to it, which is the endpoint cluster-autoscaler dials
Uses the local kubeconfig to talk to the k3d cluster
Stores the PID in /tmp/local-server.pid for cleanup
Webhooks disabled for local testing: a manager outside the cluster cannot serve the chart's ValidatingWebhookConfiguration, so the admission rules are not enforced in this environment and the internal/webhook/infra/v2alpha1 suite covers them

Phase 5: Run E2E Tests

KUBECONFIG=./kubeconfig go test -timeout 25m -tags=e2e ./test/e2e/ -v -ginkgo.v -ginkgo.focus="K3d Integration"

The -timeout is spelled out, and the number is the suite's, not a guess. go test's default is 10 minutes, and the suite was already using ~516 s of it before the labels spec arrived: the three workflow specs wait on decisions of processes outside the suite - cluster-autoscaler's loop and scale- down window, a k3d container booting, the node lifecycle controller declaring a silent kubelet - with clocks measured in minutes (caDecisionTimeout 5m, machineStateTimeout 4m, machineDownTimeout 6m, and the deadline-watch spec bounded at maxDeadlineWatch 14m). The labels spec adds reconcile-sized gates (60 s for a write, 2 m for a bind), and the CI run of the suite measured 604 s. 25 m is that history with room for the slowest honest path of every gate, and it is a ceiling, not a target: a run that reaches it is a hung suite, and go test's panic prints every goroutine, which is the diagnostic. The same flag is on make run-e2e.

The ordered container's `BeforeAll`: the pool the run starts from

Every spec in the container starts from the shape the one before it left, and a run starts from the shape the cluster is in when it is handed over: the agents registered when the cluster was created and their containers are up, and on a reused cluster (a developer's make test-e2e, a re-run of the job) the slots carry the claims, the lifecycle Jobs and the cordons of episodes that are over. So the container states the shape before any of it is graded (BeforeAll):

the claims of both slots are released (releaseSlot: the harness stops wanting its machines, and a claim with a finalizer runs the shutdown of its machine to its end);
the machine of the slot the setup's claim boots is stopped by the harness - k3d node stop, idempotent - so the first graded gate of a run is the operator's answer to a settled shape, and not a race with a shutdown the operator started a minute earlier. That race was the flake: its chain is a Job pod scheduled, an image pulled, a container stopped and a kubelet's silence declared, every one of them on a clock the suite does not own, and the gate that rode it had 240 s;
both node objects are uncordoned, because a cordon outlives the machine it was written on and a cordoned machine is unusable to the readiness gate, to cluster-autoscaler and to the report;
the pool's other machine is deliberately left up and unclaimed. It is the idle machine of a homelab, and the scale-down spec reads it coming down by the input law alone - a proof that needs a machine that is up when the run starts.

Two rules run through the gates themselves:

a gate on a slot's report asks for a pass on every poll (slotReportIs), and prints the claim, the observation, the node object and the lifecycle Jobs when the value is not the one awaited. The report is a computation, and the events that wake its writers are the ones a settled pool has stopped producing: a dead kubelet has no kubelet to move its node object, a kubelet settled into Ready heartbeats to its Lease, a claim that ended has no phase left to move. The suite's clocks for those facts are minutes; the operator's pass over a settled slot is five;
the window a VM power transition is read over is the timeline of the object that asked for it (vmRequestProblem): the claim's spec.requestedAt for a boot, the minute the autoscaler's decision was read for a power-off. The server's log is not cleared between specs, and the same two containers are booted and powered off by three specs, so a window counted backwards from the poll is answered by an earlier episode and misses a boot whose Job pod was pulled and scheduled slowly.

Test 1: Setup & Environment Validation

Purpose: Verify prerequisites and prepare environment

Steps:

Verify k3d cluster exists
Set KUBECONFIG environment variable
Clear VM control server request log
Taint agent nodes with node.kubernetes.io/role=agent:NoSchedule
- Prevents regular workloads from scheduling on agent nodes
- Ensures clean scale-down detection
Verify gRPC service forwarder exists
Verify VM control server is accessible
Apply Node CRs from examples/k3d/nodes1.yaml, and the Group CR with the pool's scale-down numbers
Verify the slots are free: no live NodeClaim names them, and the machines nobody claimed are observed coming down (draining, then off) - a machine is wanted exactly when a live claim names its slot
Claim the slot of the machine the scale-down spec runs on: the harness writes a NodeClaim (request.call: IncreaseSize, the group, a delta, requestedBy: e2e) and follows it to status.phase: ready. The pool's machines are k3d containers the suite controls, so the suite is the provider's counterpart: it asks for a machine the way the autoscaler does, and the operator boots it

Verification Points:

k3d cluster is running
Agent nodes exist
Agent nodes are tainted
gRPC service forwarder exists
VM control server is responding
Node CRs are created, and they declare configuration only
The slots are free, and the unclaimed machines report draining/off
The claimed slot reaches ready: the claim is stamped ready by the readiness gate, the VM control server was asked for the machine, the slot reports ready and the machine is observed on

Test 2: Scale-Down Workflow

Purpose: Verify Cluster Autoscaler detects unneeded nodes and triggers shutdown

Preconditions:

The machine the spec runs on is in service because the setup's NodeClaim names its slot: the node is Ready, the slot reports ready, and the pool has an empty node for the autoscaler to consider
Agent nodes are tainted, so no workload lands on them

Steps:

Wait for Cluster Autoscaler to decide, reading the line its actuator writes as it taints the node and starts the deletion: Scale-down: removing empty node "<name>" (or the drain line), naming this machine. The word "unneeded" is not consulted: the autoscaler prints it as bookkeeping on every pass (--scale-down-unneeded-time in its flag echo, "<node> was unneeded for 3s" from the candidate tracker), so a gate built on it is true when nothing was decided
Verify gRPC NodeGroupDeleteNodes reached the provider, and that the provider's log records the cancellation of the claim of this machine
Verify the claim is cancelled: the request is recorded on it (call: DeleteNodes, nodes: naming the machine) and it carries the mark that runs the shutdown
Verify the slot reports draining: the machine is observed up and a scale-down names its claim
Verify the shutdown job of this machine, created after the decision, is the job the slot declares: ServiceAccount controller-manager, the auth-secrets Secret volume, a ttlSecondsAfterFinished, and a completion read off job.status.succeeded
Verify VM control server receives /stop for this machine within a window, not somewhere in the history of the run
Verify k3d node is stopped
Verify the slot's derived status.phase is off and status.powerState is off
Verify the claim stops being live (its finalizer is released once the machine is observed off), and the slot's digest of the episode records a terminal phase
Verify Kubernetes node becomes NotReady

Verification Points:

CA's decision is in its own log, naming this machine, and the RPC reached the provider
The claim that wanted the machine carries the DeleteNodes request and the mark
The slot's report is the machine coming down, and then off; the observed power is off
The shutdown job of this machine, from this decision, has the ServiceAccount, the Secret volume and the TTL the slot declares, and it completed
VM control server receives /stop request; k3d node stops
Slot reports off, and no live claim owns it; the digest is a terminal phase
Kubernetes node becomes NotReady

Shared State: Stores shutdown node name for Test 3

Test 3: Scale-Up Workflow

Purpose: Verify Cluster Autoscaler scales up when pods need scheduling

Preconditions:

Test 2 completed (node is shutdown)
Slot is off (Node.status.phase: off) and free: no live NodeClaim names it

Steps:

Untaint the shutdown node to allow scheduling
Verify node is untainted
Apply hello-world deployment targeting the agent node
Wait for CA to decide to grow the pool, reading the line the orchestrator writes as it calls the provider: Scale-up: setting group group1 size to N. "scale up" alone is printed on every idle pass ("no need to scale up"), so it cannot gate on a decision
Verify gRPC NodeGroupIncreaseSize call was made
Verify the provider created a NodeClaim for the slot, carrying the request (call, delta, group, requestedBy), the deadline (the budget of one attempt, floored by the group's maxNodeProvisionTime), the seeded status.attempt, the TTL and the ownerReference
Verify the startup job of this attempt, on this slot, is the job the slot declares: ServiceAccount controller-manager, the startup-config volume, a TTL, and job.status.succeeded
Verify VM control server receives /start for this machine, and the k3d node starts
Verify the claim moves pending -> booting -> ready, and the slot reports ready with status.powerState: on
Relax the pool's numbers (a long scaleDownUnneededTime, and a maxNodeProvisionTime that outlasts a k3d boot) so the autoscaler keeps the machine it just bought in service while the spec observes it
Watch the served claim across its own boot deadline: the claim's deadline is stamped into the past, and the window is measured from the claim's natural deadline so the observation crosses it. The claim stays ready, the slot stays ready, no shutdown Job appears, the node stays schedulable
Verify the deployment pod is scheduled on the machine the claim bought

Verification Points:

Pod scheduling triggers the autoscaler's scale-up decision, naming the pool
gRPC IncreaseSize call is made, and the provider's claim is the record of the request
The startup job of this attempt has the ServiceAccount and the ConfigMap volume the slot declares, carries a TTL, and completed
VM control server receives /start request; k3d node starts
The claim reaches ready and the slot reports ready
A ready claim that crosses its boot deadline keeps its machine (the window outlives the budget: the operator's claim.AttemptBudget, 12m10s, and the group's number only raises it)
Kubernetes node Ready, uncordoned
Deployment pod scheduled and running on the claimed machine

Test 4: Scheduling With Slot Labels

Purpose: Verify the labels and annotations a slot declares (spec.nodeLabels, spec.nodeAnnotations) reach the corev1.Node of that slot's machine, are kept there against drift, come off when the declaration drops them, never overwrite a key Kubernetes owns, and let a pod that selects the label bind to the machine that declares it. This is the e2e half of the feature; the half cluster-autoscaler reads - the bytes of the group's node template, including the placeholder of a pool whose machines are all off - is asserted in internal/grpcserver/labels_test.go, and the admission refusal of a reserved key in internal/webhook/infra/v2alpha1, because this environment installs the chart with webhook.enable: false and starts the manager with ENABLE_WEBHOOKS=false, so nothing in that cluster refuses an apply.

Preconditions:

The machine the scale-up spec bought is Ready, uncordoned, and served by a live claim (shutdownNodeName): the spec gates on it, and refuses to run without it
The pool's two slots are k3d-homelab-autoscaler-agent-0 and -agent-1; the spec declares a zone on each, deliberately two zones in one group (that is the heterogeneous pool the docs tell an operator to avoid, and the scheduler has to tell the two machines apart)

Steps:

Declare a zone, a rack label and a serial annotation on the zoned slot, and a different zone on the other slot, with a merge patch of the two spec fields - the shape a Git apply writes
Watch the labels, the annotations and the operator's record of the keys it wrote (infra.homecluster.dev/applied-node-labels) land on the Kubernetes node
Watch the write on the slot: an Event whose reason is NodeLabelsApplied, read the way this suite reads every cluster fact - kubectl get events -n <ns> -o json, decoded into corev1.EventList, filtered on involvedObject.kind == Node and the slot's name, and the read's failure surfaced in the assertion's message
Edit the label by hand the way kubectl label --overwrite does, ask for a pass over the slot (requestSlotPass), and watch the declaration come back. Nothing in the operator watches a corev1.Node for a slot's sake: drift is re-asserted on the slot's next pass, and for a settled slot the next pass is the five-minute idle resync, so the suite asks for the pass it is asserting
Drop the rack label and the serial annotation from the declaration, and watch both come off the node (asserted over a read that reports its own failure, so a nil read cannot pass), while the key still declared stays
Declare kubernetes.io/hostname from a slot that may not, ask for a pass, and assert the writer's rule where no webhook runs: the node keeps the kubelet's hostname, the record of applied keys does not list it, the node object carries the record of the refusal (infra.homecluster.dev/skipped-node-keys), and the slot carries the SkippedReservedNodeLabelKey Warning naming the key
Apply examples/k3d/zoned-deployment.yaml, a pod with nodeSelector: {topology.kubernetes.io/zone: fsn1, infra.homecluster.dev/group: group1}, and watch the binding name the zoned slot

Gates (the spec's own clocks, chosen for what is being waited on):

gate	budget	what it is waiting for
`labelGate`	60 s	a reconcile: the write of a declaration, the re-assert after a hand edit, the removal of a dropped key, the two Events
`bindGate`	2 min	the scheduler binding a pod to a node that already carries the label

The spec writes no machine-level clock: the suite's nodeStateTimeout, machineStateTimeout and the deadline watch belong to observations of a power state, and this spec never powers a machine. It also boots nothing, which is why it lives inside the ordered container.

Verification points:

The labels, the annotations and the ownership record are on the corev1.Node
The slot's Event stream carries NodeLabelsApplied for the write and SkippedReservedNodeLabelKey for the refusal, both read from the API server's projected corev1.EventList
A hand edit is put back; a dropped key is removed; a key the kubelet owns is never written
The pod that selects the zone is bound to the slot that declares it

Cleanup: the spec registers a DeferCleanup inside the It - legal at node level - which deletes the zoned deployment and nulls both slots' spec.nodeLabels / spec.nodeAnnotations, so the labels the suite wrote come off the agents it hands to the next run. The container's AfterAll (legal only on an Ordered container) is where the harness stops wanting its machines. DeferCleanup at container level is a Ginkgo v2 construction error - it fails the whole suite before any spec runs - which is why the spec's cleanup is registered on the node, and the suite's on the container. -ginkgo.dry-run is the local check for that class of mistake.

Phase 6: Cleanup (the `cleanup-e2e` target, run by `test-e2e` at the end)

Stop Local Controller:

if [ -f /tmp/local-server.pid ]; then
    source /tmp/local-server.pid
    kill $SERVER_PID 2>/dev/null || true
    rm -f /tmp/local-server.pid
fi

Stop VM Control Server:

if [ -f /tmp/vm-control.pid ]; then
    source /tmp/vm-control.pid
    kill $VM_CONTROL_PID 2>/dev/null || true
    rm -f /tmp/vm-control.pid
fi

Delete k3d Cluster:

./examples/k3d/delete-cluster.sh

Removes all k3d nodes and cluster resources
Cleans up Docker containers

make cleanup-e2e also deletes the zoned deployment of the labels spec (examples/k3d/zoned-deployment.yaml) before the cluster goes, so a run interrupted between the spec and its AfterAll does not leave the manifest applied in default.

Final Port Cleanup:

lsof -i :8081 | awk '{print $$2}' | tail -n 1 | xargs kill -9 || echo "ok"
lsof -i :9052 | awk '{print $2}' | tail -n 1 | xargs kill -9 || echo "ok"

Frees the probe port and the VM control server's port

Key Integration Points

gRPC Communication Flow

┌─────────────────────┐     gRPC      ┌─────────────────────────┐
│  Cluster Autoscaler │◄──────────────┤  Local Controller       │
│  (in-cluster)       │   Port 50052  │  (outside cluster)      │
└─────────┬───────────┘               └─────────────────────────┘
          │
          │ Service Forwarder
          ▼
┌─────────────────────┐
│  homelab-autoscaler │
│  grpc-local:50052   │
└─────────────────────┘

Node Lifecycle Flow

┌────────────┐    HTTP       ┌──────────────────┐    k3d CLI    ┌─────────────┐
│ Startup    │──────────────►│ VM Control       │──────────────►│ k3d Node    │
│ Pod        │  /start?vm=X  │ Server           │  k3d node     │ (powered on)│
└────────────┘               │ (port 9052)      │    start X    └─────────────┘
                             └──────────────────┘

┌────────────┐    HTTP       ┌──────────────────┐    k3d CLI    ┌─────────────┐
│ Shutdown   │──────────────►│ VM Control       │──────────────►│ k3d Node    │
│ Pod        │   /stop?vm=X  │ Server           │   k3d node    │ (powered    │
└────────────┘               │ (port 9052)      │    stop X     └─────────────┘
                                                            (stopped)

Test Validation Points

Test 1: Setup & Environment Validation

k3d cluster running: Cluster exists and is accessible
Agent nodes exist: Both agent nodes are present
Node tainting: Agent nodes tainted with node.kubernetes.io/role=agent:NoSchedule
gRPC forwarder exists: Service forwarder for local controller is configured
VM control server responding: HTTP server accessible on the forwarded port
Node CRs created: Node custom resources (v2alpha1) applied successfully, declaring configuration only
Slots are free: no live NodeClaim names a slot, and an agent machine that is up with no claim is reported draining, then off - the law "a machine is wanted exactly when a live claim names the slot" is what makes the idle pool of a homelab readable
The harness claims a machine: a NodeClaim written by the suite (requestedBy: e2e, request.call: IncreaseSize) boots its k3d machine through the operator and reaches ready; the slot reports ready, the machine is observed on, and the VM control server saw /start for it

Test 2: Scale-Down Workflow

CA decided, and said so: the autoscaler's actuator line Scale-down: removing empty node "<machine>" is in its log - not the word "unneeded", which it prints on every pass
gRPC DeleteNodes call: CA calls NodeGroupDeleteNodes, and the provider logs cancelling the claim of this machine
The claim is cancelled: it carries request.call: DeleteNodes with the machine named, and the mark that runs the shutdown; the slot with no live claim is asserted after the machine is off
Slot reports draining: the machine is observed up while its claim is named for deletion
Shutdown job of this machine, from this decision: scoped by the slot label and by the minute the spec started watching
Shutdown job ServiceAccount / volume / TTL: controller-manager, auth-secrets, and a ttlSecondsAfterFinished, read off the typed Job
VM control server receives /stop: request logged within a window, not somewhere in the history
k3d node stops: node status is not "running"
Slot phase=off, observed off: Node.status.phase is off and status.powerState is off, and the slot's digest of the last claim records a terminal phase
Kubernetes node NotReady: Node condition is False or Unknown

Test 3: Scale-Up Workflow

Node untainted: Taint removed, allowing scheduling
CA decided to grow the pool: the orchestrator's line Scale-up: setting group group1 size to N is in its log - not "scale up", which the idle pass prints
gRPC IncreaseSize call: CA calls NodeGroupIncreaseSize via gRPC
NodeClaim created for the slot: the scale-up is an object, with its attribution (request.call: IncreaseSize, request.group, request.delta, spec.requestedBy), its deadline, its seeded status.attempt, its TTL and its ownerReference on the slot
Startup job of this attempt, on this slot: scoped by the slot label and by the minute the spec started watching
Startup job ServiceAccount / volume / TTL: controller-manager and startup-config, read off the typed Job
VM control server receives /start: within a window, so the boot of the setup's machine is not the request this spec reads
k3d node starts: node status is "running"
Startup job completes: job.status.succeeded is at least 1
Claim phase=ready, slot phase=ready: the claim asserts the machine it asked for can take the pool's work, the slot reports ready, status.powerState is on, and the slot keeps the digest
A ready claim past its boot deadline keeps its machine: the observation is measured from the claim's own deadline (the operator's budget is 12m10s, and the group's maxNodeProvisionTime is a floor that only raises it), and the claim, the slot, the node and the pod are left alone
Kubernetes node Ready: Node condition is True, and uncordoned
Deployment pod running: hello-world pod scheduled on the machine the claim bought

Environment Variables

KUBECONFIG=./kubeconfig: Path to k3d cluster kubeconfig
ENABLE_WEBHOOKS=false: Disable webhooks for local testing

Test Constants

Defined in test/e2e/k3d_integration_test.go:

const (
    clusterName      = "homelab-autoscaler"
    grpcServerPort   = "50052"
    kubeconfigPath   = "./kubeconfig"
    namespace        = "homelab-autoscaler-system"
    vmControlPort    = 9052
    vmControlHost    = "localhost"
    vmControlLogFile = "/tmp/vm-control-requests.log"

    // Node names
    agentNode0 = "k3d-homelab-autoscaler-agent-0"
    agentNode1 = "k3d-homelab-autoscaler-agent-1"

    // Taints
    agentTaint = "node.kubernetes.io/role=agent:NoSchedule"

    // Timeouts
    // Gates the autoscaler is asked to pass: a decision, then the call it leads to
    caDecisionTimeout     = 5 * time.Minute  // CA's loop + the unneeded window + the drain
    deleteNodesRPCTimeout = 2 * time.Minute  // the taint delay, then the call

    // The machine's own clock: a container stops, the kubelet goes silent, the node lifecycle
    // controller declares it NotReady after its monitor grace period
    vmShutdownTimeout    = 60 * time.Second
    vmStartupTimeout     = 60 * time.Second
    jobCompletionTimeout = 120 * time.Second
    podScheduleTimeout   = 200 * time.Second
    nodeStateTimeout     = 120 * time.Second
    machineStateTimeout  = 4 * time.Minute

    // The claim's phases: created, committed, ready, collected
    claimCreatedTimeout   = 60 * time.Second
    claimPhaseTimeout     = 90 * time.Second
    claimReadyTimeout     = 5 * time.Minute
    claimCollectedTimeout = 3 * time.Minute

    // The window of the deadline spec, and the VM control request window
    minDeadlineWatch  = 3 * time.Minute
    maxDeadlineWatch  = 14 * time.Minute
    bootRequestWindow = 90 * time.Second

    // Check intervals
    checkInterval = 5 * time.Second
)

Expected Runtime

Test	Duration	Notes
Test 1: Setup	3-5m	The unclaimed machines come down, then the harness's claim boots one and follows it to `ready`
Test 2: Scale-Down	3-6m	The autoscaler's decision window, the drain, the shutdown Job, the machine observed off
Test 3: Scale-Up	15-20m	The boot to `ready`, then the observation that outlives the claim's 12m10s budget
Test 4: Slot Labels	2-4m	No boot and no machine clock: the spec's own gates are 60 s per reconcile and 2 min for the bind
Total	~25 minutes	The long spec is the deadline observation, which cannot be shortened: the budget is a sum of the operator's constants. The CI run of the suite measured 604 s

Test Tags

-tags=e2e: Build tag to include e2e tests
-ginkgo.focus="K3d Integration": Filter to run only K3d integration tests
Serial: Tests run sequentially (not in parallel)

Comparison: `test-e2e` vs `run-e2e`

Aspect	`test-e2e`	`run-e2e`
Cleanup	Automatic	Manual
Post-test state	Cluster deleted	Cluster running for inspection
Use case	CI/CD, full validation	Debugging, manual testing

Files Involved

Test Orchestration

Makefile: Test orchestration (test-e2e, run-e2e targets)
development/install-dev.sh: Environment setup (cluster creation, Helm installation)
examples/k3d/create-cluster.sh: Cluster creation script
examples/k3d/delete-cluster.sh: Cluster deletion script

Test Implementation

test/e2e/k3d_integration_test.go: the suite - one Describe("K3d Integration", Ordered) with the four Contexts listed in Test Structure
test/utils/utils.go: Helper functions for node management, VM control, job verification

Test Fixtures

examples/k3d/nodes1.yaml: Node CR definitions (agent-0, agent-1)
examples/k3d/hello-world-deployment.yaml: Test workload (nginx deployment)
examples/k3d/vm_control_server.py: HTTP server for node power control

Runtime Files

/tmp/vm-control-requests.log: VM control server request log
/tmp/local-server.pid: Local server PID file
/tmp/vm-control.pid: VM control server PID file
./kubeconfig: Cluster kubeconfig (created by install-dev.sh)

Test Structure​

Architecture Components​

1. VM Control Server (examples/k3d/vm_control_server.py)​

2. Homelab Autoscaler Controller (cmd/main.go)​

3. Kubernetes Cluster Autoscaler (Helm chart)​

4. Node Custom Resources (examples/k3d/nodes1.yaml)​

5. Service Forwarders (created during test setup)​

6. Test Utilities (test/utils/utils.go)​

Test Execution Flow​

Phase 1: Cleanup (the head of the test-e2e target)​

Phase 2: Environment Setup (./development/install-dev.sh)​

Phase 3: Start VM Control Server​

Phase 4: Start Local Controller​

Phase 5: Run E2E Tests​

The ordered container's BeforeAll: the pool the run starts from​

Test 1: Setup & Environment Validation​

Test 2: Scale-Down Workflow​

Test 3: Scale-Up Workflow​

Test 4: Scheduling With Slot Labels​

Phase 6: Cleanup (the cleanup-e2e target, run by test-e2e at the end)​

Key Integration Points​

gRPC Communication Flow​

Node Lifecycle Flow​

Test Validation Points​

Test 1: Setup & Environment Validation​

Test 2: Scale-Down Workflow​

Test 3: Scale-Up Workflow​

Environment Variables​

Test Constants​

Expected Runtime​

Test Tags​

Comparison: test-e2e vs run-e2e​

Files Involved​

Test Orchestration​

Test Implementation​

Test Fixtures​

Runtime Files​

Test Structure

Architecture Components

1. VM Control Server (`examples/k3d/vm_control_server.py`)

2. Homelab Autoscaler Controller (`cmd/main.go`)

3. Kubernetes Cluster Autoscaler (Helm chart)

4. Node Custom Resources (`examples/k3d/nodes1.yaml`)

5. Service Forwarders (created during test setup)

6. Test Utilities (`test/utils/utils.go`)

Test Execution Flow

Phase 1: Cleanup (the head of the `test-e2e` target)

Phase 2: Environment Setup (`./development/install-dev.sh`)

Phase 3: Start VM Control Server

Phase 4: Start Local Controller

Phase 5: Run E2E Tests

The ordered container's `BeforeAll`: the pool the run starts from

Test 1: Setup & Environment Validation

Test 2: Scale-Down Workflow

Test 3: Scale-Up Workflow

Test 4: Scheduling With Slot Labels

Phase 6: Cleanup (the `cleanup-e2e` target, run by `test-e2e` at the end)

Key Integration Points

gRPC Communication Flow

Node Lifecycle Flow

Test Validation Points

Test 1: Setup & Environment Validation

Test 2: Scale-Down Workflow

Test 3: Scale-Up Workflow

Environment Variables

Test Constants

Expected Runtime

Test Tags

Comparison: `test-e2e` vs `run-e2e`

Files Involved

Test Orchestration

Test Implementation

Test Fixtures

Runtime Files