Compose Today, k3s Only If It Earns Its Place - A Capacity Decision Record
- infrastructure
- capacity
- compose
- architecture
Every infrastructure team eventually gets asked the same question: when do you move to Kubernetes? For a small company running several products on one server, the honest answer is a set of conditions, not a date. This post writes down ours.
Norditco serves production from Docker Compose on a single VPS. Learn Persian with Sepideh, the norditco.com hub, Midad, Farsi text-to-speech, an API and an operator dashboard all share a host with roughly 1.9 GiB of RAM and about 40 GB of disk. Kubernetes, in its lightweight k3s distribution, is a credible future option for us. It is not in production today, and Compose remains the single source of truth for what serves traffic. What follows explains why that is the right call right now, what evidence would change it, and how we gather that evidence without putting a visitor-facing product at risk.
If you want the context on how the box is run day to day, our earlier post on edge routing, cold apps and a lean footprint covers the front door, the on-demand layer and the reliability work. This post picks up where that one ends, at the capacity decision.
The decision in one paragraph
Production stays on Compose. A k3s pilot becomes eligible only after the host has headroom, the edge design is settled, rollback is rehearsed and a specific workload shows a need that Compose cannot meet without disproportionate cost. Until then, any cluster experimentation happens in an environment that has no network path to production and no shared state with it. The rest of this article is the reasoning behind each of those clauses.
Start from the footprint, not the tooling
Capacity decisions go wrong when they start with a tool and work backward to a justification. We start from measurements.
When we wrote the server roadmap, the constraints were concrete:
| Constraint | Measured reality | What it implies |
|---|---|---|
| Memory | About 1.9 GiB total, around 0.7 GiB available under load, swap already in use | Any new always-on component competes directly with product containers |
| Disk | About 40 GB, mostly full of images and content | Image pulls and cluster state need explicit room before they are safe |
| Always-on agent budget | Under 100 to 150 MB resident for ops tooling | The control plane for our own operations must stay thin |
| Redundancy | One host, one node | A cluster here would orchestrate a single machine |
Two rows of that table matter most for the Kubernetes question.
The first is memory. A k3s server is the lightest mainstream Kubernetes distribution, and it is still a real control plane: an API server, a datastore, a scheduler, controllers, a container runtime shim and typically an ingress controller and metrics components. Even tuned, that is a standing cost in the hundreds of MiB on top of the workloads. On a host where we were already swapping, adding it would have meant either starving products or accepting the very out-of-memory events we had just worked to eliminate.
The second is the last row. Much of Kubernetes' value comes from spreading workloads across nodes, rescheduling them when a node fails and rolling changes through a fleet. On one node there is nothing to spread across and nowhere to reschedule to. A single-node cluster still gives you declarative manifests, rolling updates and a uniform API, which are real benefits. Compose gives us a workable version of most of those for a fraction of the memory.
What Compose gives a small multi-product company
It is easy to treat Compose as the thing you outgrow. For our situation it is the right-sized tool, and it is worth being specific about why.
- One file describes the service set. Each product's containers, volumes, restart policies and memory limits live in version-controlled Compose definitions. A reviewer can read what runs on the server in a few minutes.
- The failure modes are familiar. When something goes wrong, there is a container, a log stream and a restart policy. There is no scheduler, admission chain or CNI plugin to rule out first.
- Memory limits and restart policies cover our blast-radius needs. After the reliability work, always-on product containers have explicit ceilings, and a core container that restarts repeatedly raises an alert.
- Cold apps already scale to zero. Our on-demand layer stops idle containers and wakes them on the first request. That is the single most valuable resource-efficiency feature a cluster could have offered us, and it is already running on the existing stack.
- Rollback is a short list of known steps. Previous images are retained, and checkpoints record the service set, image versions and the ops repository revision. Restoring means going back to a named state and verifying the public URLs.
That makes Compose sufficient for our workload shape: a handful of services, uneven traffic, one operator team and no multi-node requirement.
The traffic shape argues against a cluster
On a representative day in late October, norditco.com saw about 780 requests, Learn Persian about 360, the API about 120, the monitoring dashboard about 75, TTS about 27 and Midad about 5. Those are small numbers. Kubernetes autoscaling exists to answer a question we do not currently have: what happens when one service needs to go from two replicas to twenty.
Our actual questions are different. Which of these workloads is worth keeping resident at all? Which should sleep until someone asks for it? Where does an extra 130 MiB for a speech engine used a handful of times a day belong? We answer those with a classification, not a scheduler:
| Class | Meaning | Policy |
|---|---|---|
| Hot | Daily traffic matters | Always on, memory capped, latency watched |
| Warm | Needed but quiet | Always on for now, strict cap, reviewed monthly |
| Cold | Rare traffic, non-zero RAM | Sleeps when idle, wakes on first request |
| Tax | Operational overhead such as monitoring | Slim first, relocate if still heavy |
A cluster does not change that table. It would change where the policies are expressed. Moving them into Kubernetes primitives would cost memory and add moving parts without altering which workloads are hot or cold.
What Compose does not give us
The honest version of this argument includes the costs of staying put.
- A single host is a single point of failure. Compose does not fix this and neither would a one-node cluster. Our mitigations are fast restore, honest alerting and a small always-on set. They reduce the blast radius of a bad change. They are not redundancy.
- Cold start latency is real. The first request after idle time waits for a container to start. We measure it instead of hiding it.
- Declarative convergence is weaker. Compose applies what you tell it to. It does not continuously reconcile drift the way a controller loop does. We cover that with checkpoints, pinned image tags recorded in inventory and health contracts.
- Network policy and workload identity are coarser. Compose gives us private networks and loopback-only port binding. Kubernetes can express finer east-west rules. With a small number of first-party services on one host, the coarser model is an acceptable trade today.
- The ecosystem pulls toward Kubernetes. Many tools, operators and tutorials assume it. That is a real long-term cost, and it is part of why we keep the option open and keep measuring.
Naming these costs is what makes the exit gates below meaningful. A gate is only worth writing down if there is a genuine reason to expect it might eventually pass.
Exit gates: what would have to be true
We treat "adopt k3s" as a decision that has to be earned through a sequence of gates, not a preference to be justified after the fact. Each gate has a measurable condition, and failing any one stops the process. The gates map to the phases of our server roadmap.
Reading the tree top to bottom, five questions decide the outcome.
Gate 1: Footprint is under control
Before anything new runs on the host, the host has to be healthy without it. That means the right-sizing work is done: every service has a class and a memory policy, at least one cold-app lever is applied and calm for 48 hours, and the available-memory trend is stable or improving with no sustained swapping.
We have made real progress here. Memory ceilings on always-on products, a lean monitoring core, the dashboard moved to on-demand and an honest deploy health gate together took available memory under load from roughly 0.7 GiB to roughly 1.2 GiB. That is a good result and it is also exactly why we cannot spend the headroom casually. The rule we wrote down is simple: do not expand a platform while the host is swapping.
Gate 2: The edge and the rollback story are settled
A cluster introduces an ingress layer, and two components terminating TLS for the same hostnames is a reliable way to create an outage. So the edge decision has to be recorded first: keep nginx, move to something else, or put a cluster ingress in front. Our current default is to keep nginx as the public edge and, if a cluster ever arrives, have nginx forward to it rather than handing over the front door.
Rollback has to be proven before it is needed. We require a rehearsed restore from a named checkpoint, with a target of under 10 minutes to return a migrated hostname to its Compose-served state and verify the public URLs. An untested rollback is a hope, and we do not start a migration on hope.
Gate 3: A workload has a need Compose cannot meet without disproportionate cost
This is the gate that most often decides these discussions for lean platforms, and it is the one that keeps us honest. "Kubernetes is the industry standard" is not a workload need. A workload need looks like one of these:
- A service that has to run as several replicas with rolling updates and automatic replacement, because a single container restart is a user-visible outage.
- A job or worker pool whose demand is bursty enough that horizontal autoscaling would save meaningful money or protect latency.
- A requirement for fine-grained network isolation between services that Compose networks cannot express.
- A move to multiple nodes, where scheduling and failover are the point.
If none of those is true for any of our workloads, the cluster would be infrastructure in search of a problem. Today, none of them is.
Gate 4: A capacity option is chosen and funded
If the first three gates pass, we choose explicitly among three options and write the choice down:
| Option | When it fits |
|---|---|
| Stay on Compose | Headroom is fine and no workload need has appeared |
| Resize this VPS, then run a pilot | We want a cluster behind the same public addresses and the larger instance is justified |
| Add a second small node | We want to protect the current host and run the cluster, or cold apps, elsewhere |
Resizing means moving to at least 4 GiB of RAM and more disk, and the budget has to be approved before the change, with a checkpoint taken first. The second-node option is attractive because it protects the production host from the new control plane's memory cost, and it is the first step toward a topology where a cluster actually has more than one node to schedule across.
Gate 5: The pilot passes on a non-critical app
Even with every earlier gate passed, the first workload to move is deliberately boring. The pilot candidate is a cold or warm application, such as the speech service or the backend, and not the monitoring stack, the content site or the hot product. The conditions to pass:
- The pilot is healthy for at least seven days.
- Returning to the Compose path takes under 10 minutes and has been drilled.
- Resident memory and CPU of the pilot, plus the control plane, are within the budget we approved.
- The Compose definition is retained throughout, so the fallback is a hostname flip and not a rebuild.
Only after that do we consider moving anything else, one application per checkpoint, with hot applications last and the content site and monitoring last or never on the cluster. A migrated application needs 48 hours without a rollback before the next one starts.
The isolated lab and how it relates
A fair question follows from all of this. If the gates are strict, how do we know whether a cluster would actually meet them? We cannot evaluate a platform purely on paper, and we will not evaluate it on production.
So we maintain an isolated lab: a disposable cluster that runs on a developer workstation, entirely separate from the production VPS. It is a controlled environment for generating evidence against the gates, and the isolation is the main design property.
The boundaries are written down as rules, not left to habit:
- No cluster on the production VPS. The lab never runs there, and nothing in the lab depends on it.
- No product hostnames point at the lab. It has no public ingress and serves no real traffic.
- No shared credentials or state. Lab secrets, if any, are throwaway and local. Production secrets never enter it.
- No production data. Anything the lab runs is a stand-in workload or a copy of a service that carries no real user data.
- Everything that matters lives in git. Manifests and runbooks are version controlled, so the lab can be torn down and recreated at will, and its results are reproducible by anyone on the team.
The only thing that crosses the boundary is information. The lab produces evidence, and the gate review consumes it.
What the lab is for
Each lab exercise maps to a question one of the gates asks. We have run the basics a production workload would depend on:
- Rollouts and rollback. Rolling updates with zero unavailable replicas, a deliberate bad image, and a rollback to the previous revision. The takeaway is a measured sense of how long a rollout takes and that a brief error window during replacement is expected until new replicas report ready.
- Probes and self-healing. Readiness and liveness checks, and the behavior when a pod is deleted: the controller recreates it and traffic resumes.
- Persistent storage. A small volume that survives pod replacement, which is the prerequisite for any stateful service we might ever consider.
- Network policy. A default-deny policy and a narrow allow rule, with checks that permitted callers succeed and others are refused. This tells us whether the isolation benefit mentioned in Gate 3 is real in our setup.
- Autoscaling. A synthetic load against a deployment with resource requests, scaling replicas up and then back down. This tells us whether horizontal scaling is a meaningful lever at our scale.
- Ingress and TLS. Host-based routing and a local certificate path, which feeds directly into the edge decision in Gate 2.
The lab also tells us what a cluster costs. We want a measured footprint for the control plane plus a pilot workload, because the central question of Gate 1 is whether the memory exists to run it. Those numbers are collected in an environment where being wrong costs nothing, and compared against the budget of the production host.
The next lab step is to run a copy of one of our small cold applications in its own namespace on the workstation cluster. That tells us whether the real manifests, health contract and wake behavior translate cleanly, without moving the production service. Production for that application stays on Compose until the gates say otherwise.
What the lab is not
It is not a staging environment, because it does not mirror production's host, network or data. It is not a migration rehearsal, and it is not a commitment. A lab that finds a cluster too expensive for our footprint has done its job.
Why we keep a cluster option open at all
If the gates are demanding and the lab runs on a laptop, why not close the question permanently? Because the conditions that make Compose the right answer are properties of a moment, not of the company.
A second node, a larger instance, a product with real replica requirements or a team that outgrows a single operator would each change the calculation. We would rather have the evidence and the procedure ready than be forced into an unplanned migration. The footprint work did two things for that: it made the host healthier today, and it cleared the memory headroom that any future cluster decision depends on.
The same principle applies to the edge, where we trialed a candidate on one low-traffic hostname. A new component has to justify its resident cost, operational surface and blast radius against what it replaces.
What we measure to keep the decision honest
A capacity decision that is never revisited turns into dogma. These are the signals that trigger a review:
- Available memory trend and swap activity, reviewed weekly. A sustained drop reopens the footprint conversation before any platform conversation.
- Requests versus memory per application, reviewed monthly. A cold app whose traffic grows graduates to always on, and a workload that outgrows a single container becomes a candidate for Gate 3.
- Cold start time for each on-demand app, and the share of requests that hit a sleeping container.
- Restart counts for core containers, the earliest signal that a memory cap is too low or a workload is outgrowing its slot.
- Time to restore from a checkpoint, measured in drills. A quarterly restore drill keeps the rollback gate honest.
- Incident cause. If a real outage would have been avoided by multi-node scheduling or automated replacement, that is direct evidence for the cluster case, and we record it.
If these numbers move the right way, the next capacity review will be short. If they do not, the gates tell us what to do next, and the answer might be a bigger instance, a second node or a smaller always-on set, with Kubernetes only one of several outcomes.
Where this leaves us
Today, Norditco serves production from Docker Compose on one VPS, and a cluster would add cost and complexity without removing a problem we have. The 1.2 GiB of headroom we recovered is worth more as breathing room for visitors than as a down payment on a control plane.
Kubernetes is neither rejected nor scheduled. It is a decision with a defined set of conditions, an isolated environment that produces evidence against them, and a pilot plan that starts with the least important workload and keeps the old path one step away. When a workload needs replicas, when we add a node, or when the numbers show a cluster fitting inside the footprint, we will run the gates, publish the outcome and either proceed with a pilot or record why not.
Until then, the most useful thing we can do for capacity is keep the always-on set small and the rollback path fast.
