From guild operator to edge: how the games estate fits together
Five views of the same system, wide to narrow. Two games — iwars and
pontu — are built by an in-cluster CI, deployed by GitOps, spun up
per-guild by a Kubernetes operator, fed art by a GPU/NPU asset factory on the
workstation, and published to the internet through a hardened edge box with zero
inbound connections to the home network.
edge / public kubernetes / cluster CI / GitOps AI / asset generation hardware / hosts identity / data
View 01 · wide shot
The estate at a glance
Everything public flows through napuka, a rented dedicated server that
reverse-proxies into the homelab over WireGuard — the home network accepts no inbound
connections at all. Inside, the xlab Kubernetes cluster runs the games
and guild services, Gitea and Jenkins drive builds, and the zlab
workstation is the AI asset factory whose output lands back in git like any other change.
A guild is a single GuildSiteInstance custom resource. The owner
registers and creates it through the portal; the Go API translates that into the CR;
the operator reconciles it into a per-guild namespace holding a Matrix homeserver,
a chat client, a database, both games' web and websocket servers, and a leaderboard.
Identity is shared: OpenLDAP holds accounts and groups, Dex fronts them with OIDC
for every client (portal, API, games) using cross-client audiences.
Portal: zekup.xlab.expansible.io — registration 201 via LDAP registrarPromotion: portal PATCH re-pins a guild's game build tag, pods rollChat: Synapse server_name is immutable — chosen once per guildweb / ws: independently optional per guild in the CR spec
View 03 · build and deploy loop
CI & GitOps: push once, everything follows
A push to Gitea fires a webhook into in-cluster Jenkins, which schedules ephemeral pod
agents on whichever architecture the job needs — including a Podman agent on the zlab
workstation for heavy ROCm image builds. Images land in the Gitea registry;
ArgoCD Image Updater notices new tags and writes the pin back into
git, where the argo-apps app-of-apps syncs it onto the cluster. No
one deploys by hand: the cluster is whatever k8s/live/ says it is.
Secrets: SOPS + age, decrypted by Argo CD at sync timeRegistry creds: gitea-registry JCasC credential shared across 7 reposVersion scheme: tag = build number; Image Updater ranks by push time, not semverRule: no kubectl apply on managed workloads — changes go through git
View 04 · asset factory on zlab
Asset factory: prompt in, committed game art out
The Asset Manager (React SPA + FastAPI, a rootless Podman quadlet on
zlab) is the front door for all generated art: 3D race figures, terrain maps, and each
game's UI art. Its scheduler fills one GPU slot and two CPU slots, cheapest jobs first,
and dispatches to a single ComfyUI instance — one container only, because two on the
Radeon 890M silently degrade kernels mid-generation — or to the Lemonade stack for fast
Flux text-to-image and NPU-hosted LLM work. Promotion is the key move: it applies each
game's size and post-process SSOT, enforces the triangle budget, and drops finished
assets straight into the game repos, where the CI loop from View 03 takes over.
Deploy: image-baked — Jenkins builds iwars-asset-manager:0.1.N, re-pin the quadletDiscipline: drain the generation queue before any restart — overnight qwen batches run longProvenance: gallery tracks origin; "IN GAME v{n}" badge follows the live promoted sampleModels: checkpoints, controlnets, LoRAs, VAEs under comfyui/models/, bind-mounted
View 05 · iron and wires
Hosts & network: what it all runs on
The control plane is three KVM virtual machines on one Ryzen mini-PC hypervisor,
joined over a WireGuard mesh by an ARM worker and a RISC-V worker, each tainted so
only build jobs land there. zlab is both the owner's workstation and the GPU/NPU
inference host; the Synology NAS is the lone Docker holdout in an otherwise
Podman-everywhere estate. napuka reaches the cluster only through its own
WireGuard tunnel.
Provisioning: Ansible (site.yml) — VMs, WireGuard, kubeadm, Cilium, Argo CDStorage: Longhorn on the VM nodes, raw LVM on Kingston NVMeObservability: Prometheus + Grafana + Loki; systemd node_exporters on hosts