How we designed the architecture for RACoN

How we designed the architecture for RACoN

Running large models locally usually hits a hardware wall pretty fast. One machine has a decent GPU with limited VRAM, another has plenty of system memory, and a third is just sitting idle with spare disk space and CPU cycles. With RACoN (Reciprocal Agentic Compute Network), we wanted to see if we could pool these mixed machines together and expose them as one logical AI inference service.

Designing a system like this means separating cluster management from raw tensor math. Here is how we laid out the architecture.

The core and control plane

Everything starts with the control plane. Before a cluster can run a single token of inference, it needs to know what hardware actually exists on the network.

The RACoN core handles node registration and keeps a live hardware inventory. When a worker joins, the control plane logs its specs, tracks its health status, and maps out the network topology. It also manages model availability, task assignments, failure recovery, and credit accounting so the network knows who contributed what.

Scheduling across different hardware

Most cluster schedulers assume homogeneous server racks where every node looks identical. Home labs and office networks are the exact opposite. You might have PC A acting as a CPU and RAM worker, PC B holding the GPU and VRAM, and PC C providing extra RAM and storage.

Our heterogeneous resource scheduler looks at the actual cost of a workload and matches it to the hardware on hand. It decides which machine should execute each part of a model by weighing available CPU, GPU/VRAM, system memory, network bandwidth, and link latency. If a node is behind a slow connection, the scheduler accounts for that before assigning tensor operations across the wire.

The inference engine

We decided early on to start with an existing inference backend instead of writing model execution from scratch. Building custom CUDA or Metal kernels from day one would slow down the actual distributed systems research.

Our initial direction uses llama.cpp and its ggml-rpc capabilities to handle low-level execution across workers. Once the coordination layer is solid, we plan to bring in vLLM at a later stage for higher-throughput batching and serving.

Model repository and placement

Splitting a model across multiple computers requires strict bookkeeping. The model repository layer tracks model files, compatible formats, and quantization levels across the cluster. It maps out model partitions and keeps a record of which specific nodes hold the required tensors or weights on disk, avoiding unnecessary file transfers before inference starts.

Dealing with the distributed KV cache

During inference, the attention key/value (KV) cache grows with the context length and quickly becomes a memory bottleneck. In a distributed setup, remote RAM is not automatically equivalent to local RAM. Moving cache blocks over a standard network connection introduces real latency.

Rather than treating cluster memory as one flat pool, RACoN includes a dedicated distributed KV cache coordinator. It enforces explicit ownership and transfer rules for cache allocation and lifecycle management across the inference data plane. Workers know exactly which node owns which context state and when tensors need to move.

Recovery and resource accounting

When you run workloads across normal desktop machines, nodes will drop offline. Someone closes a laptop, a process crashes, or Wi-Fi fluctuates.

The recovery and accounting layer watches for disconnected workers, reassigns failed tasks, and adapts when available system resources change mid-run. Alongside fault tolerance, it measures actual compute usage and issues credit receipts. Because RACoN is built as a reciprocal network, tracking resource contributions accurately is baked directly into the core architecture.

One service, multiple interfaces

All of this coordination happens underneath a single logical inference service. We designed the core to run headless with a shared API so you can interact with the cluster however you prefer. Whether you are scripting against the CLI, monitoring nodes in an interactive shell/TUI, opening the web UI, or running the Tauri desktop app, every client talks to the same underlying API.

The goal is simple: let the control plane and data plane handle the messy reality of distributed hardware so the end user just sees a model that runs.