/ Hardware

The rigs behind the numbers

Every endpoint on the live dashboard runs on one of these five machines, sitting on one LAN. 11 GPUs across five hosts, no cloud instances, no rented capacity. Specs below were read off each host directly — CPU, board, memory and GPU lines come from the machine, not from a spec sheet.

FORGE

.233:8001

The fastest single-stream engine in the fleet and the only Blackwell card in it. One 5090 holds the whole 26B checkpoint with room for an fp8 KV cache, so there is no tensor-parallel sync cost per layer — which is most of why it beats the dual-3090 engines on a single request.

GPUs

1× RTX 5090 (Blackwell, SM120)

32 GB VRAM

Engine

vLLM, single GPU, Docker

131,072 ctx

Model served

gemma4-26b · AWQ W4 + MTP draft

Host

AMD Ryzen 5 3600X (6C/12T)

ASRock X570 Phantom Gaming 4 · 16 GB DDR4

gemma4-26b 348.1 tok/s

TITAN

.230:8001 (Engine A) · .230:8002 (Engine B)

Four cards run as two independent TP=2 engines rather than one TP=4 engine: Engine A on GPUs 1+2, Engine B on GPUs 0+3. That costs some single-request speed and buys two separate failure domains and two batching pools, so one engine can be restarted without taking the model offline. All four cards negotiate PCIe 3.0 x8.

GPUs

4× RTX 3090 (Ampere, SM86), 250 W cap each

96 GB total · 48 GB per engine VRAM

Engine

vLLM TP=2 × 2 independent engines, Docker

131,072 per engine ctx

Model served

gemma4-26b · AWQ W4 + MTP draft

Host

Intel i9-7920X (12C/24T)

MSI MEG X299 Creation · 48 GB DDR4

Engine A 235.9 tok/s Engine B 242.7 tok/s Full build + bill of materials →

SCOUT

.234:8002

Mixed 3090 Ti and 3090 pair running as one TP=2 engine. Both cards sit on PCIe 3.0 x16, which is why it edges out TITAN on single-stream decode despite the same generation of silicon. Rebuilt on 2026-09-10 to run the identical container image as TITAN after an older vLLM build silently fell back to a slower MoE kernel.

GPUs

1× RTX 3090 Ti + 1× RTX 3090, 265 W cap each

48 GB VRAM

Engine

vLLM TP=2, Docker

131,072 ctx

Model served

gemma4-26b · AWQ W4 + MTP draft

Host

Intel i9-7900X (10C/20T)

ASRock X299 Steel Legend · 32 GB DDR4

gemma4-26b 250.5 tok/s

HYDRA-R

.231:8000 (GPU 0) · .231:8001 (GPU 1)

The only AMD box in the fleet, and the only one not on vLLM. It runs a GGUF build under llama.cpp with no speculative decoding, which is the main reason its per-request speed is roughly a third of the vLLM engines. It holds up fine under parallel load and serves as overflow capacity rather than the primary path.

GPUs

2× AMD Radeon AI PRO R9700 (RDNA4, gfx1201)

64 GB total · 32 GB per GPU VRAM

Engine

llama.cpp + HIP, one engine per GPU, 2 slots each

32,768 per sequence ctx

Model served

gemma4-26b · Q4_K_M GGUF

Host

Intel i9-7920X (12C/24T)

ASUS Pro WS X299 Sage II · 32 GB DDR4

GPU 0 75.6 tok/s GPU 1 75.2 tok/s

NEST

.222:8001 (GPU 0) · .222:8002 (GPU 1)

The small-model tier. 12 GB per card is not enough to hold the 26B checkpoint with a usable KV cache, so NEST serves an 8B model for memory and retrieval work instead of competing on headline tok/s. Its numbers should be read against the other 8B-class rows, not against the 26B engines.

GPUs

2× RTX 3060 12 GB, 130 W cap each

24 GB total · 12 GB per engine VRAM

Engine

vLLM, one single-GPU engine per card

20,480 ctx

Model served

Qwen3-8B-AWQ

Host

Intel i9-7960X (16C/32T)

ASUS PRIME X299-A II · 32 GB DDR4

GPU 0 38.2 tok/s GPU 1 37.3 tok/s

Also on the dashboard

Ollama Cloud

The Ollama Cloud rows aren't a rig — they're a managed cloud route, probed with the same prompt on the same schedule so there's a like-for-like reference point next to the local engines. Their timing is measured server-side rather than from the engine's own counters, which the methodology page explains.