Five steps from bare metal to a running LLM endpoint. No black boxes — here's exactly what we do, why we do it, and what you end up with.
Every deployment follows the same sequence. Each step feeds directly into the next — skipping any of them is how deployments go wrong. We've refined this across enough installs to know exactly where the edge cases are.
Before touching a model file, we need to know exactly what we're working with. GPU model, VRAM, system RAM, storage speed, CPU architecture — every spec influences which models will run well and which will struggle.
Not every model is right for every use case. A 70B general-purpose model is overkill for a focused code assistant; a 7B coding model won't hold up for complex document analysis. We match capability to need.
A full-precision 70B model weighs ~140GB — impossible on a single consumer GPU. Quantization reduces each weight from 16-bit to 4-8 bits, shrinking file size dramatically with minimal quality loss if done correctly.
Rule of thumb: at Q4, a model needs roughly 0.6 GB of memory per billion parameters, plus a few GB for context. A 27B model fits a 24 GB card; a 70B model needs about 48 GB.
A model running in a terminal isn't a production system. We configure a proper inference server with process management, automatic restarts, health checks, logging, and optional load balancing for multi-GPU or multi-instance setups.
The final layer is an OpenAI-compatible REST API. Your existing apps, scripts, and integrations can point at your local endpoint — same request format, same response format. One URL change and you're off cloud.
Each layer serves a specific purpose. Together they give you a private, fast, fully-owned inference endpoint that any application can call — with no changes to how you already write code.
| MODEL | SIZE | BEST FOR | STATUS |
|---|---|---|---|
| Qwen3 27B | 27B dense | General / Chat | PROD READY |
| Gemma 4 | 26B A4B MoE | RAG / Fast | PROD READY |
| Qwen3-Coder / Devstral | 24–32B | Code Gen | PROD READY |
| gpt-oss | 20B / 120B MoE | Reasoning | PROD READY |
| Llama 4 Scout | 109B MoE · 17B active | Long Context | HIGH VRAM |
| DeepSeek V4 | Large MoE | Research | HIGH VRAM |
Last resort when VRAM is severely limited. Noticeable degradation on complex reasoning tasks.
Best balance of size and quality. A 70B needs two 24GB GPUs at Q4; a 27–32B model fits a single RTX 4090. Hard to tell apart from Q8 in practice.
Marginal improvement over Q4 for most tasks. Worth it on multi-GPU setups where VRAM isn't a constraint.
Effectively full quality. Requires 80GB+ VRAM for 70B. Overkill for most production workloads.
Now you know the stack — let's run it on yours. Tell us your hardware and use case and we'll have you live inside 48 hours.