The Full
Stack Explained

Before touching a model file, we need to know exactly what we're working with. GPU model, VRAM, system RAM, storage speed, CPU architecture — every spec influences which models will run well and which will struggle.

• GPU VRAM is the primary constraint — it determines max model size
• System RAM matters for models that overflow to CPU offload
• NVMe read speed affects model load time (42GB+ files)
• Multi-GPU setups unlock tensor parallelism for larger models
• We check CUDA version, driver state, and thermal headroom

Not every model is right for every use case. A 70B general-purpose model is overkill for a focused code assistant; a 7B coding model won't hold up for complex document analysis. We match capability to need.

• General chat / assistant → Llama 3.3 70B or Mistral Small
• Code generation / completion → Qwen 2.5 Coder 32B
• Fast RAG retrieval → Mistral 7B or Phi-4 14B
• Complex reasoning → DeepSeek R1 or Phi-4
• Embeddings → nomic-embed-text or bge-m3 (run locally)

A full-precision 70B model weighs ~140GB — impossible on a single consumer GPU. Quantization reduces each weight from 16-bit to 4-8 bits, shrinking file size dramatically with minimal quality loss if done correctly.

• Q4_K_M is our default — best size/quality balance for most use cases
• Q8_0 for maximum quality when VRAM allows
• Q2_K only for CPU-only machines with severe memory limits
• GGUF format for llama.cpp/Ollama; AWQ for vLLM
• We benchmark quality drop before committing to a quantization

A model running in a terminal isn't a production system. We configure a proper inference server with process management, automatic restarts, health checks, logging, and optional load balancing for multi-GPU or multi-instance setups.

• Ollama for ease-of-use and multi-model management
• vLLM for maximum throughput on high-traffic deployments
• llama.cpp server for lean CPU/hybrid setups
• systemd service with restart policies and logging
• Optional: nginx reverse proxy with auth and rate limiting

The final layer is an OpenAI-compatible REST API. Your existing apps, scripts, and integrations can point at your local endpoint — same request format, same response format. One URL change and you're off cloud.

• Identical to the OpenAI API spec — /v1/chat/completions
• Streaming support (SSE) out of the box
• Works with any OpenAI SDK — Python, Node, Go, etc.
• Optional: Open WebUI for a browser-based chat interface
• Optional: basic API key auth for multi-user environments