Project — Active
Local
LLM.
Self-hosted AI inference with llama.cpp (llama-server) on 3x NVIDIA Tesla V100 GPUs (96GB VRAM). A resident 35B MoE model at ~95 tokens/sec, plus vision and embedding endpoints, serving development, personal assistant, and content generation — zero cloud dependency.
llama.cpp / AI / Self-hosted / NVIDIA V100
- ~95 tok/s
- Inference speed (measured)
- 100%
- Data stays local
- 4
- Parallel AI sessions
01 — Configuration
The hardware.
02 — Case study
GPU-accelerated local AI infrastructure.
The challenge
Run production AI inference for a personal assistant (Nova), 4 parallel AI coding workers, blog generation, and voice transcription — without recurring cloud API costs eating into a bootstrapped business budget.
The solution
- Deployed 3x NVIDIA Tesla V100 GPUs with llama-server (llama.cpp) serving a 35B MoE model across all 3
- Model chosen by measured head-to-head eval against larger dense models — the 35B MoE beat a 27B dense by 2.5x on generation at the same quant
- Separate vision endpoint (27B + mmproj) added for screenshot review, kept advisory so it cannot gate a release on its own
- GPU-accelerated faster-whisper for voice transcription on a dedicated GPU
- Integrated with Claude Code workers — summaries and analysis patterns run locally for free
- Nova personal assistant routes queries through 3 tiers: regex (free) → local LLM (free) → Claude API (paid)
- Weekly blog posts auto-generated on the same resident model
03 — Available models
What's loaded in VRAM.
qwen3.6-35b
35B MoE (~3B active) / Always loaded across 3x V100 — no cold starts
The resident workhorse — coding, reasoning, tool calling, Nova assistant, blog generation, classification. Text only, by design.
qwen3.8-27b-vision
27B dense + mmproj / GPUs 0+1 (~26GB)
Screenshot and image review for the app release pipeline. Advisory only — it never issues a pass/fail verdict on its own.
nomic-embed-text-v1.5
Embedding model / Resident alongside the workhorse
Vector embeddings for semantic search across the workspace and content pipeline.
04 — Benefits
Why bother self-hosting.
Cost Reduction
Free inference for simple tasks that would otherwise use paid API calls. Saves money on summarization, status checks, and simple questions.
Privacy
Sensitive code and data never leaves the local network. No cloud provider sees your prompts or responses.
Speed
Local inference with no network latency. Responses start immediately without waiting for API round-trips.
Availability
Works offline and during API outages. Not dependent on external service availability.
05 — Integrations
Wired into everything.
06 — Resources