Local-first AI, explained without the buzzwords

What "local-first AI" really means

The model runs on your machine, not someone else's. That's it. Your prompts and your responses never leave your hardware. No usage caps, no terms-of-service clauses about training on your data, no rate limits on a free tier that you hit at 11pm when you actually need them.

In practice, this means running Ollama (or vllm, or llama.cpp directly) with one of the small-to-medium open models — Llama 3, Qwen, Mistral, Gemma, DeepSeek. They're free, the weights are downloadable, and a 7B-parameter model fits on a consumer GPU with 8GB of VRAM. A 3B model fits on CPU only.

Where local wins, plainly

• Sensitive data. Medical notes, legal drafts, internal company docs — anything you'd hesitate to paste into ChatGPT. Local AI gets to read them without them leaving your network.
• Cost at scale. Heavy use of GPT-4 or Claude Sonnet 4 stacks up quickly. A small qwen2.5:7b on your own GPU costs you electricity and nothing else.
• Latency. Local models on a decent GPU return tokens in tens of milliseconds with no network round-trip. Snappy chat without TTFT spikes.
• Offline. Trains, planes, the cabin without internet. Local works.
• Habit-forming feedback. When tools cost effectively zero per query, you experiment more. We've seen Pulse users go from "I'll ask the AI carefully" to "I'll fire 30 prompts at it to figure out which one I like". That's a feature.

Where cloud still wins

• Hard reasoning. A 7B local model writes decent prose and follows instructions. It won't outsmart Claude Opus or GPT-5 on a deep code review or a novel math problem. Local is for the 80% you do every day; cloud is for the 20% that needs the big brain.
• Long context. Local context windows run 8K-32K typically. Cloud offers 200K-2M. If you're feeding entire codebases or hour-long meeting transcripts, cloud handles it more gracefully.
• Multimodality. Cloud has the strongest image / audio / video understanding. Local LLaVA is good but lags GPT-4o vision on tricky photos.
• No infrastructure. Cloud is free of "your GPU is at 100% and ComfyUI grabbed all the VRAM" problems. Pay $20/month, never think about it.

The hybrid approach (what Pulse does)

We don't pick a side. Vex (Pulse's agent) runs on local Ollama by default — qwen2.5:3b for fast chat, qwen2.5:7b for tool use. When the intent classifier detects code review, deep research, or multi-step reasoning, it routes the round to Pulse+: built-in cloud routing that brings GPT-5, Claude Sonnet 4, Gemini 2.5 Pro into the same chat without you needing to manage an API key.

This is the right shape: local for the daily 80%, cloud for the 20% that earns it. Costs stay reasonable, latency stays snappy, sensitive prompts stay home, hard problems get the model that can solve them.

"But will my GPU melt?"

A 7B-parameter model in 4-bit quantization needs about 5GB of VRAM. A 3B in 4-bit needs about 2GB. Modern integrated graphics or any consumer NVIDIA card from the last 5 years runs both fine. Apple Silicon Macs run them well too.

The actually-hard models — 70B+ — need a setup. But you don't need them for 99% of what AI assistants help with.

"What about my data sovereignty?"

In Europe and increasingly globally, "this data must stay on EU/EEA hardware" is a real legal requirement. Pulse runs in your own datacenter, on your own NAS, in your own office basement. The model never sees an Anthropic/OpenAI server. Your data never crosses a border that hasn't been audited.

This is the un-glamorous reason teams adopt local-first: not paranoia, just data locality that helps with GDPR and Schrems II concerns.

When NOT to bother

If you're a solo user who hits the AI 30 times a day and doesn't care about data sovereignty, just use ChatGPT or Claude. The friction of standing up Ollama, picking a model, configuring a frontend, and explaining why the smaller local model isn't quite as smart — that's not worth $20/month for one person.

Local-first earns its keep when you're a team, when you have data you can't ship to a US cloud, when you want to experiment without watching a cost meter, or when you want to build agents that do things on your machine (read your files, run your scripts, fire your workflows) that no cloud LLM can reach into.

What we'd build differently if starting today

Same architecture, sharper defaults. Ollama + a small smart local model + a hybrid cloud escalation tier + an MCP-like tool layer. The thing that took us the longest was figuring out how to make local models reliably usetools rather than describe using tools. The fix was the grounded auto-fires we wrote up in our other piece: fetch the data first, give it to the model as ground truth, override if it ignores you.