Frontier Models Comparison
A practical comparison of major LLM families for selecting the right model for different use cases. The landscape evolves rapidly - model capabilities, pricing, and rankings shift frequently.
Key Facts
- Scaling laws: more data + more parameters = better results (GPT-4 training estimated ~$100M)
- Closed-source leaders: GPT-4o, Claude 3, Gemini 1.5 Pro
- Open-weight leaders: Llama 3, Mistral, DeepSeek
- Chatbot Arena (LMSYS) is the most reliable benchmark - crowdsourced human preference via blind comparison
- Open-source models are improving rapidly - the capability gap with closed models is narrowing
Model Families
GPT (OpenAI)
| Model | Key Feature |
| GPT-3 (175B) | Few-shot learning, in-context learning |
| GPT-4 | Multimodality, best-in-class reasoning |
| GPT-4o | Omni-modal: text, images, audio, video |
| GPT-4o-mini | Cost-efficient for simpler tasks |
Claude (Anthropic)
- Constitutional AI (CAI) - model critiques itself against principles
- Family: Haiku (small/fast), Sonnet (balanced), Opus (most capable)
- Strengths: very long context (200K tokens), document analysis, careful reasoning
- Best for safety-sensitive applications
| Model | Sizes | Training Data | Achievement |
| Llama 2 | 7B, 13B, 70B | 2T tokens | RLHF safety, commercial license |
| Llama 3 | 8B, 70B, 405B | 15T tokens | 85% MMLU (close to GPT-4) |
Open weights, best for privacy/local deployment and fine-tuning.
Mistral
- Models: Mistral 7B, Mixtral 8x7B (MoE), Mistral Large
- Best quality-per-parameter ratio
- Mixtral: 8 experts, 2 active per token, 47B total but ~13B active
- Sliding window attention for efficient long contexts
Gemini (Google)
- Natively multimodal (text + images + audio + video in single model)
- Gemini 1.5 Pro: 1M token context (up to 2M preview)
- Best for multimodal applications
DeepSeek
- Code-specialized models (1.3B to 33B)
- Outperforms Code Llama on many benchmarks
- Open-source
Model Selection Decision Framework
| Need | Recommendation |
| Maximum quality | GPT-4o or Claude 3 Opus |
| Privacy / local deployment | Llama 3 or Mistral (via Ollama) |
| Cost efficiency at scale | Mistral or GPT-4o-mini |
| Long document processing | Claude 3 (200K) or Gemini 1.5 Pro (1M) |
| Multimodal | Gemini or GPT-4o |
| Code generation | GPT-4o, DeepSeek Coder, Claude 3 |
| Fine-tuning | Llama 3 or Mistral (open weights) |
Context Windows
| Model | Window |
| GPT-4 | 8K / 128K tokens |
| Claude 3 | 200K tokens |
| Gemini 1.5 Pro | 1M tokens |
| Llama 3 | 8K (extended variants) |
| Mistral Large | 128K tokens |
Longer context = more information per request but higher cost and potential "lost in the middle" quality issues.
Benchmarks
| Benchmark | What It Measures |
| MMLU | Broad knowledge (57 subjects) |
| HumanEval | Code generation (pass@1) |
| MATH | Mathematical reasoning |
| GSM8K | Grade school math word problems |
| Chatbot Arena | Crowdsourced human preference (Elo scores) |
Chatbot Arena is the most reliable real-world benchmark because it measures actual user preference in blind A/B tests, not synthetic metrics.
Gotchas
- Benchmark scores don't always predict real-world task performance - always test on YOUR data
- Model pricing changes frequently - check current provider pricing before architecting costs
- "Frontier" status is temporary - today's best model is tomorrow's commodity
- Context window size doesn't mean all that context is used equally well ("lost in the middle")
- Open-weight doesn't mean free - infrastructure costs for running large models are significant
- Model versioning: providers update models behind the same API name, sometimes changing behavior
See Also
- [[ollama-local-llms]] - Running open-weight models locally
- [[model-optimization]] - Making models smaller and faster
- [[fine-tuning]] - Customizing open-weight models
- [[llm-api-integration]] - Connecting to provider APIs
- [[llmops]] - Cost optimization across models