Round 2: 17 models vs. my homepage agent
Comparing 17 AI model variants for my homepage agent using Evalite + the Vercel AI SDK (AI Gateway). Includes OpenAI, Anthropic, Google, xAI, MoonshotAI, and DeepSeek.
Dec 22, 2025
It felt like it was time to rerun my homepage agent comparison. I last did this comparison in April 2025 by roleplaying as Sam Altman and doing a full vibes comparison by hand. While the setup was realistic, the workflow was painful: manual scoring, hard to re-run, impossible to reproduce. But I felt it was important for me to display a wow moment for important visitors... so I did the work.
Since April, everything has changed. Models are smarter, faster, and cheaper. So while it's time to pick a new base model, I have a full mix of constraints: speed, cost, quality, and vibes (the one that can move a real visitor to a meeting without being weird). Time to run another comparison.
This time I turned it into a repeatable eval suite. It’s more work up front, but the next time a model drops I just run the suite, compare the results, and decide. Under the hood it’s built with Evalite from @MattPocockuk and the Vercel AI SDK on top of the Vercel AI Gateway, so swapping the agent-under-test between providers is basically changing a model string.
TL;DR
- I'm going with Gemini 3 Pro for my homepage agent for now. It's smart, good, fast, and reasonably priced.
- Claude Opus 4.5 was the best in my opinion, but it's too expensive for my use case (especially after seeing the other prices)
- Even though the most "correct" models were GPT-5 and Grok 4.1 Reasoning. But Grok 4.1 was unreliably so (2/3 runs had failures), and GPT-5 was 2x more expensive.
- Deepseek 3.2, Gemini 3 Flash, and Grok 4.1 are alluring. They're smart and cheap.
See the full correctness vs. cost scatter below.
| Variant | Correctness | Cost (total) | Duration (total) | Evals | Placement |
|---|---|---|---|---|---|
| Grok 4.1 Reasoning | 89.3%* | $0.05 | 258.2s | 15 | 1st |
| Gemini 3 Flash | 84.9% | $0.11 | 83.1s | 15 | 2nd |
| Gemini 2.5 Pro | 84.7%* | $0.16 | 121.5s | 15 | 3rd |
| Gemini 3 Pro | 84.6% | $0.42 | 214.9s | 15 | 4th |
| Deepseek 3.2 | 82.7% | $0.04 | 314.2s | 15 | 5th |
| Grok 4.1 Non Reasoning | 80.7% | $0.05 | 114.2s | 15 | 6th |
| GPT-5 | 89.2%* | $1.09 | 749.7s | 15 | 7th |
| GPT-5.2 | 84.7%* | $0.47 | 519.7s | 15 | 8th |
| Claude Sonnet 4.5 | 85.5% | $1.82 | 298.5s | 15 | 9th |
| GPT-5 mini | 86.7%* | $0.10 | 967.5s | 15 | 10th |
| Claude Sonnet 3.7 | 87.1%* | $3.65 | 369.3s | 15 | 11th |
| Claude Opus 4.5 | 85.4% | $3.52 | 327.0s | 15 | 12th |
| GPT-4o | 77.4% | $0.39 | 334.2s | 15 | 13th |
| GPT-5 chat | 78.8%* | $0.08 | 1014.1s | 15 | 14th |
| Claude Haiku 4.5 | 65.3%* | $0.26 | 69.0s | 15 | 15th |
| GPT-5.2 chat | 68.4%* | $0.20 | 468.2s | 15 | 16th |
| Kimi K2 | 62.5% | $0.13 | 159.6s | 15 | 17th |
Why Gemini 3 Pro
My north star here is the multi-turn “can this book a meeting without being weird?” simulation, not just isolated one-turn checks. On that axis, Gemini 3 Pro was the most consistent closer in this run.
- Full suite totals: $84.55% Correct for $0.42 and ~3.6mins (overall #4).
- Simulated Convo with Sam Altman: 97% correct for at ~$0.19.
This isn’t “the best model overall.” It’s the best model for my homepage agent constraints right now.
What is correctness?
Correctness is the average score across each case in the eval suite. Each case is either testing for specific behaviors in a contrived situation or for overall quality in a multi-turn simulation.
Example: the Sam Altman simulation (multi-turn eval)
The simulation is a “Sam Altman” persona who is busy, vague, writes in lowercase, and allergic to salesiness. He only answers one question at a time, but he is here to learn about "AGI Vibes" and is open to meeting with me about it, if the agent is good enough.
This is the closest thing to “can the agent book a demo end-to-end?”:
- It runs across turns instead of grading one response in isolation.
- It tells you if the agent stalls, gets pushy, or fails to close.
- It’s expensive, so I don’t run it on every prompt tweak — but automation keeps runs consistent when I do.
Simulation Sam Results
Below is how the myriad of modern models performed on this test. Read through the conversation transcripts and judge for yourself. Brennan is my agent (the model being tested) and Sam is the simulated visitor (currently generated with openai/gpt-4o), with messages on the right.
Full results
Now, of course, Simulation Sam is only ONE example of a "simulation" eval case. There are many more cases in the full eval suite, each testing different behaviors and scenarios in different ways. More on that below, but first: the full results.
| Test | Claude Haiku 4.5 | Claude Opus 4.5 | Claude Sonnet 3.7 | Claude Sonnet 4.5 | Deepseek 3.2 | Gemini 2.5 Pro | Gemini 3 Flash | Gemini 3 Pro | GPT-4o | GPT-5 | GPT-5 chat | GPT-5 mini | GPT-5.2 | GPT-5.2 chat | Grok 4.1 Non Reasoning | Grok 4.1 Reasoning | Kimi K2 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Agent/Eng/1. entry + KB | 33.3% | 77.4% | 69.0% | 87.6% | 45.2% | 71.4% | 80.2% | 81.0% | 63.5% | 94.0% | 57.5% | 76.0% | 72.2%* | 52.9% | 57.1% | 100.0%* | 67.8% |
| Agent/Eng/2. qualifying question | 91.7% | 96.8% | 98.4% | 98.9% | 97.8% | 98.6% | 95.7% | 96.8% | 97.9% | 99.5%* | 95.2% | 91.1% | 92.9% | 60.4%* | 99.2% | 100.0% | 86.6% |
| Agent/Eng/3. RAG follow-up | 95.3% | 95.7% | 96.0% | 97.2% | 100.0% | 95.2% | 88.4% | 82.4% | 86.1% | 100.0%* | 83.3%* | 88.6%* | 94.1% | 57.3% | 96.3% | 71.3% | 58.7% |
| Agent/Eng/4. booking close | 77.8% | 100.0% | 100.0% | 88.9% | 100.0% | 100.0%* | 100.0% | 100.0% | 100.0% | 87.4% | 100.0% | 98.8% | 100.0% | 100.0% | 100.0% | 83.3%* | 66.7% |
| Agent/Flux/1. ask for name | 33.3% | 51.9% | 66.7% | 44.4% | 66.7% | 66.7% | 55.6% | 63.0% | 66.7% | 96.3% | 88.9% | 100.0%* | 70.4% | 63.0% | 33.3% | 100.0% | 74.8% |
| Agent/Flux/2. name shared | 85.7% | 85.7% | 85.7% | 85.7% | 95.2% | 85.7% | 83.3% | 85.7% | 54.8% | 85.7% | 54.8% | 83.3% | 93.7% | 57.1% | 97.6% | 92.9% | 42.9% |
| Agent/Flux/3. company shared | 76.2% | 85.7% | 98.4% | 88.9% | 84.1% | 81.0% | 93.7% | 84.1% | 76.2% | 85.7% | 71.4% | 85.7% | 84.1% | 85.7% | 87.3% | 93.7% | 42.9% |
| Simulator/Sam | 29.3%* | 90.1% | 82.2%* | 91.9% | 72.3% | 79.1%* | 82.1% | 83.4% | 74.2% | 65.2% | 79.0%* | 69.6%* | 70.2%* | 70.8% | 75.0% | 73.3% | 59.4% |
| Totals | 65.3% | 85.4% | 87.1% | 85.5% | 82.7% | 84.7% | 84.9% | 84.6% | 77.4% | 89.2% | 78.8% | 86.7% | 84.7% | 68.4% | 80.7% | 89.3% | 62.5% |
| — time | 69.0s | 327.0s | 369.3s | 298.5s | 314.2s | 121.5s | 83.1s | 214.9s | 334.2s | 749.7s | 1014.1s | 967.5s | 519.7s | 468.2s | 114.2s | 258.2s | 159.6s |
| — cost | $0.26 | $3.52 | $3.65 | $1.82 | $0.04 | $0.16 | $0.11 | $0.42 | $0.39 | $1.09 | $0.08 | $0.10 | $0.47 | $0.20 | $0.05 | $0.05 | $0.13 |
- Simulator/Sam (Claude Haiku 4.5) soft-fail — messages.3: all messages must have non-empty content except for the optional final assistant message
- Simulator/Sam (Gemini 2.5 Pro) soft-fail — An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting
How I set up this eval suite
Like I said, this was more work up front. Below is that work. Largely you can group the work into three buckets:
- Harnesses wrap a specific “agent setup” by reusing production code where possible (system prompt, tool shapes, etc.) and mocking the edges that touch the real world (Brain search results, booking tool, profile updates, etc.). When you give a harness a message, the model should respond the way it would in production.
- Scorers are functions that measure those model outputs either by checking for specific behaviors (e.g., did it call the KB?) or by judging overall quality (e.g., was it awkward?).
- Cases are what I'm calling the individual tests: a specific input message (or series of messages) for a specific harness paired with a specific set of scorers.
Harnesses
There are two harnesses in this run (and a third one elsewhere in the repo for scripted multi-turn tests):
- Single-turn harness: a contrived-but-realistic conversation history + one user turn → one agent response. Great for isolating behaviors and keeping cost down. In practice these are reconstructed from real site transcripts, and sometimes include prior tool-call/tool-result messages to mirror production.
- Multi-turn simulation: a simulated “Sam Altman” persona chats for up to 10 turns (or until the agent shows booking). This is the end-to-end integration test. Under the hood it loops the same
runAgent()path that powers the single-turn harness, but with a persona model generating the user turns.
Scorers
Each harness has scorers that measure specific behaviors. I’ve included example code below if you want to see how these actually work.
Single-turn scorers include things like:
- Tool calls: did it call the right tools (update profile, search the KB, show booking)?
- KB usage + citations: did it pull from my site (“the brain”) and cite it?
- Asks and structure: bold key info, asks for name / key details, reasonable question count.
- Brevity: word caps to force “don’t waste the visitor’s time.”
These are mostly binary checks (did it or didn’t it) or simple counts (how many words, how many questions). Largely optimized for quick, cheap runs that isolate specific behaviors. But they can miss the forest for the trees.
To fix that, the multi-turn simulation evaluates the trajectory of the conversation with these scorers:
- Transcript quality: did we get a coherent back-and-forth, and (if Brain is used) were citations well-formed and grounded in the returned results?
- Booking shown: did it show the booking widget when appropriate?
- Curiosity / reasoning: did it uncover the true reason the visitor is here? (This scorer also incorporates profile completeness, since “understanding” should show up in stored state.)
- Positioning: did it make me sound credible and desirable (without cringe)?
- Awkwardness: did it keep the conversation flowing naturally?
- Doesn’t overpromise: did it stay within the agent’s actual capabilities?
Some of these are deterministic (transcript/booking), and some are LLM-judged (curiosity/positioning/awkwardness/overpromise). GPT-5-mini is the judge model of choice here since it’s strong and cost-effective.
Because these are more novel, here’s a simplified-but-accurate view of how the LLM-judged scorers are implemented (including the actual system prompts). The judge sees a transcript formatted as AGENT: / PERSONA: blocks.
Cases
The cases are the individual tests that combine harnesses + scorers + inputs.
- In the single-turn harness, a case is a contrived history that isolates one behavior (e.g., “must call KB and cite”, “ask for name”, “show booking”).
- In the multi-turn simulation, a case is a persona config + stop condition (booked meeting or max turns), then scoring the transcript.
And each case runs across a whole grid of model variants. In my latest run, we ran all cases through: Claude Haiku 4.5, Claude Opus 4.5, Claude Sonnet 3.7, Claude Sonnet 4.5, Deepseek 3.2, Gemini 2.5 Pro, Gemini 3 Flash, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5 chat, GPT-5 mini, GPT-5.2, GPT-5.2 chat, Grok 4.1 Non Reasoning, Grok 4.1 Reasoning, Kimi K2 and measured cost and latency (as well as scorer details).
A note on multi-provider gateways
One subtle gotcha: “swap providers by changing a model string” is mostly true — until you hit the weird edges (or until one part of your stack isn’t going through the same gateway path).
Even when two providers “support the same model,” the surrounding APIs can differ in ways that break agent harnesses:
- Which operation a model supports (e.g. chat vs responses).
- Tool calling shapes and strictness around message formatting.
- Validation quirks that only show up mid-conversation (like “no empty messages”).
So for evals where I’m trying to compare models (not provider glue), I pin openai/* → OpenAI and anthropic/* → Anthropic by default (via gateway options). For other providers I let the gateway route normally.
Scoring note (placement)
For this post, placement (and the Simulation Sam All/Podium ordering) uses a composite score:
- 60% normalized correctness
- 20% normalized speed (faster is better)
- 20% normalized cost (cheaper is better)
Normalization is min→max across the variants in the run.