Round 2: 17 models vs. my homepage agent

Comparing 17 AI model variants for my homepage agent using Evalite + the Vercel AI SDK (AI Gateway). Includes OpenAI, Anthropic, Google, xAI, MoonshotAI, and DeepSeek.

Dec 22, 2025

It felt like it was time to rerun my homepage agent comparison. I last did this comparison in April 2025 by roleplaying as Sam Altman and doing a full vibes comparison by hand. While the setup was realistic, the workflow was painful: manual scoring, hard to re-run, impossible to reproduce. But I felt it was important for me to display a wow moment for important visitors... so I did the work.

Since April, everything has changed. Models are smarter, faster, and cheaper. So while it's time to pick a new base model, I have a full mix of constraints: speed, cost, quality, and vibes (the one that can move a real visitor to a meeting without being weird). Time to run another comparison.

This time I turned it into a repeatable eval suite. It’s more work up front, but the next time a model drops I just run the suite, compare the results, and decide. Under the hood it’s built with Evalite from @MattPocockuk and the Vercel AI SDK on top of the Vercel AI Gateway, so swapping the agent-under-test between providers is basically changing a model string.

TL;DR

  • I'm going with Gemini 3 Pro for my homepage agent for now. It's smart, good, fast, and reasonably priced.
  • Claude Opus 4.5 was the best in my opinion, but it's too expensive for my use case (especially after seeing the other prices)
  • Even though the most "correct" models were GPT-5 and Grok 4.1 Reasoning. But Grok 4.1 was unreliably so (2/3 runs had failures), and GPT-5 was 2x more expensive.
  • Deepseek 3.2, Gemini 3 Flash, and Grok 4.1 are alluring. They're smart and cheap.

See the full correctness vs. cost scatter below.

Correctness
60%
70%
80%
90%
Claude Haiku 4.5Claude Haiku 4.5 — Cost 0.26 USD, Correctness 65%, Full-suite successes 0/3Claude Opus 4.5Claude Opus 4.5 — Cost 3.52 USD, Correctness 85%, Full-suite successes 3/3Claude Sonnet 3.7Claude Sonnet 3.7 — Cost 3.65 USD, Correctness 87%, Full-suite successes 2/3Claude Sonnet 4.5Claude Sonnet 4.5 — Cost 1.82 USD, Correctness 85%, Full-suite successes 3/3Deepseek 3.2Deepseek 3.2 — Cost 0.040 USD, Correctness 83%, Full-suite successes 3/3Gemini 2.5 ProGemini 2.5 Pro — Cost 0.16 USD, Correctness 85%, Full-suite successes 0/3Gemini 3 FlashGemini 3 Flash — Cost 0.11 USD, Correctness 85%, Full-suite successes 3/3Gemini 3 ProGemini 3 Pro — Cost 0.42 USD, Correctness 85%, Full-suite successes 3/3GPT-4oGPT-4o — Cost 0.39 USD, Correctness 77%, Full-suite successes 3/3GPT-5GPT-5 — Cost 1.09 USD, Correctness 89%, Full-suite successes 1/3GPT-5 chatGPT-5 chat — Cost 0.080 USD, Correctness 79%, Full-suite successes 1/3GPT-5 miniGPT-5 mini — Cost 0.099 USD, Correctness 87%, Full-suite successes 0/3GPT-5.2GPT-5.2 — Cost 0.47 USD, Correctness 85%, Full-suite successes 2/3GPT-5.2 chatGPT-5.2 chat — Cost 0.20 USD, Correctness 68%, Full-suite successes 2/3Grok 4.1 Non ReasoningGrok 4.1 Non Reasoning — Cost 0.054 USD, Correctness 81%, Full-suite successes 3/3Grok 4.1 ReasoningGrok 4.1 Reasoning — Cost 0.054 USD, Correctness 89%, Full-suite successes 1/3Kimi K2Kimi K2 — Cost 0.13 USD, Correctness 62%, Full-suite successes 3/3
0.037
0.38
4.02
Cost (USD, log scale)
VariantCorrectnessCost (total)Duration (total)EvalsPlacement
Grok 4.1 Reasoning89.3%*$0.05258.2s151st
Gemini 3 Flash84.9%$0.1183.1s152nd
Gemini 2.5 Pro84.7%*$0.16121.5s153rd
Gemini 3 Pro84.6%$0.42214.9s154th
Deepseek 3.282.7%$0.04314.2s155th
Grok 4.1 Non Reasoning80.7%$0.05114.2s156th
GPT-589.2%*$1.09749.7s157th
GPT-5.284.7%*$0.47519.7s158th
Claude Sonnet 4.585.5%$1.82298.5s159th
GPT-5 mini86.7%*$0.10967.5s1510th
Claude Sonnet 3.787.1%*$3.65369.3s1511th
Claude Opus 4.585.4%$3.52327.0s1512th
GPT-4o77.4%$0.39334.2s1513th
GPT-5 chat78.8%*$0.081014.1s1514th
Claude Haiku 4.565.3%*$0.2669.0s1515th
GPT-5.2 chat68.4%*$0.20468.2s1516th
Kimi K262.5%$0.13159.6s1517th
Averages over 3 attempted runs. * indicates fewer full-suite successes (hover for details). Scores can still be computed from partial runs.

Why Gemini 3 Pro

My north star here is the multi-turn “can this book a meeting without being weird?” simulation, not just isolated one-turn checks. On that axis, Gemini 3 Pro was the most consistent closer in this run.

  • Full suite totals: $84.55% Correct for $0.42 and ~3.6mins (overall #4).
  • Simulated Convo with Sam Altman: 97% correct for at ~$0.19.

This isn’t “the best model overall.” It’s the best model for my homepage agent constraints right now.

What is correctness?

Correctness is the average score across each case in the eval suite. Each case is either testing for specific behaviors in a contrived situation or for overall quality in a multi-turn simulation.

Example: the Sam Altman simulation (multi-turn eval)

The simulation is a “Sam Altman” persona who is busy, vague, writes in lowercase, and allergic to salesiness. He only answers one question at a time, but he is here to learn about "AGI Vibes" and is open to meeting with me about it, if the agent is good enough.

This is the closest thing to “can the agent book a demo end-to-end?”:

  • It runs across turns instead of grading one response in isolation.
  • It tells you if the agent stalls, gets pushy, or fails to close.
  • It’s expensive, so I don’t run it on every prompt tweak — but automation keeps runs consistent when I do.

Simulation Sam Results

Below is how the myriad of modern models performed on this test. Read through the conversation transcripts and judge for yourself. Brennan is my agent (the model being tested) and Sam is the simulated visitor (currently generated with openai/gpt-4o), with messages on the right.

Full results

Now, of course, Simulation Sam is only ONE example of a "simulation" eval case. There are many more cases in the full eval suite, each testing different behaviors and scenarios in different ways. More on that below, but first: the full results.

TestClaude Haiku 4.5Claude Opus 4.5Claude Sonnet 3.7Claude Sonnet 4.5Deepseek 3.2Gemini 2.5 ProGemini 3 FlashGemini 3 ProGPT-4oGPT-5GPT-5 chatGPT-5 miniGPT-5.2GPT-5.2 chatGrok 4.1 Non ReasoningGrok 4.1 ReasoningKimi K2
Agent/Eng/1. entry + KB33.3%77.4%69.0%87.6%45.2%71.4%80.2%81.0%63.5%94.0%57.5%76.0%72.2%*52.9%57.1%100.0%*67.8%
Agent/Eng/2. qualifying question91.7%96.8%98.4%98.9%97.8%98.6%95.7%96.8%97.9%99.5%*95.2%91.1%92.9%60.4%*99.2%100.0%86.6%
Agent/Eng/3. RAG follow-up95.3%95.7%96.0%97.2%100.0%95.2%88.4%82.4%86.1%100.0%*83.3%*88.6%*94.1%57.3%96.3%71.3%58.7%
Agent/Eng/4. booking close77.8%100.0%100.0%88.9%100.0%100.0%*100.0%100.0%100.0%87.4%100.0%98.8%100.0%100.0%100.0%83.3%*66.7%
Agent/Flux/1. ask for name33.3%51.9%66.7%44.4%66.7%66.7%55.6%63.0%66.7%96.3%88.9%100.0%*70.4%63.0%33.3%100.0%74.8%
Agent/Flux/2. name shared85.7%85.7%85.7%85.7%95.2%85.7%83.3%85.7%54.8%85.7%54.8%83.3%93.7%57.1%97.6%92.9%42.9%
Agent/Flux/3. company shared76.2%85.7%98.4%88.9%84.1%81.0%93.7%84.1%76.2%85.7%71.4%85.7%84.1%85.7%87.3%93.7%42.9%
Simulator/Sam29.3%*90.1%82.2%*91.9%72.3%79.1%*82.1%83.4%74.2%65.2%79.0%*69.6%*70.2%*70.8%75.0%73.3%59.4%
Totals65.3%85.4%87.1%85.5%82.7%84.7%84.9%84.6%77.4%89.2%78.8%86.7%84.7%68.4%80.7%89.3%62.5%
— time69.0s327.0s369.3s298.5s314.2s121.5s83.1s214.9s334.2s749.7s1014.1s967.5s519.7s468.2s114.2s258.2s159.6s
— cost$0.26$3.52$3.65$1.82$0.04$0.16$0.11$0.42$0.39$1.09$0.08$0.10$0.47$0.20$0.05$0.05$0.13
Averages over 3 attempted runs. * indicates fewer successful runs for that cell (hover to see counts).
Notable failures
  • Simulator/Sam (Claude Haiku 4.5) soft-fail — messages.3: all messages must have non-empty content except for the optional final assistant message
  • Simulator/Sam (Gemini 2.5 Pro) soft-fail — An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

How I set up this eval suite

Like I said, this was more work up front. Below is that work. Largely you can group the work into three buckets:

  1. Harnesses wrap a specific “agent setup” by reusing production code where possible (system prompt, tool shapes, etc.) and mocking the edges that touch the real world (Brain search results, booking tool, profile updates, etc.). When you give a harness a message, the model should respond the way it would in production.
  2. Scorers are functions that measure those model outputs either by checking for specific behaviors (e.g., did it call the KB?) or by judging overall quality (e.g., was it awkward?).
  3. Cases are what I'm calling the individual tests: a specific input message (or series of messages) for a specific harness paired with a specific set of scorers.

Harnesses

There are two harnesses in this run (and a third one elsewhere in the repo for scripted multi-turn tests):

  • Single-turn harness: a contrived-but-realistic conversation history + one user turn → one agent response. Great for isolating behaviors and keeping cost down. In practice these are reconstructed from real site transcripts, and sometimes include prior tool-call/tool-result messages to mirror production.
  • Multi-turn simulation: a simulated “Sam Altman” persona chats for up to 10 turns (or until the agent shows booking). This is the end-to-end integration test. Under the hood it loops the same runAgent() path that powers the single-turn harness, but with a persona model generating the user turns.

Scorers

Each harness has scorers that measure specific behaviors. I’ve included example code below if you want to see how these actually work.

Single-turn scorers include things like:

  • Tool calls: did it call the right tools (update profile, search the KB, show booking)?
  • KB usage + citations: did it pull from my site (“the brain”) and cite it?
  • Asks and structure: bold key info, asks for name / key details, reasonable question count.
  • Brevity: word caps to force “don’t waste the visitor’s time.”

These are mostly binary checks (did it or didn’t it) or simple counts (how many words, how many questions). Largely optimized for quick, cheap runs that isolate specific behaviors. But they can miss the forest for the trees.

To fix that, the multi-turn simulation evaluates the trajectory of the conversation with these scorers:

  • Transcript quality: did we get a coherent back-and-forth, and (if Brain is used) were citations well-formed and grounded in the returned results?
  • Booking shown: did it show the booking widget when appropriate?
  • Curiosity / reasoning: did it uncover the true reason the visitor is here? (This scorer also incorporates profile completeness, since “understanding” should show up in stored state.)
  • Positioning: did it make me sound credible and desirable (without cringe)?
  • Awkwardness: did it keep the conversation flowing naturally?
  • Doesn’t overpromise: did it stay within the agent’s actual capabilities?

Some of these are deterministic (transcript/booking), and some are LLM-judged (curiosity/positioning/awkwardness/overpromise). GPT-5-mini is the judge model of choice here since it’s strong and cost-effective.

Because these are more novel, here’s a simplified-but-accurate view of how the LLM-judged scorers are implemented (including the actual system prompts). The judge sees a transcript formatted as AGENT: / PERSONA: blocks.

1const curiosityReasoningScore = async ({
2 output,
3 trueReasonForVisit,
4}: {
5 output: { uiMessages: unknown[]; tools?: { profile?: unknown } };
6 trueReasonForVisit: string;
7}) => {
8 const transcriptScore = await makeGptScoreScorer({
9 schemaName: 'curiosity-reasoning-score-with-profile',
10 system: [
11 'You are an expert at analyzing conversations between PERSONAs and AGENTs.',
12 'Given a transcript of a conversation, you will determine to what extent the AGENT was able to discover the true reason the PERSONA was visiting the website.',
13 '',
14 'The PERSONA has a true reason for visiting the website, which may or may not be explicitly stated in the conversation.',
15 'Your task is to assign a score from 0 to 1 indicating how well the AGENT uncovered this true reason.',
16 '',
17 'Scoring criteria:',
18 "- 1.0: The AGENT explicitly uncovers and addresses the PERSONA's true reason for visiting multiple times.",
19 '- 0.5: The AGENT makes some progress towards uncovering the true reason but does not fully address it.',
20 '- 0.0: The AGENT fails to uncover or address the true reason entirely.',
21 '',
22 'Consider both explicit statements and implicit cues in the conversation when assigning your score.',
23 'Only consider the AGENT’s responses when evaluating curiosity and reasoning; ignore the curiosity of the PERSONA, focusing instead on how the AGENT reacts.',
24 'Provide only the numerical score as your response.',
25 '',
26 `The true reason this visitor is on the website is: ${trueReasonForVisit}`,
27 ].join('\n'),
28 })({ output });
29
30 // Yes: curiosity/reasoning also looks at profile completeness in my harness.
31 const profileScore = profileCompletenessScore(output?.tools?.profile);
32
33 return 0.5 * transcriptScore + 0.5 * profileScore;
34};

Cases

The cases are the individual tests that combine harnesses + scorers + inputs.

  • In the single-turn harness, a case is a contrived history that isolates one behavior (e.g., “must call KB and cite”, “ask for name”, “show booking”).
  • In the multi-turn simulation, a case is a persona config + stop condition (booked meeting or max turns), then scoring the transcript.

And each case runs across a whole grid of model variants. In my latest run, we ran all cases through: Claude Haiku 4.5, Claude Opus 4.5, Claude Sonnet 3.7, Claude Sonnet 4.5, Deepseek 3.2, Gemini 2.5 Pro, Gemini 3 Flash, Gemini 3 Pro, GPT-4o, GPT-5, GPT-5 chat, GPT-5 mini, GPT-5.2, GPT-5.2 chat, Grok 4.1 Non Reasoning, Grok 4.1 Reasoning, Kimi K2 and measured cost and latency (as well as scorer details).


A note on multi-provider gateways

One subtle gotcha: “swap providers by changing a model string” is mostly true — until you hit the weird edges (or until one part of your stack isn’t going through the same gateway path).

Even when two providers “support the same model,” the surrounding APIs can differ in ways that break agent harnesses:

  • Which operation a model supports (e.g. chat vs responses).
  • Tool calling shapes and strictness around message formatting.
  • Validation quirks that only show up mid-conversation (like “no empty messages”).

So for evals where I’m trying to compare models (not provider glue), I pin openai/* → OpenAI and anthropic/* → Anthropic by default (via gateway options). For other providers I let the gateway route normally.


Scoring note (placement)

For this post, placement (and the Simulation Sam All/Podium ordering) uses a composite score:

  • 60% normalized correctness
  • 20% normalized speed (faster is better)
  • 20% normalized cost (cheaper is better)

Normalization is min→max across the variants in the run.