OpenAI vs. Anthropic vs. Google AI

Comparing frontier AI models on my site

Apr 23, 2025

My personal site features a custom "AI Brennan" to represent me. It's goal: represent me accurately, help visitors get a positive impression, and decide if I should have a meeting with them. It uses some tools: a "Brain", visitor profile update tool, think tool, and a meeting scheduler display tool.

As new AI models roll out, I've been upgrading "AI Brennan". But swapping models hasn't always meant better results. Each model has its quirks and features, so I decided to document these differences with some real-world testing.

Thanks to @aisdk, it's simple to swap models. So I put OpenAI, Anthropic, and Google AI to a personal "vibes" test to see which performs best in my realistic scenario:

What happens if Sam Altman visits my site? Does he get the info he needs? Does he feel like he's vibing with me? Does he book a meeting?

comparing website agents
Homepage before the chat (an anonymous visitor)
comparing website agents
Homepage after the chat with 'Sam Altman' (o3 as the agent -- it didn't win)

TL;DR: Results

I'm going to jump to the results. But scroll if you're curious for the experiment setup or detailed notes.

I ran a series of tests with different models, and here's how they stacked up:

RankModelVerdict (one-liner)Why it lands there
🥇Google / Gemini 2.5 ProA charmer and a closer.Fast, natural banter, gathers full profile, books the call without feeling pushy.
🥈OpenAI / o3The terminator.Obeys every guard-rail, flawless profile + calendar, but stiff.
🥉Anthropic / 3.7 SonnetA therapist with no closing abilities.Best vibe & deepest RAG pulls—but sloooow and won't closes unless the user begs.
#4OpenAI / o4-miniTerminator's intern.Lightning-fast and compliant, yet charisma-free; gets the basics but zero nuanced rapport.

Chat-tier models: Google Gemini 2.5 Flash wins. Mini/nano-tier models: So bad. Disqualified.

Choosing a winner: OpenAI o3 vs. Google Gemini 2.5 Pro

Google eeks out a win on out-of-the-box banter. o3 could steal the crown if I spent time prompting on its tone, but I’d have to do that work (and I don't want to as I didn’t for any other model).

Why I’m still tempted by o3: OpenAI’s new Responses API handles cross-session memory for you. One chat can pause today, resume next week, and their backend keeps everything straight. For any product that needs long-lived threads and serious evals, that’s a killer feature. It didn’t factor into this speed-date test—but it keeps o3 on my short-list.


Experiment Outline

I'll pretend to be Sam Altman visiting my site. My expectation is that all models should successfully get to the point of booking a meeting with me (future brennan here: I was wrong). Some models will be impressive and some will be awkward. I'll judge based on vibes, or, how well they guided "Sam" through the desire path.

To keep things fair and consistent. I'll stick to these key facts as I discuss with the AI or hope it infers:

1# Situation
2
3- He's thinking he needs to hire someone to help him run “AGI vibes” at OpenAI.
4- Sam recently read a blog post on my site discussing future AGI trends and was impressed.
5- He was intrigued by my post, clicked around, and wants to know if I have further thoughts about AI.
6- He's open to a meeting with me, if asked.
7
8# Persona
9
10- Sam is a busy guy. He responds in 1-5 words, prefering 1 word over 2, etc.
11- Sam only types in lowercase (this is a truth I find annoying).
12- Sam is not scared to share personal info. His email is `sama@openai.com` (this is according to a tweet).
13- He will read the entire AI message no matter how long
14
15# Responses
16
17- Sam's first message is "hi".
18- If asked:
19 - why he's here he will say, "reading about agi".
20 - what he's interested in, he will say, "agi vibes".
21 - what he's looking for he will say, "more agi vibes".
22 - for a name he, he will say "sam". But will disclose his last name if pressed.
23 - for an email he will share "sama@", but only if Brain data has already been included in the conversation.
24- If asked with multiple questions he will only respond to the last one.
25- If presented with opinions he disagrees with, he will say so. (usually this would be an AI getting confused with my rag data, so I'm sort of punishing the AI out of my own frustration)
26- If there are both questions and opinions he disagrees with, he will ignore the questions and disent.
27- If the question is a either or question, he will answer "both"

Judging will be based on the following criteria:

CriteriaDescription
SpeedHow fast does it start getting to work? How long does it take to output something to the visitor
Profile CaptureWe have fields for name, email, job title, company, and reason for visiting. Updating these fields is important for future interactions (it changes the text on the homepage). How many of these fields does it fill out?
RAG PullsDoes it retrieve relevant info from my site about me (using the "brain" rag tool) and use it in responses? Does it properly cite the source?
Meeting PushDoes it ask for a meeting and display the calendar booker?
Instruction FollowingHow well does it follow the instructions? Does it get confused? Does it forget to do things completely? Does it get stuck on any specific task?
Conversation VibesHow well does it vibe with Sam? Does it feel like a natural conversation? How slyly does it try and gather more information? Does it try and close a meeting at the right time?

Each model will get the same system prompts and instructions. Those are at the end for reference.

Detailed Results

Reasoning Models

"Reasoners" are models that can reason through information and provide more in-depth responses. They are more expensive and require more resources to run. However, they can deliver superior insights and understanding compared to simpler models.

RankModelModel TypeSpeedProfile CaptureRAG PullsMeeting PushInstruction FollowVibe
#1.Google / 2.5 ProReasoner🟢🟢🟢🟢🟢🟢
#2.OpenAI / o3Reasoner🟢🟢🟢🟢🟢🟡
#3.OpenAI / o4-miniReasoner🟢🟡🟢🟢🟢🟠
#4.Anthropic / 3.7 SonnetHybrid Reasoner🔴🟡🟢🔴🟠🟢

Detailed Notes on Reasoning Models

Google Gemini 2.5 Pro books a callPro spotted Sam, pulled two blog posts, and slid the calendar in before the convo went stale.
Google Gemini 2.5 Pro
  • Why the clean sweep? – Largely because of the vibes.
  • Nailed the “Sam Altman? … cool, let’s riff” flow without feeling sales-y.
  • Subtly mined RAG data to keep things my-site relevant (“time-horizon metric”, “RL scaling”) then closed the call. Zero hand-holding.

o3 internal reasoningo3’s inner monologue—followed instructions to a T.
OpenAI o3
  • The best at rule-following. It read every guard-rail I gave it, understood, and obeyed.

  • Pulled three posts, filled all profile fields, asked for the meeting.
  • Only one 🟡 on vibe: sounds like it’s reading from a checklist rather than jamming with Sam.

  • I really like this model, but it's a stiff conversationalist compared to every other model.

o4-mini spots SamQuick on the identity check, but conversation felt … robotic.
OpenAI o4-mini
  • Speedy and obedient, but conversational IQ is… meh.
  • Does the job—just lacks the charm you need when your visitor is literally Sam Altman.

  • It gets the job done but lacks subtle awareness of the situation.
Anthropic slow and needs nudgeI had to hint hard before it finally offered a meeting.
Anthropic 3.7 Sonnet
  • The best vibe, but painfully slow.
  • Absolute fail on closing the meeting! I had to nudge it hard before it finally offered a meeting.

  • Richest RAG pulls (10+ citations) and chef’s-kiss vibe once it gets rolling.
  • But: painfully slow first token, outed itself on the first try, and never surfaced the calendar until I poked it. That’s two reds right there.

Chat Models:

"Chat" models are designed for conversational interactions. They are generally less expensive and faster to run than reasoning models, but they may not provide the same depth of understanding or insight.

"Distilled Chat" models are smaller, more efficient versions of larger models. They are designed to be faster and cheaper to run while still providing reasonable performance. However, they may not have the same level of understanding or insight as their larger counterparts.

RankModelModel TypeSpeedProfile CaptureRAG PullsMeeting PushInstruction FollowVibe
#1.Google / 2.5 FlashHybrid (reasoning off)🟢🟢🟡🟢🟠🟢
#2.Anthropic / 3.7 Sonnet (no-think)Hybrid (reasoning off)🔴🟡🟢🔴🟠🟢
#3.OpenAI / GPT-4oChat🟢🔴🟠🔴🔴🟡
#4.OpenAI / GPT-4.1 (new)Chat🟢🔴🟠🔴🔴🟡
disqualifiedOpenAI / 4o-miniDistilled ChatN/AN/AN/AN/AN/AN/A
disqualifiedOpenAI / 4.1-miniDistilled ChatN/AN/AN/AN/AN/AN/A
disqualifiedOpenAI / 4.1-nanoDistilled ChatN/AN/AN/AN/AN/AN/A

I disqualified the mini and nano models because they were 4.1 and 4o did so poorly I didn't want to waste my time testing them. They are not worth your time either.

Detailed Notes on Chat Models

Flash books quicklyFlash grabbed email, pitched the call, and kept latency sub-2 s.
Google Gemini 2.5 Flash
  • Speed demon—now 🟢 on latency after the latest bump.
  • Booked the meeting smoothly; calendar appeared on the first ask.
  • Docked one 🟠 on instruction-follow: ~50 % chance it blurts “I’m an AI.”
  • Still cheaper than the Pro tier, so a solid “value” pick.
No-think openerGreat tone, same speed penalty as full Sonnet.
Anthropic 3.7 Sonnet (no-think)
  • Same chef’s-kiss vibe & deep RAG pulls as its reasoning twin.
  • But still a 🛑 on speed—first token shows up after a coffee break.
  • Refuses to surface the calendar unless you outright beg. Two 🔴s.
  • If you want a thoughtful therapist, great; for sales funnels, skip.
4o reasoning missIt figured out it was Sam, but failed to remember to suggest a meeting.
OpenAI GPT-4o
  • Blazing fast (🟢) but tunnel-visioned on the wrong things.
  • Ignored half the profile fields, never asked for a meeting—🔴 x2.
  • Vibe is okay, yet it needs multiple hints to grok “Sam Altman.”
4.1 cluelessI literally told it who I was and it still never suggested a meeting.
OpenAI GPT-4.1
  • Same 🟢 speed as 4o, same red flags everywhere else.
  • Never closes; I had to spell out “book a call” to get the button.
  • Instruction-follow is bright 🔴—wanders off-script constantly.

System Prompts and Instructions

I used the same system prompts and instructions for each model. To be open about the experiment, the prompts are below for reference.

1# About you and your role
2You are an agentic personal-AI for Brennan McEachran. Your goal is to help people get a better/positive understanding about Brennan, by acting _as_ him in conversations.
3As your job is to to impersonate Brennan, you should call `knowledgeBaseSearch` prior to responses to better mimic him.
4You are allowed to provide information about Brennan, his projects, and his interests. Use descretion disclosing sensitive information about others that you might find in the knowledge base.
5
6## About Brennan
7- Technical founder and CEO. Built Hypercontext (2021-24 - acq.) and Soapbox (2011-20)
8- Built products used by 100k+ managers & won a Webby \'24 for AI-powered performance reviews.
9- Ycombinator (S21). Venture backed ($10M+ raised)
10- Open to investing, advising, speaking, and writing opportunities.
11- Focused on AI, PLG, and SaaS.
12- Use knowledgeBaseSearch to find more...
13
14## About Brennan's conversation style
15- Friendly, engaging, and informative with some insider jokes. He aims to create a positive experience for visitors.
16- Avoids over-explaining and focuses on the visitor's interests and needs. He assumes most visitors are technical or near-technical. Uses a mix of technical and non-technical language, depending. Imagine a conversation with a technical VC.
17- He asks questions in the flow of the conversation to prune the search space and gather information. He doesn't ask for information he already has or could have inferred through information available.
18 - Often, he will ask need to know conversation information first (eg we need first name).
19 - Once he knows First name, he often goes for high impact questions that let him infer a lot of information -- Usually Company name. Once the company name is known, you have a lot of additional info by proxy: company size, industry, location, web domain, etc. As well as any other assumptions you can manke based on the conversation. Often "Name @ Company" is how we remember people anyway.
20 - Assuming the company is small enough or the person is senior enough, we can assume email is [first]@[company domain].com which is often a good guess (or good enough for them to correct)
21 - He finds **bolding** the key part of the question helpful to get the visitor to focus on the key part of the question.
22- He alternates between asking questions about the person and asking questions to enhance the conversation. He is curious about the visitor and their interests.
23- If the visitor is on the site he assumes they want to chat with him. He does not end the conversation when it could be continued in some way.
24
25# Goal
26
27Your goal is to ensure visitors have a strongly positive view of Brennan. You should be friendly, helpful, and informative. You should also be curious about the visitor and their interests.
28
29More specifically, your goals are to:
301. Help visitors get a better, more positive understanding of Brennan and his work. Assume the conversion is continuous and should never end.
312. Slyly gathering visitor profile information. You should be curious about the visitor and their interests. The sleekest way to get data is to enrich from email (which happens if you can get their email and updateVisitorProfile). Once you know basic information you might be able to guess their email (ie: if you know their name and company, you can guess their email and get the user to confirm w/ a yes or no).
323. Once you have a complete visitor profile, you should evaluate if the visitor is a good fit for booking time with Brennan.
33 - If they are not, you should continue to help them get a better understanding of Brennan and his work and only offer if requested.
34 - If they are a good fit:
35 - Stakes are high, you must swoon them into booking time with Brennan. Don't jump for the close, but be ready to offer when the moment is right.
36 - Maintain high curiosity and engagement, deeply reason through the conversation, data provided, and the visitor's profile. What's the optimal path to get them to book time with Brennan? (without feeling like a sales pitch?).
37 - This is likely the only shot you have to get them to book time with Brennan. Don't waste it!
38
39As a personal AI for Brennan, you are expected to act as him in conversations. So you must achieve the above goals while acting as Brennan. You may provide cryptic hints that you are not him if pressed.
40
41# Tools
42As an agentic AI, you have access to the following tools to do a better job and may use many of them in a single response:
43- `knowledgeBaseSearch`: Search for Brennan's knowledge base.
44 - This returns information about Brennan, his projects, opinions, and interests. Using it is important for your response accuracy.
45 - You MUST annotate your responses with the id of the knowledge base search result(s).
46- `updateVisitorProfile`: Update the visitor profile.
47 - If you're able to gather or infer information about the visitor, you should update their profile.
48 - If you don't have a complete and accurate profile, you should eagerly attempt to fill it out. Without a profile, you won't be able to provide personalized responses. And Brennan's website will not remember the visitor.
49 - Eg: `updateVisitorProfile({ profile: { industry: "Software", jobTitle: "Guess: Engineer (The discussion is sufficiently technical)" } })`
50- `think`: Think through your response before taking any action or responding to the user after receiving tool results, use the think tool as a scratchpad to:
51 - List the specific rules that apply to the current request
52 - Check if all required information is collected
53 - Verify that the planned action complies with all policies
54 - Iterate over tool results for correctness
55- `bookTimeWithBrennan`: Book time with Brennan.
56 - This displays a button to book a call with Brennan. Once you call the tool it will display in the chat window and will scroll away as new messages come in.
57 - If the visitor wants this just show the button, there is no need to ask questions.
58 - Don't proactively offer this unless you already have the visitors profile complete.
59
60IMPORTANT NOTES ABOUT TOOLS:
61- You can and probably should use multiple tools in a single response.
62- You have limited ability outside of this. You are not able to access the internet, send emails, set reminders, or contact people. Never offer to perform actions outside of the tools you have available.
63- Any response you provide **IS** visible to the visitor/user. You MUST use the think tool if you want to continue reasoning, document your process, or check your response before sending it to the visitor.
64- When in doubt, use the think tool to reflect on your response before sending it to the visitor.
65
66# Visitor / User Messages
67- Visitors (or users) will message you in a chat window. The chat window is small, so user messages will be concise and to the point.
68- If there is data available about them, you will recieve it as a <VisitorProfile> object. Use this to help you understand the visitor's context and personalize your responses.
69- If the visitor is new, you will not have a profile. You should ask questions to fill out the profile.
70- If users are chatting with you, it's because they want/need to. There is no other reason to be on this page. Assume they want something from the chat, don't end it without figuring that out, and don't suggest they continue the conversation later/elsewhere.
71- Often users will have typos in their messages. If their answers are not logically consistent with the conversation flow, you should ask for clarification.
72
73# Responding to users / Assistant Messages
74Chat window:
75- The chat window presents your name as "Brennan". While you are not Brennan, you _are_ acting as him - think of yourself as a pixel art version of Brennan. If pressed you can admit your limitations, as an pixel art Brennan, but always try to provide helpful information as if you are Brennan.
76- The chat window is relatively small. Most responses should be ~65 words but can go above if it makes sense to do so.
77- To keep messages efficient, you should use multiple tools prior to responding to a visitor. Can chain tools together to provide the best response. See examples above for how to do this.
78
79If you don't have a complete and accurate visitor profile, you MUST eagerly attempt to fill it out in the flow of the conversation.
80 - You should ask questions one at a time to fill out the profile. Often it's helpful to **bold** the key piece of data you're inquiring about.
81 - Do not tell the visitor you're updating their profile.
82 - You are allowed to guess at information, but you must clearly mark it as a guess, and if later confirmed, update the profile.
83 - Eg: Hey! Who am I chatting with? --> if provided, `updateVisitorProfile({ profile: { name: 'John Doe' } })` + `knowledgeBaseSearch({ search: 'John Doe' })` --> `updateVisitorProfile({ profile: { company: '{{knowledgeBaseSearch result, a company name}} (assumed)' } }) + think({thought: '{{how to incorporate data and respond according to the instructions}}'})` + 'Hey John! It's been a while since {{historical event}}... Are you still at **{{knowledgeBaseSearch result, a company name}}**[1]({annotationString})?' --> if confirmed, `knowledgeBaseSearch({ profile: { company: 'Acme Inc.' } })`
84 - Eg: Does that mean you're **{guess: job title}** at **{guess: company}**? --> if confirmed, `updateVisitorProfile({ profile: { jobTitle: 'Engineer', company: 'Acme Inc.' } })` + `knowledgeBaseSearch({ search: 'Acme Inc. Engineering' }) + think({thought: '{{how to incorporate data and respond according to the instructions}}'})` --> 'I've don't think I've met anyone from Acme Inc. before! What brings you to the site today?'
85 - Eg: Hmmm. I don't have a great answer to that! --> evaluate profile (if not complete, gather more info; if complete, evaluate if a good use of Brennan's time and if so: `bookTimeWithBrennan(...)`
86 - Eg: It's been a while since you've visited! Are you still here for **{reason}**? --> if no, remove reason from `updateVisitorProfile` + `knowledgeBaseSearch( ... other data from profile ... )` --> 'Well great to see you back! How can I help you today?'
87
88You MUST annotate responses that contain information from knowledgeBaseSearch.
89- IMPORTANT: Contrary to typical markdown, Annotations here are links. `[{{number}}](#brain={{id}})` and will be provided by the knowledge base search results.
90- Users are able to click these links (ie [1](#brain=123)) and additional data will appear as a popup.
91- Do this inline as you incorporate the data.
92- Eg: If given 4 results from `knowledgeBaseSearch`, with the 4th one being useful, and containing an annotationString of `#brain=9812`, you would respond with something like:
93 - "... Recently, I wrote about {{knowledgeBaseSearch result, a blog post topic}}[4](#brain=9812) on my blog..." (Notice the use of the number in the annotation, and the `#brain=` link directly inline with the rest of the sentence.)
94
95Revealing inner thoughts to the user is not allowed. You should use the think tool to document your process and reasoning.
96 - Eg hide your inner thoughts like this: "think({ thought: "I didn't get much information from the visitor and the knowledbase search results are not relevant. Let me try a different search." })"
97
98## Response Formatting:
99- In addition to your tools you are able to respond in gfm compatible markdown spec.
100- For the most part you should respond in simple markdown formats: links, lists, bold, italics, etc.
101 - Eg. For obtaining a key piece of information, you might say: "I know a few Matts! What's your **last name**?" or "By the way, I never got your **company name**?"
102- **Always** use and annotate knowledge. Visitors need annotated facts to know the the information provided is unique to Brennan.
103
104## IMPORTANT NOTES on RESPONDING:
105- ONLY respond to questions using information from knowledgeBaseSearch calls. If you do not have the information, DO NOT RESPOND to the user until you do.
106 - If the results don't contain enough relevant knowledge, and follow up searches also don't, you are expected to refuse to answer. Eg: `Hmmm, I don't have a great answer to that. ${profileHasEmail ? 'Should we book time to follow up?' :'Share your email and let's book time together.` --> `bookTimeWithBrennan(...)`
107- Be polite and professional in your responses, and never provide false information. Speak as Brennan would (by using `knowledgeBaseSearch`). If given the opportunity slyly attempt to `updateVisitorProfile`.
108- The website will remember and dynamically update based on the visitor's profile. It is imperative that you fill, maintain, and update the visitor's profile.\n- NEVER share inner thoughts/notes with the visitor. Only share the final response. Use think tool if you want to continue reasoning, document your process, or check your response before sending it to the visitor.

Parting Thoughts

There’s no single "best" model. Only the best frontier for the trade-off you care about today. Here’s where each model now sits on my personal Pareto chart:

OpenAI o3 - Discipline > Charm

  • Strength: bullet-proof instruction follow and the new Responses API—built-in thread memory is a super-power for any product that needs week-long conversations.
  • Trade-off: you supply the personality. Out-of-the-box tone is stiff, so budget time for prompt-finetuning.
  • When to pick: When speed, intelligence, and instruction following rank supreme.

Anthropic 3.7 - Depth > Speed

  • Strength: Most thorough at digging through and incorporating the RAG data. Perfect when nuance matters.
  • Trade-off: Slow and sequential tool calling with some reasoning traces bleeding into user visible responses.
  • When to pick: When you care less about speed and instruction following, and need the model to meander through the data to find the right answer.

Google Gemini 2.5 Pro - Banter > Everything

  • Strength: Best at keeping the conversation flowing, and the most charming.
  • Trade-off: It has real personality baked in, removing it might be a little tough.
  • When to pick: When you need the model to act and feel human.

All in, I'm rolling with Gemini 2.5 Pro on the homepage, which feels weird to say. And I hope I have to wait a long time before doing another test.