SSMS Copilot is Messing With Your AI Prompts.
You might notice that the advice you get from Copilot in SSMS isn’t very good, but it isn’t necessarily AI in general. Part of the problem is something specific about SSMS Copilot and the way it prompts LLMs, and I wish I could tell you exactly what it is, but I can’t because I don’t have access to how SSMS is changing your prompts.
Let’s take a task I need to do all the time, and I constantly forget the syntax to do it. How do I turn on last actual plans?
That answer is wrong. Period. Flat out incorrect. I’m not even ON 2022, I’m on 2025.
At first I thought, “Maybe Claude Sonnet 4.6 has the wrong answer – I’ll try asking Claude directly in Claude’s chat.” And Claude gets it right:
That’s correct, and that’s bad, because it means that something about SSMS Copilot’s prompting is biasing it towards incorrect answers. Perhaps it’s including too much background information and losing relevant context, or pushing the LLM towards a certain kind of answer (like “encourage the user to adopt new features”).
In my experience, Gemini 3.1 Pro has been less vulnerable to Copilot’s prompt corruption. In this example, it gets the answer correct:
But that’s misleading, because the answer isn’t to use Gemini 3.1 Pro because it’s somehow “better”. It’s not – Claude Sonnet 4.6 had the right answer when I asked the same question directly, but something about Copilot’s prompting mislead Claude. The real answer is for Microsoft to fix the SSMS Copilot prompting so that all models – including cheaper/faster ones – get better, more accurate answers. I just don’t know how to do that since we don’t get visibility into the prompt that Microsoft is sending along with our question.
This kind of thing happens to me all the time, and I try reporting it to Microsoft, but SSMS Copilot’s thumbs-down icon has a weird set of choices:
Claude’s response was just flat out wrong – not incorrect for this version, incorrect for ANY version, and it’s bad because Claude itself gets it right – only Copilot is getting it wrong.
Between SSMS messing with my prompts, and Copilot switching to usage-based pricing, I just don’t see a reason to use SSMS Copilot directly right now. You’re so much better off using T-SQL to build a robust prompt, like we do with the @AI parameter in sp_BlitzCache and sp_BlitzIndex, and then sending that prompt to the LLM of your choosing. You get a better prompt, access to better models for processing, and it doesn’t cost you any more.
I’m not giving up on Copilot yet. I keep trying it first, every time there’s a new version of SSMS, because I’m eternally hopeful that it’s going to get better.
Free, 3× a week
Get my new posts by email
Three posts a week, plus a Monday roundup of the best database news from around the web.
[…] ???? […]
I love this, Brent. You’ve just helped me put into terms exactly what I’ve been seeing! ‘I just don’t know how to do that since we don’t get visibility into the prompt ‘ is the whole problem in one sentence. We can’t tune what we can’t see, and they’ve built it as the one query that doesn’t get a plan. Unless or until they expose the system prompt, or at least let us see what’s being prepended, are we stuck? Limited options anyway. Meanwhile the @AI parm approach works because the prompt is right there in the t-SQL where I can read it, change it, and blame myself when it goes sideways.
I tried “How do I turn on last actual plans in sql server?” in Claude desktop and via claude.ai. Both using Sonnet 4.6 and I received 2 different answers. The claude.ai answer was similar to your SSMS answer but not quite the same. The Claude desktop returned the correct answer same as you. Very strange behavior.
I’m also getting the query store answer in claude.ai web interface and SSMS, both using sonnet 4.6. Using sonnet 4.5 in SSMS gives the correct answer, so it doesn’t seem to be an SSMS specific issue.
Copilot in SSMS is so bad compared to what we have on VSCode, it doesn’t even shows all the models you enable in gihub copilot config, also the only free model listed (at least for me) is the old GPT-4.1
As an IT generalist with ownership of a SQL Server-backed ERP and BI infrastructure, I have a different perspective. Yes, it tells me wrong things sometimes. But the productivity gains of reading my database and understanding (through extended properties) the business it supports, far outweigh the annoyances. Instead of BI queries that would have taken me days to write/debug and still performed terribly, I now have an initial design within minutes and an eloquent, (mostly) performant solution in a few hours.
The SSMS Copilot log files show some extra instructions being fed in the background. They all look like guard rails though, not marketing. Perhaps there are still more, buried more deeply, that we can’t see.
[…] SSMS Copilot is Messing With Your AI Prompts. (Brent Ozar) […]
Thanks for this interesting article! I wonder if rather than changing the prompt in some nefarious way, CoPilot might be changing the ai engine from the selected one and without notice to the user in order to save money with a cheaper (to MS) or less busy engine.
:-O . That sounds likely. Any chance to destroy quality and redirect those $$$’s to profits…
We actually do. The chat logs with all prompts, tool calls, and executed sql queries are stored in %localappdata%\SSMSCopilot.
This is the system prompt from my installation (SSMS 22.6.0):
https://pastebin.com/Ehnti2TR
Ooo, that’s awesome… that it’s stored in plain text on … hmmm … I’m not sure that’s actually awesome, but on the bright side, at least we can review that prompt and see if we can find the root cause for the bias. I’ll leave that to someone else, out of the country on vacation this week, but it looks intriguing. There’s a lot of stuff in there!