I was back home in Ireland for a few days after my dad needed emergency surgery. Thankfully, it all went well, and he’s now home recovering.
While I was there, I read a LinkedIn post from fellow Irishman Kieran Flanagan about the pace of AI that resonated with me. Things are moving at a speed that’s hard to comprehend, never mind keep up with. His advice was practical: pick one area and go deep. And make time to step outside and touch grass.
It wasn’t the reason I’d have chosen to come home, but a few days in the west of Ireland, close to family, helped put things in perspective.
Between work and family commitments, I didn’t get around to last week—or the following week’s—edition of The Second Shift Club. My nine-week streak came to an end. To my rabid fanbase, all 25 subscribers, I’ll try not to let it happen again.
I managed some writing on the journey back to Brussels. A flight without Wi-Fi also gave me time to start rereading Steinbeck’s The Winter of Our Discontent. Probably healthier than another hour spent doomscrolling.
This week, I’m writing about prompt tracking: how to choose the questions worth monitoring, and how much faith to put in the resulting number.
Is any of this worth measuring?
This post from Rand Fishkin sums up much of the industry’s scepticism with prompt tracking. (Rand has been a pretty consistent and fair critic of the AEO industrial complex.)
His argument, as I interpret it, is that AI visibility scores tell us little about commercial impact. We don’t know how often real customers ask those questions or whether the answers influence what they buy.
I sympathize with a lot of his reasoning. But allow me to gently push back.
If your customers are using AI to explore problems, compare options and choose vendors, understanding how you appear in those conversations seems worth some investment. You don’t need to have attribution completely solved before you start learning.
That won’t tell you the size of the opportunity or how many sales it generated. But it can tell you something useful.
Provided you’ve picked sensible prompts of course.
Our first prompt list wasn’t great
About a year ago, we built our first tracker with 141 prompts. We were scrambling to get it running, so we picked prompts based on keywords we cared about.
The problem is that an AI search engine can run several searches before answering you. What it searches for depends on what you’re trying to do.
Ask for the best product and it may look for comparisons and vendor reviews. Ask how to set something up and it may look for instructions and developer docs.
Different questions surface different pages and different brands much like your grandparents’ Google algorithm.
When I pulled the engine’s searches across our 141 prompts, just 1.9% were how-to queries. My hunch is that people ask LLMs more “how do I achieve this things?” questions than “what’s the best X?” questions. Our prompt list barely reflected that.
So I rebuilt the list around one rule: track what buyers are trying to do and not just the keywords they might use.
Use your buyers’ words
Start with what you sell. Well, duh.
But then find out how buyers describe it. Your carefully chosen product names and feature labels may bear very little resemblance to what someone types when they need help.
Sales call transcripts, customer emails and Search Console queries are good places to look. What do people call the product? How do they describe the problem?
I used our truth layer, our documented source of product information, to establish what we offer. Then I used buyers’ language to turn that into prompts.
Don’t fudge the numbers
“What is [brand]?”, “Best alternatives to [brand]”, “[Brand] vs [competitor]”.
These are useful questions. But if you put your brand in the question, you can’t count its appearance in the answer as evidence that the engine discovered you.
So I split the list.
Discovery prompts that name no vendors. Do we show up when nobody mentions us?
Branded prompts that are likely to name us, a rival or both. What does the engine say about the companies the buyer is already considering?
Both are worth tracking, but just shouldn’t be mixed into one visibility score.
Determine the questions that belong
For discovery prompts, I set a threshold: the engine had to name vendors in at least half the answers I sampled.
Of the 129 prompts in the new draft, 25 failed. They returned definitions, regulations or technical explanations without naming providers.
Fine questions, but expensive to track if you want to get granular on your “buying-intent” visibility.
This doesn’t mean removing every how-to question. Some practical answers name providers and cite their documentation. Others explain the process without mentioning a company. This is why you have to check.
You also have to count carefully. A link to a company’s website isn’t the same as naming it in the answer. And naming it isn’t necessarily recommending it.
Different wording doesn’t always mean a different question
Ask an AI engine the same question twice and you can get different searches, sources and recommendations.
So how do you tell whether two prompts add useful coverage?
I ran each prompt several times and recorded:
What the engine searched for.
Which websites it cited.
Which vendors it named.
First, I measured how much the results varied when I repeated one prompt. Then I compared different prompts against that.
If two prompts produced results as similar as repeated runs of a single prompt, I treated them as duplicates.
I also included checks: reworded versions of the same question and pairs of obviously unrelated questions. If the method couldn’t tell those apart, I discarded the run.
Small wording changes can matter
Consider an illustrative example: “I need to accept payments on my website. What should I use?”
It sounds practical, but “What should I use?” asks for a product recommendation. You may get a list of payment providers instead of setup instructions.
“How do I add payments to my website?” makes the implementation question clearer.
Other changes may do very little. “Best payment processor” and “best service for accepting online payments” could return much the same recommendations.
Adding “for a growing website” might change nothing. Specifying a country will likely change which providers are recommended.
Don’t get carried away with the deleting
You still need enough distinct questions in each category to tell whether a change is meaningful or just normal variation.
Where I had too few distinct questions, I included them in a broader reporting category instead of giving them their own score.
Testing all 129 prompts five to ten times each, including automatically grading the answers, cost about $20 in API usage.
If you want to try this
Start by looking at the answers and searches behind whatever you track today. Are you covering real buyer questions, or mostly variations of “best product”?
Then:
Find your buyers’ language. Use customer conversations and search queries to describe what you sell.
Separate discovery from branded questions. Being suggested unprompted and being discussed by name measure different things.
Sample the answers. Check that each prompt tells you something about the visibility you want to measure.
Test for duplicates. Compare the results, not just the wording.
Keep enough distinct questions per category. Combine small categories where a separate score would be misleading.
Hold the list steady. Record changes so you know what you’re comparing.
The original keyword list took an afternoon. Rebuilding it took a few days and about $20 in API calls. At least now, if someone asks why we’re tracking these prompts, I have an answer.