Short answer: AI search engines can only cite what they can fetch, read and connect to your brand. Technically that means six things: let the search and retrieval bots in through robots.txt, serve your key content in the initial HTML, ship one linked JSON-LD graph for your organization and product, write sections that make sense on their own, send accurate freshness signals, and measure AI referrals and bot visits. The code for each step is below, and the full checklist is at the end.
Most GEO advice is about content. Content matters, but on SaaS marketing sites we keep finding the same technical problems first: a robots.txt copied from a 2023 block list, a pricing table that only exists after JavaScript runs, schema injected through Google Tag Manager, and no way to tell whether ChatGPT or Perplexity ever sent a visitor. None of these show up in a normal SEO audit, because Googlebot renders JavaScript and most AI crawlers do not.
This playbook works on any stack: Webflow, WordPress, Next.js or a headless CMS. The examples use example.com; swap in your own domain.
Crawl access: decide which AI bots get in
Every major AI vendor now runs separate agents for training, search indexing and live fetches on a user's behalf. They have different robots.txt tokens, and blocking one does not block the others. That split is the most useful thing to know about AI crawl access:
- OpenAI: GPTBot collects training data. OAI-SearchBot indexes pages for ChatGPT search, and OpenAI says sites that block it will not be shown in ChatGPT search answers. ChatGPT-User fetches a page when a user asks, and OpenAI says robots.txt rules may not apply to it.
- Anthropic: ClaudeBot (training), Claude-SearchBot (search indexing) and Claude-User (user requests). Anthropic says all three follow robots.txt.
- Perplexity: PerplexityBot (index) and Perplexity-User (live retrieval).
- Google: Google-Extended is a control token, not a separate crawler. It controls whether your content is used for Gemini models. It does not remove you from Google Search or AI Overviews, which run on Googlebot.
A common setup for a SaaS marketing site: opt out of training if legal wants that, keep every search and retrieval agent open, and point all of them at the sitemap.
# Training crawlers: your call. Blocking them does not remove you from AI search.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search and retrieval: keep these open if you want to be cited.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xmlTwo traps to check after you deploy it:
- Stale tokens.
anthropic-aiandClaude-Webare retired. Rules that name them do nothing. - Your CDN or firewall. Bot protection in Cloudflare or a WAF can block a crawler that robots.txt allows. Check its bot settings and logs for 403 responses to these user agents.
Want to apply this to your setup?
Rendering: make sure the HTML contains the content
Googlebot renders JavaScript. Most AI crawlers do not. Vercel's crawler analysis found that GPTBot, OAI-SearchBot, ChatGPT-User and ClaudeBot download JavaScript files but don't execute them. Whatever your server sends in the first HTML response is all they read.
On marketing sites the content is usually server-rendered already. The problem is the parts added in the browser:
- Pricing tables or plan details loaded from an API.
- Review, case study and logo widgets embedded through third-party scripts.
- CMS lists that load more items with JavaScript (infinite scroll, "load more" buttons, client-side filters).
- JSON-LD added through Google Tag Manager. It is invisible to any crawler that doesn't run scripts.
Content hidden with CSS, like closed tabs and accordions, is fine. It is still in the HTML. The question is only whether the text exists before scripts run. Test your five most important pages:
# What a non-rendering crawler receives: raw HTML, no JavaScript.
curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.4)" https://www.example.com/pricing > raw.html
# 1) Is the key sentence in the HTML?
grep -c "Plans start at" raw.html
# 2) Is the JSON-LD in the HTML (not injected later by GTM)?
grep -c "application/ld+json" raw.html
# 3) Rough size check: visible words in the raw response
sed -e 's/<[^>]*>/ /g' raw.html | wc -wIf a check fails, move that content into the server-rendered template, or use server-side rendering or prerendering for that route. For widgets you can't change, add a plain-HTML summary next to them: three customer quotes as text, the price as text.
Structured data: publish one linked entity graph
AI engines need to know that the company on your homepage, the product on your pricing page and the profiles on LinkedIn and G2 are the same thing. Separate schema blocks on each page don't say that. A single @graph with stable @id values does:
<script type="application/ld+json">
{
"@context": "https://schema.org",
"@graph": [
{
"@type": "Organization",
"@id": "https://www.example.com/#org",
"name": "Example",
"url": "https://www.example.com/",
"logo": "https://www.example.com/logo.png",
"sameAs": [
"https://www.linkedin.com/company/example",
"https://www.g2.com/products/example",
"https://www.crunchbase.com/organization/example",
"https://www.wikidata.org/wiki/Q000000"
]
},
{
"@type": "WebSite",
"@id": "https://www.example.com/#website",
"url": "https://www.example.com/",
"name": "Example",
"publisher": { "@id": "https://www.example.com/#org" }
},
{
"@type": "SoftwareApplication",
"@id": "https://www.example.com/#product",
"name": "Example",
"applicationCategory": "BusinessApplication",
"operatingSystem": "Web",
"publisher": { "@id": "https://www.example.com/#org" },
"offers": { "@type": "Offer", "price": "49", "priceCurrency": "USD" }
},
{
"@type": "WebPage",
"@id": "https://www.example.com/pricing#webpage",
"url": "https://www.example.com/pricing",
"name": "Example pricing",
"isPartOf": { "@id": "https://www.example.com/#website" },
"about": { "@id": "https://www.example.com/#product" },
"dateModified": "2026-11-16"
}
]
}
</script>- Keep
@idvalues stable. Use the same IDs on every page so all the pages point to one organization and one product. - Use
sameAsfor profiles you control or that describe you accurately. LinkedIn, G2, Crunchbase, GitHub and Wikidata, if you have an entry. - Put it in the page source. Add it in the site-wide head or the page template, not through a tag manager.
- Match the visible page. The price, name and dates in the markup should be the same as on the page.
About FAQPage: since 2023 Google shows FAQ rich results only for a small set of authoritative government and health sites. The markup is still valid and still describes the page, so keep it where there are real questions and answers, but don't expect a rich result from it.
Content structure: write sections that stand alone
AI engines retrieve passages, not whole pages. A section gets cited when it still makes sense after being cut out of the page. On every key page:
- Make headings into the questions buyers ask. "How much does Example cost?" works better than "Simple pricing".
- Answer in the first two or three sentences under the heading, then add the detail. Don't lead with a story.
- Name things in full in every section. Write "Example's SOC 2 Type II report", not "it" or "our report". A passage on its own has no earlier paragraph to refer back to.
- Use specifics with units and dates. "Sync runs every 5 minutes on all plans (November 2026)" can be quoted. "Lightning-fast sync" can't.
- Put comparisons in HTML tables. Plans, integrations and "us vs. alternative" pages are pulled into answers often.
- Use the same names everywhere. The same product name, category and one-line description on your site, LinkedIn, G2 and directories.
The pages that usually earn citations on SaaS sites are pricing, integrations, comparison and alternatives pages, security and compliance pages, and docs. Start there, not with the blog.
Freshness signals: sitemaps, IndexNow and llms.txt
Answer engines prefer current information, and they need a way to find out a page changed.
- Accurate
lastmod. Update it only when the content really changes. A sitemap where every page changes every day teaches crawlers to ignore the field. - A visible "Updated" date on pages where it matters (pricing, comparisons, guides), matching
dateModifiedin the JSON-LD. - IndexNow. Bing, Yandex and other engines accept change notifications. Bing powers Copilot's answers, so a fast Bing recrawl is worth having. Google does not use IndexNow.
# Tell Bing (and other IndexNow engines) a page changed. Google does not use IndexNow.
curl -X POST "https://api.indexnow.org/indexnow" \
-H "Content-Type: application/json; charset=utf-8" \
-d '{
"host": "www.example.com",
"key": "YOUR_INDEXNOW_KEY",
"keyLocation": "https://www.example.com/YOUR_INDEXNOW_KEY.txt",
"urlList": ["https://www.example.com/pricing", "https://www.example.com/blog/new-post"]
}'llms.txt is a proposed Markdown file at /llms.txt that lists your most important pages for language models. None of the major AI search providers has said it uses the file to choose citations. It takes minutes to add, so add it if you like, but don't count on it.
Measurement: AI referrals, bot logs and prompt audits
You can't improve what you don't measure, and AI visibility needs three separate measurements.
AI referral traffic in GA4
Create a custom channel group (Admin → Data display → Channel groups) with a channel called "AI assistants", where Source matches regex:
chatgpt\.com|chat\.openai\.com|perplexity\.ai|gemini\.google\.com|copilot\.microsoft\.com|claude\.aiMove the channel above Referral, because GA4 assigns a session to the first channel whose rules it matches. Expect an undercount: some AI apps send no referrer, so part of that traffic shows up as Direct.
Bot visits in your server or CDN logs
Your logs show which pages the retrieval bots read. GA4 can't, because these bots don't run your analytics script.
# Which AI agents hit the site, and how often (combined log format)
grep -Eio "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user" access.log \
| tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn
# Top pages fetched by the retrieval bots (the ones that feed citations)
awk -F'"' 'tolower($6) ~ /oai-searchbot|chatgpt-user|claude-searchbot|claude-user|perplexity/ \
{ split($2, r, " "); print r[2] }' access.log | sort | uniq -c | sort -rn | head -20User-agent strings can be faked. Where it matters, check the request IP against the vendor's published ranges (OpenAI publishes openai.com/searchbot.json and openai.com/gptbot.json). On hosted platforms without raw logs, use your CDN's logs or the platform's own bot reports.
A monthly prompt audit
Write 15 to 25 prompts your buyers actually ask ("best [category] for [use case]", "[you] vs [competitor]", "[you] pricing"). Run the same set each month in ChatGPT, Perplexity, Gemini, Claude and Google AI Overviews. Log whether you're mentioned, whether you're cited, which of your URLs is cited, and which competitors appear. The cited URLs tell you which pages to improve next.
Technical GEO checklist
| Check | How to verify | Pass when |
|---|---|---|
| Search and retrieval bots allowed | Read /robots.txt | OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user agents are not disallowed |
| No stale tokens | Search robots.txt for anthropic-ai, Claude-Web | None left, or replaced with current tokens |
| CDN or WAF not blocking | Bot settings and logs | No 403s for AI user agents |
| Key content in raw HTML | curl the page and grep | Price, plan names and key claims are found |
| JSON-LD in page source | grep for application/ld+json | Present without JavaScript |
| One entity graph | Validate at validator.schema.org | Stable @id values, sameAs filled in |
| Sections stand alone | Read each H2 section on its own | Question heading, answer first, names in full |
| Accurate freshness | Sitemap lastmod and on-page date | Change only when content changes |
| IndexNow | Ping after publishing | Returns 200 or 202 |
| AI channel in GA4 | Traffic acquisition report | AI assistants channel above Referral |
| Bot log review | Monthly log query | Retrieval bots reach your key pages |
| Prompt audit | Monthly prompt sheet | Mentions and cited URLs tracked over time |
A good order for a team with one sprint: robots.txt and CDN first (an hour), then the rendering checks on your top pages, then the entity graph, then the GA4 channel. Content restructuring and prompt audits come after, because there's no point in better passages that crawlers can't fetch.
Related: Webflow AI chatbot: RAG setup guide · AI workflow implementation · Tell us about your project
Frequently Asked Questions
No. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, which has its own robots.txt token. You can block GPTBot to opt out of training and still allow OAI-SearchBot so your pages can appear in ChatGPT search answers. The same split exists at Anthropic (ClaudeBot vs. Claude-SearchBot) and Perplexity.
Mostly not. Crawl analyses have found that OpenAI's and Anthropic's crawlers download JavaScript files but don't execute them, so they only see the initial HTML. Google is the exception because its AI features use Googlebot, which renders pages. Check a page with curl or with JavaScript turned off to see what a non-rendering crawler gets.
Create a custom channel group with an AI assistants channel whose source matches a regex such as chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and claude.ai, and place it above Referral. Some AI apps send no referrer, so treat the number as a floor, and use server logs to see which pages AI bots fetch.





