Learn how to make your WordPress site discoverable by AI crawlers like ChatGPT, Google Gemini, and Perplexity using Rank Math. Step-by-step guide with robots.txt, schema, and llms.txt
AI-powered search is no longer the future, it’s the present. Tools like ChatGPT Search, Google Gemini, and Perplexity are pulling live web data to answer user queries. If your WordPress site isn’t optimized for AI crawlers, you’re invisible in this new search landscape.
Good news: Rank Math makes it easy to control how AI agents crawl, index, and cite your content. In this guide, we’ll walk you through every step using examplewebagency.com, a fictional Montreal-based WordPress agency as our live example.
Step 1: Configure robots.txt with AI Crawler Rules
Your robots.txt file is the first line of defense (and opportunity) for AI bots. Rank Math lets you edit it directly from the dashboard.
Rank Math → General Settings → Edit robots.txt
Recommended AI-Optimized robots.txt
# Allow search AI crawlers (indexing & citation) User-agent: OAI-SearchBot Allow: / User-agent: Google-Extended Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / # Block training crawlers (prevent data scraping) User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / # Standard WordPress protections User-agent: * Disallow: /wp-admin/ Disallow: /?s= Allow: /wp-admin/admin-ajax.php Sitemap: https://examplewebagency.com/sitemap_index.xml
OAI-SearchBot to appear in ChatGPT results, but block GPTBot to prevent your content from training OpenAI models.Step 2: Add llms.txt – The New Standard for AI
The llms.txt file is like robots.txt but specifically for large language models. Rank Math doesn’t support it natively yet, but you can add it manually.
- Go to Appearance → Theme File Editor
- Create a new file:
llms.txtin your root directory - Paste this content:
User-agent: * Allow: query Disallow: train Disallow: store Contact: https://examplewebagency.com/contact Policy: https://examplewebagency.com/ai-usage-policy
Place llms.txt at https://examplewebagency.com/llms.txt
Step 3: Add Schema Markup with Rank Math
AI agents love structured data. Rank Math’s Schema Generator makes it point-and-click.
How to Add Organization & Service Schema
- Edit your homepage
- Scroll to Rank Math → Schema
- Select Organization
- Fill in:
| Field | Value |
|---|---|
| Name | Example Web Agency |
| URL | https://examplewebagency.com |
| Logo | /wp-content/uploads/2023/01/logo.png |
| Address | 123 Rue Example, Montreal, QC H3B 1X1 |
| Telephone | +1-514-555-0192 |
AI agents use this to cite your business accurately.
Step 4: Submit Sitemap & Monitor AI Crawlers
Submit to Search Consoles
- Google Search Console: Submit
sitemap_index.xml - Bing Webmaster Tools: Same URL (critical for ChatGPT)
Rank Math auto-generates your sitemap
Track AI Bot Visits
Use MonsterInsights or server logs to monitor:
| Bot | User Agent | Purpose |
|---|---|---|
| ChatGPT | OAI-SearchBot | Indexing for answers |
| Gemini | Google-Extended | AI overview data |
| Perplexity | PerplexityBot | Real-time search |
Proof your AI optimization is working.
Step 5: Write AI-Friendly Content
AI doesn’t just crawl, it understands. Structure your blog posts like this:
- H1: Clear, keyword-rich title
- H2/H3: Answer user questions directly
- Bullet points & tables: Easy to parse
- Internal links: Build topical authority
Final Checklist: Is Your Site AI-Ready?
| ✅ | robots.txt allows OAI-SearchBot, blocks GPTBot |
| ✅ | llms.txt published with clear policy |
| ✅ | Organization + Service schema via Rank Math |
| ✅ | Sitemap submitted to Google & Bing |
| ✅ | AI crawler traffic visible in logs |
| ✅ | Content uses H2s, lists, and schema |
Start Getting Cited in AI Answers
Within 7–14 days of implementing these changes, your site can appear in:
- ChatGPT’s “Browse with Bing” results
- Google’s AI Overviews
- Perplexity’s real-time answers
Ready to dominate AI search? Install Rank Math PRO and follow this guide. Need help? Book a call with Dasweb for a free AI SEO audit or technical assistance.
