.png){kind=icon}
.webp)
A patient in Calgary asks ChatGPT for a physiotherapist near her office, and three local clinics appear in the answer. Your practice may be missing for a purely technical reason. Many developers copy a block AI crawlers robots.txt template without checking which bots it stops. A short review lets you allow ChatGPT to crawl website pages patients need, while you keep control over AI training.
The Quick Answer on AI Crawler Access for Clinics
Some hosting and security settings, including Cloudflare AI bot controls, can stop AI tools from reaching a clinic site without anyone noticing. The robots.txt rules and the firewall settings together decide which bots get through.
Most practices do best when search bots such as OAI-SearchBot and PerplexityBot have open access. A clinic can treat training bots as a separate decision, because a training block does nothing to hurt search results.
Key Takeaways
- Search bots such as OAI-SearchBot and PerplexityBot decide whether a clinic appears in AI answers.
- A GPTBot block keeps content out of AI training and leaves ChatGPT answers untouched.
- Google-Extended covers Gemini training only and has no effect on Google Search rankings.
- A training block in Cloudflare can now stop Googlebot along with AI training crawlers.
- Private patient areas need a login, because a robots.txt file only asks bots to stay away.
- A quick look at server logs every few months confirms that AI tools still reach the site.
How AI Search Tools Read Your Clinic Website
AI tools learn about a clinic through small programs called crawlers. Each crawler carries a name, its user agent, with every page request it makes.
These visitors fall into three groups. One group collects public text to train future AI models. Search crawlers build the index that ChatGPT search and similar tools use to cite sources.
The third group fetches a page live when a patient asks an AI assistant about a specific clinic. ChatGPT-User and Perplexity-User work this way, and robots.txt rules may not apply to them because a person started the request.
Many AI crawlers read only the raw HTML and skip heavy scripts. Hours, location and treatments in plain text give these tools far more to work with.
Also Read: Why Your Clinic Isn't Appearing in Google AI Overviews or ChatGPT
Settings That Can Block AI Crawlers Without You Knowing
Very few clinic owners decide to hide from ChatGPT on purpose. The block usually comes from a default setting, an old plugin or a security rule that someone switched on years ago.
- Old robots.txt templates often place GPTBot, OAI-SearchBot and PerplexityBot under one shared Disallow rule.
- Cloudflare AI bot controls now sort bots into search, agent and training groups. A training block in these controls can also stop mixed crawlers such as Googlebot and Bingbot since September 15, 2026.
- Strict firewall rules and security plugins sometimes challenge any visitor that looks automated, and AI crawlers fail those checks.
- Country blocking cuts spam for many Canadian clinics, but it also turns away AI crawlers that run from servers outside Canada.
- Separate booking subdomains follow their own robots.txt rules, so the main site file has no say over them. The site still loads normally for patients, so nobody sees a warning. That silence makes the problem easy to miss for months.
.webp)
Search Bots vs Training Bots and What Clinics Should Allow
The most useful rule for clinics is simple. Search bots bring visibility, while training bots only feed future AI models.
Each major AI company now runs these jobs through separate user agents. That split lets a clinic welcome search bots and still say no to training.
OpenAI Crawlers Behind ChatGPT
OpenAI runs three main agents for its products. OAI-SearchBot finds pages for ChatGPT answers, and sites that block it stop appearing in those answers apart from basic navigational links.
GPTBot collects content for training future models. A clinic can block GPTBot and keep full visibility in ChatGPT search.
ChatGPT-User handles live page visits that a person requests inside ChatGPT. Robots.txt may not stop it, since the visit begins with a human request.
robots.txt Example for ChatGPT Visibility Without Training
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
Google Crawlers and the Google-Extended Token
Googlebot crawls pages for Google Search, and its rules cover every Search feature, including AI Overviews. A clinic that blocks Googlebot loses far more than AI visibility.
Google-Extended works differently, because it acts as a control token rather than a separate crawler. This token decides whether Google may use a site's content to train Gemini models and to ground answers in the Gemini app.
Google-Extended has no effect on rankings or on inclusion in Google Search. A clinic can opt out of Gemini training and still appear in every Google result it earns.
robots.txt Example for Google Search Without Gemini Training
User-agent: Googlebot
Allow: /
User-agent: Google-Extended
Disallow: /
Perplexity and Other AI Search Bots
PerplexityBot builds the index behind Perplexity answers and follows robots.txt rules. Perplexity-User fetches pages live for a user's question and generally ignores robots.txt.
Anthropic splits its crawlers the same way. Claude-SearchBot improves search results in Claude, while ClaudeBot gathers training content. Common Crawl runs CCBot, a public archive that many AI developers use for training.
robots.txt Example for Perplexity, Claude and Common Crawl
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
AI Crawler Cheat Sheet for Canadian Clinics
The table below sums up the main bots a clinic will meet and a sensible starting point for each one.
The Allow rows protect AI search visibility for any clinic that wants patients to find it. The training rows stay a business choice.
Rankingeek reviews robots.txt, firewall and CDN settings for clinics across Canada as part of a free audit.
How to Check and Fix Your robots.txt File
Every website keeps this file at one fixed address, such as yourclinic.ca/robots.txt. Anyone can open that address in a browser and read the rules in plain text.
Two patterns cause most of the trouble here. An asterisk group with Disallow and a single slash shuts every crawler out of the whole site. A named AI bot above that same line loses access to every page.
Patient portals, intake forms and booking confirmations belong behind a login. A robots.txt file is public and voluntary, so it never replaces real security for health information.
A crawler follows only the group that names it most specifically. A portal rule under the asterisk group therefore does not carry over to a named bot group.
The sample below opens public pages to search bots and keeps private paths closed.
User-agent: *
Disallow: /patient-portal/
Disallow: /intake-forms/
Allow: /
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Disallow: /patient-portal/
Disallow: /intake-forms/
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Sitemap: https://www.yourclinic.ca/sitemap.xml
The sitemap line points crawlers to every public service and location page. ChatGPT usually reflects a robots.txt change within about a day.
Firewall and CDN settings need the same review, because a welcomed bot can still hit a block at the server. The CDN dashboard and security plugin log show which bots met a block.
.webp)
Testing Whether AI Tools Can See Your Site
A fix only counts once the bots actually get through. Three simple checks give a clear answer.
- Google Search Console offers a robots.txt report that shows when Google last fetched the file and flags any errors.
- Server logs or the CDN dashboard reveal visits from each AI bot along with the status code it received. A 200 code means the bot read the page, while a 403 code means something stopped it.
- A direct question in ChatGPT or Perplexity that names the clinic, the service and the city shows whether the tool cites your pages. Plugin updates, host moves and new security rules can undo a good setup. A crawler check every few months keeps the clinic visible.
Also Read: ChatGPT Sending Patients to Your Competitors? What Canadian Clinics Need to Do About AI Search in 2026
Final Thoughts on Opening Your Clinic Website to AI Search
Patients in Canada now ask AI tools for clinic recommendations at all hours. A few lines in a robots.txt file can decide whether your practice appears in those answers.
The safest setup for most clinics is clear. Search bots get open access to public pages, private paths stay closed to everyone, and training remains a separate decision.
Rankingeek works as an AI search optimization agency for healthcare practices across Canada. Our team audits crawler access and page structure, then fixes whatever keeps your clinic out of AI answers. This process helps clinics allow ChatGPT to crawl website pages that matter while patient areas stay private.
Frequently Asked Questions About AI Crawler Access
Does Blocking GPTBot Remove My Clinic From ChatGPT?
A GPTBot block does not remove a clinic from ChatGPT answers. OpenAI uses a separate crawler, OAI-SearchBot, to find pages for search results. The GPTBot rule only affects whether content goes into future model training.
Does Google-Extended Affect Google Search Rankings?
Google-Extended has no effect on rankings or on whether a page appears in Google Search. The token only covers Gemini model training and grounding in the Gemini app. AI Overviews follow Googlebot rules instead.
Is Cloudflare Blocking AI Bots on My Site?
The answer depends on the settings your clinic or developer picked. A training block in Cloudflare can now stop crawlers that serve search and training together, including Googlebot. The AI crawler section of the dashboard lists each bot with its current status.
Should a Clinic Block AI Training Bots?
That choice depends on how comfortable the clinic feels about AI training. A training block leaves search results untouched, so many practices block training and allow search. Pages with patient stories or staff details deserve extra thought before any decision.