Skip to content
Book a strategy call
DARKSTAR_INSIGHTS 9 MIN READ AI & Technical SEO

AI Crawlers and Your Law Firm Website: What You Need to Know

Which AI bots crawl your site, what blocking each one really does, and the settings that quietly keep law firms out of AI answers.

Open your server logs and you will find visitors named GPTBot, ClaudeBot, OAI-SearchBot and PerplexityBot. They belong to OpenAI, Anthropic and Perplexity, and they collect pages for two different jobs: training AI models and answering questions in AI search tools. Google runs its AI features off the same Googlebot that already crawls your site.

Each company now publishes a separate name for each job, so you can allow one and block another. That matters for a law firm, because the bots that decide whether your practice-area pages can be cited in ChatGPT, Claude or Perplexity answers are not the same bots that collect training data. This guide covers which bots exist, what blocking each one actually does, the settings that block them by accident, and how to tell whether any of it is working. For the wider picture, see our guide to AI for law firms.

Which AI bots visit a law firm website?

The table below lists the user agents the major vendors document today. The names matter because robots.txt rules are written per user agent.

User agentCompanyWhat it doesFollows robots.txt?If you block it
GPTBotOpenAICollects content for training OpenAI’s modelsYesYour content should not be used in training
OAI-SearchBotOpenAIFinds pages to show and link in ChatGPT searchYesYour site will not be shown in ChatGPT search answers
ChatGPT-UserOpenAIFetches a page when a ChatGPT user’s request needs it“May not apply,” per OpenAILimited effect; it is user-initiated
ClaudeBotAnthropicCollects content for training ClaudeYesFuture content excluded from training
Claude-SearchBotAnthropicIndexes pages to improve Claude’s search resultsYesYour pages are not indexed for Claude search
Claude-UserAnthropicFetches a page when a Claude user asks a questionYesClaude cannot retrieve your page for that user
PerplexityBotPerplexityFinds and links websites in Perplexity results; not used for trainingYesYour pages drop out of Perplexity’s index
Perplexity-UserPerplexityVisits a page when a Perplexity user asks a questionGenerally noLittle effect
GooglebotGoogleCrawls for Google Search, including AI Overviews and AI ModeYesYou leave Google Search entirely
Google-ExtendedGoogleNot a crawler; a robots.txt token for Gemini training and groundingYesNo effect on Search or AI Overviews

Sources for each row: OpenAI’s crawler overview, Anthropic’s help article on its bots, Perplexity’s bot guide and Google’s list of common crawlers.

Rows of black server cabinets in a data center with red and blue network cables running into an open rack
Every request from an AI bot lands in your server logs with its user agent name, which is where an audit starts.

Does blocking Google-Extended keep you out of AI Overviews?

No. This is a common mistake in law firm robots.txt files, and an earlier version of this article got it wrong too.

Google says Google-Extended controls whether content it crawls can be used to train future Gemini models and to ground answers in the Gemini apps and Vertex AI. In Google’s words, it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”.

AI Overviews and AI Mode are part of Google Search. Google’s page on AI features and your website says the controls are the same ones you already use for Search: robots.txt rules for Googlebot, plus nosnippet, data-nosnippet, max-snippet and noindex. To be eligible as a supporting link, a page only has to be indexed and eligible to show with a snippet. Google adds that there are no extra technical requirements.

So a firm that blocks Google-Extended has opted out of Gemini training and nothing else. A firm that adds nosnippet to its practice-area pages has removed itself from AI Overviews and shortened its regular search snippets at the same time. Check for both.

Should a law firm block AI crawlers?

Split the question in two, because the vendors now let you.

Search and answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot) decide whether your pages can be found and linked inside AI answers. Blocking them removes your firm from those answers. OpenAI states it plainly: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. For a firm that publishes practice-area pages to attract clients, we recommend allowing all of them.

Training bots (GPTBot, ClaudeBot, and the Google-Extended token) decide whether your content goes into future models. That is a business judgment, not an SEO requirement. Blocking them does not remove you from any of the search tools above. Some firms block training because their articles are a real asset and they would rather not hand them over for free. Others allow it on the theory that a model trained on their content may describe their practice areas accurately. Nobody has published evidence that either choice changes how often a firm gets recommended, so do not let a vendor sell you certainty on this one.

Our default for most firms: allow the search bots, decide on the training bots with the partners, and write the decision down so the next web developer does not reverse it by accident.

What does a sensible robots.txt look like?

Your robots.txt file sits at the root of your domain (for example, yourfirm.com/robots.txt). Any bot not named is allowed by default. A firm that allows AI search but opts out of AI training could use:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://www.yourfirm.com/sitemap.xml

Delete the first three blocks if the partners decide to allow training. Two cautions apply either way.

First, robots.txt is a request, not a lock. Google’s own robots.txt introduction says it “is not a mechanism for keeping a web page out of Google” and recommends password protection for anything private. Client portals, intake uploads and internal documents need a login, not a Disallow line.

Second, user-initiated fetchers behave differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. If a page must stay private, it cannot be public.

Code editor in dark mode showing the head section of an HTML file with meta charset, viewport and stylesheet tags
A single meta tag or robots.txt line can remove a practice-area page from AI answers, so read the source, not just the page.

What might be blocking AI bots without you knowing?

Many firms never wrote a rule about AI bots and are blocking them anyway. Check these four places.

  1. Your CDN or firewall. On July 1, 2025, Cloudflare announced that every new domain signing up would be asked whether to allow AI crawlers, with blocking as the starting point. If your site moved to Cloudflare after that, look at its AI crawler settings. A firewall block happens before robots.txt is ever read.
  2. Security plugins. WordPress security and bot-management plugins often block unfamiliar user agents. Check their logs for 403 responses to the bots in the table.
  3. Robots.txt written by someone else. Themes, SEO plugins and past agencies add rules. Read the live file, not the one you think you have.
  4. Page-level tags. Search the page source for noindex, nosnippet and max-snippet:0. Each one limits how Google can use the page in Search, including AI features.

Is llms.txt worth adding?

llms.txt is a proposal published by Jeremy Howard in September 2024 for a Markdown file at your site root that summarizes your site and links to its key pages. The llms.txt proposal says it was meant mainly to help AI tools at the moment they answer a question, not for training.

Be realistic about it. Google’s guide to optimizing for generative AI features says you do not need llms.txt or any other new machine-readable file for Google Search, and Google’s documentation changelog adds that these files will not help or hurt your rankings, though it is fine to keep one for other systems. A short llms.txt listing your firm name, offices, practice areas, attorneys and best pages takes an hour and costs nothing to maintain. Just do not expect it to move results by itself.

What makes an AI tool cite a law firm page?

No vendor publishes a citation formula, so be wary of anyone who claims one. What Google has published for its own AI features is useful, and it matches what works elsewhere.

  • Write content nobody else has. Google’s generative AI guide asks for “non-commodity content” with a unique point of view. A page that explains how your county’s court actually handles a first DUI beats a page that restates the statute.
  • Do not rewrite for robots. The same guide says you do not need to chunk content into tiny pieces, add special schema, or rewrite pages just for AI systems. Clear headings and direct answers help readers, and that is reason enough.
  • Do not mass-produce near-duplicate pages. Google warns that creating separate content for every possible query variation can violate its scaled content abuse policy. Fifty city pages with swapped place names are a risk, not a strategy.
  • Show who wrote it. Attorney bylines linked to real bio pages help readers judge the source, and they tie into Google’s E-E-A-T guidance.
  • Be careful with results. Case results and testimonials are attorney advertising. Most states restrict how you present them and require disclaimers. Check your state’s version of the advertising rules before you publish results to win citations.

Earning mentions on other trusted sites still matters too. That is the work behind our backlinks and AI citations service.

How do you know if any of this is working?

Use three sources of data, and do not mix them up.

Server logs show crawling. Search your access logs for each user agent in the table. You want to see the search bots fetching your practice-area pages with 200 responses. A sudden stop usually means a robots.txt, firewall or hosting change. Your host can export the logs if you do not have access.

Search Console shows Google’s AI impressions. In June 2026 Google introduced a generative AI performance report in Search Console and rolled it out to all sites by August 31, 2026. It counts impressions, meaning how often your pages appeared in AI Overviews and AI Mode, with breakdowns by page, country, device and date. It does not report AI clicks separately. Those clicks stay inside the regular Performance report under the Web search type.

Analytics shows visits from AI tools. In GA4, look at referral traffic from chatgpt.com, perplexity.ai, claude.ai and gemini.google.com. The numbers are usually small. What counts is whether those visitors call or fill out a form, so tie the referral to your intake records.

Then check by hand once a month. Ask ChatGPT, Claude, Perplexity and Google the questions your prospects ask, in your city, and note which firms get named and linked. Keep the log. It will show you which pages earn citations and which competitors you are up against. Our guide to Google’s AI Overviews covers the Google side in more depth.

A 30-minute checklist

  1. Open yourfirm.com/robots.txt and list every AI user agent it mentions.
  2. Confirm OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot are not blocked.
  3. Decide with the partners whether to allow GPTBot, ClaudeBot and Google-Extended, and write the decision down.
  4. Check Cloudflare or your firewall and your security plugin for AI bot blocking.
  5. Search your top ten pages for noindex, nosnippet and max-snippet.
  6. Make sure anything private is behind a login.
  7. Open the generative AI report in Search Console and note your baseline.

Sources

  1. OpenAI, “Overview of OpenAI Crawlers,” accessed November 2026
  2. Anthropic, “Does Anthropic crawl data from the web, and how can site owners block the crawler?,” accessed November 2026
  3. Perplexity, “Perplexity Crawlers,” accessed November 2026
  4. Google Search Central, “Google’s common crawlers,” accessed November 2026
  5. Google Search Central, “AI Features and Your Website,” accessed November 2026
  6. Google Search Central, “Google’s Guide to Optimizing for Generative AI Features on Google Search,” accessed November 2026
  7. Google Search Central, “Latest Google Search documentation updates,” accessed November 2026
  8. Google Search Central, “Introduction to robots.txt,” accessed November 2026
  9. Google Search Central Blog, “Introducing Search Generative AI performance reports in Search Console,” June 2026
  10. Cloudflare, “Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large,” July 2025
  11. Jeremy Howard, “The /llms.txt file,” first published September 2024

Want this handled for your firm?

Darkstar runs marketing for law firms end to end: SEO, content, reviews, ads, and call tracking, with a live dashboard so you always know what is working. Book a strategy call.

Book a strategy call →
JW
Founder, Darkstar. 14+ years engineering growth systems for elite law firms. $500M+ in attributed revenue across 500+ practices.
Share
Own Your Space.

Ready to direct your growth?