Who reads your website when you're not looking

Imagine a prospective client no longer googling "who services servers in town" but simply asking ChatGPT or Claude — and getting a ready-made answer with two or three names. Where did the assistant get those names? It read them somewhere. And if what it read included someone's website but not yours, the answer is more likely to feature a competitor than you.

This isn't tomorrow; it's already today. And it's quietly changing what it means to "be online."

Readers you don't see

When a person visits a website, that shows up in ordinary analytics. But websites are read by more than people. Every day they're crawled by programs — and among them, more and more that gather material specifically for AI assistants: ClaudeBot from Anthropic, GPTBot from OpenAI, PerplexityBot and several others.

They name themselves honestly in the server's technical records (the logs): you can see which bot came, when, and which pages it fetched. So these readers can be seen — you just have to look in the right place. Most businesses never look there, and don't even suspect their site may well already be a source for someone's AI answer — a good one or a garbled one.

And they don't read in passing. The same bot comes back again and again, picks up new and updated pages, and follows internal links deep into the site. To it, your website isn't a picture but text to be understood and retold. If that text is written confusingly, or hides the main point behind pretty graphics, the assistant will retell exactly that confusion — and your future client will hear it.

You know SEO. Now there's GEO too

For twenty years businesses competed for a spot in search results — that's called SEO. Now a second game has appeared alongside it: visibility in the answers of AI assistants. It's called GEO, and it has its own rules.

The logic is simple. For an assistant to mention you in an answer, it first has to read your website — and understand from it who you are, what you do and for whom. A site built so that it looks good to a human but is unclear to a machine loses in this new game. Knowing which AIs already read you, and what exactly they pull, is an advantage, not an abstraction.

Why llms.txt

There's a simple thing that helps AI readers not get lost — a file called llms.txt. It's a kind of short map of the site, written in plain language specifically for machine readers: who you are, what you do, where to find the main pages. Search engines have read robots.txt for years; llms.txt is younger, not yet supported by everyone, but a very logical convention for the AI era.

Without such a map an AI reader will still try to work it out on its own — from raw pages, menus and random fragments. Sometimes it'll guess right, sometimes not. llms.txt doesn't force it to do anything, but it gives a tidy hint: here's who we are, here's the main stuff, don't wander. A small thing that costs half an hour of work once and then works for you on its own.

We keep such a file on our own site — not because it's trendy, but because we recommend it to clients and check everything on ourselves first. When we tell a client "this works better," there's our own site behind it, not a piece of advice parroted from somewhere.

Letting them in or blocking them is a decision, not an accident

You can ask AI crawlers not to come in — with a line in that same robots.txt. Some do exactly that, afraid the "AI will steal the content." We deliberately do the opposite — we let them in: if assistants increasingly answer instead of a search engine, then being read by them is the new visibility, not a threat.

The main thing here is that it's your conscious decision, not something that happened on its own while no one was looking. Someone has to decide who to let in and who not, and to see who actually came. Without that you don't manage your AI visibility — you simply don't know what's happening to it.

Look at our readers

The best explanation is a live one. We show the AI-bot showcase of our own site: real readers from real logs, by name. In a recent snapshot it logged 2,124 AI-crawler requests out of 2,886 total, from 16 named bots — meta-externalagent alone accounted for 488 of them. It's the same private stack we set up for clients — together with the rest of our live analytics: who visited, which AIs read the site and what background noise we muffle — all live, with no cookie banner and without handing your data to someone else's cloud.

The question used to be "will they find me in search." Now there's another one alongside it: "will the assistants read me — and what exactly will they understand."

Where to start

You don't need to rebuild the site right away. Just take one look at who's already reading it, and whether the machine understands what's written there. That's where work on visibility begins — in search and in AI alike. It's part of what we do in Storefront; and a free audit will show exactly where you should start.