How Llms.txt And Robots.txt Affect AI Crawlers

What We Genuinely Do Not Know Several things are worth admitting rather than papering over. We do not know how the systems weight their signals against each other. We do not know how much residual influence training data has once retrieval is involved. We cannot reliably distinguish a change in your visibility from a change in the model's behaviour.

Ask how they will handle being wrong. Every engagement in this field produces at least one confident recommendation that does not work, because the systems change and the published research is thin. What matters is whether that gets reported or quietly dropped from the next deck, and asking the question directly at the outset makes it considerably more likely to be reported.

Entity Coherence Before a model can recommend you it has to be confident that the scattered mentions of your name refer to one company. That confidence comes from consistency across the details that identify you.

Corroboration Beats Assertion The single clearest pattern in observed behaviour is that independent agreement outweighs self description. A claim made only on your own site is treated as a claim. The same claim appearing on a review platform, in a trade publication and in a forum thread is treated as a fact about the world.

Ask sales to note the question asked on every call for a month, in the prospect's words rather than paraphrased. Export support tickets and sort by frequency. Pull the query report from Search Console. And read the first message from inbound enquiries before anyone has reshaped it.

What llms.txt Proposes It is a proposed convention: a file at your root offering a curated, plain text guide to your site for language model consumers, pointing at the documents you consider authoritative.

Why Real Questions Beat Generated Ones Questions produced by keyword tools are smoothed. They use category vocabulary, they avoid awkward specifics, and they tend to be the questions everyone has already answered.

Direct Answers Beat Positioning When a model composes a recommendation it needs sentences it can attribute. Positioning language supplies none. A paragraph about being a trusted leader committed to excellence contains no attachable claim, so it is passed over in favour of a competitor who wrote down their turnaround time.

Ask to See Their Own Position This one is unfair and revealing. Ask an assistant to recommend an agency for this kind of work, using a prompt a buyer would write, and see whether the company sitting in front of you appears.

The idea is reasonable and adoption is inconsistent. Support varies by provider and no major system currently treats it as required. Treat it as a cheap and speculative addition rather than a deliverable worth paying much for.

Ask What Happens in Month One A proposal that opens with content production has skipped the diagnosis. There is no way to know what to write before you know which questions matter, which assistants answer them badly and which sources they draw on.

Good versions read like this: mention rate on evaluation prompts rose from two in fifteen to six in fifteen, which we attribute to the three directory corrections completed in week two, though a competitor also stopped publishing during the same period.

The Blocks Nobody Chose Most blocking discovered during audits was never a decision. A disallow copied from a template. A staging rule that survived a migration. A security plugin with an aggressive default. A content delivery network setting labelled bot protection with a switch nobody has looked at since launch.

Deciding Whether to Block Anything There is a legitimate argument for restricting training crawlers, particularly for publishers whose archive is the product. That is a commercial and editorial decision and it deserves a real discussion rather than a default.

This is the least interesting subject in the discipline and the one that most often explains a total absence from generated answers. A brand can do everything else correctly and remain invisible because a line in a text file, or a setting nobody remembers enabling, is turning the relevant crawlers away.

Nobody outside the labs has the full picture, and anyone claiming otherwise is guessing with confidence. What we do have is a large volume of observable behaviour, published research and the citations that several assistants display openly, and those three together support some reasonably firm conclusions.

Ask How They Price It Retainers dominate this field and mostly make sense, because the work is continuous and the third party portion is slow. What matters is what the retainer covers and whether the scope is written down in units you can count.

This explains the most common frustration brands report, which is watching a competitor with a worse website get your brand recommended by ChatGPT recommended instead. That competitor is usually not better optimised. They are more written about, and the system is weighing the difference.