How Llms.txt And Robots.txt Affect AI Crawlers

The consistent surprise is that niche trade publications and specialist directories appear far more often than general consumer press. A national newspaper mention is excellent for other reasons and frequently never appears in a citation list, while a sector publication nobody outside the industry has heard of turns up repeatedly.

Why Third Party Comparisons Dominate The comparison pages cited most often are usually not published by any of the companies being compared. A review site, a trade publication or an independent blogger weighing five options reads as disinterested in a way that a vendor's own page does not.

One test of whether a prompt set is any good is to run it and see whether the answers surprise you. A set that returns exactly what you expected is usually measuring your own assumptions, because the questions were written from them. Surprises indicate the prompts reached beyond the company's internal picture of its market, which is the entire purpose.

What We Genuinely Do Not Know Several things are worth admitting rather than papering over. We do not know how the systems weight their signals against each other. We do not know how much residual influence training data has once retrieval is involved. We cannot reliably distinguish a change in your visibility from a change in the model's behaviour.

One brief worth writing once and reusing is a factual sheet for anyone writing about you: canonical name, what you do in a sentence, who you serve, where you operate, when you were founded, who leads it, and three concrete figures you are happy to see quoted. Writers use what is easy to find, and supplying this removes the friction that otherwise produces a paragraph of adjectives.

Set a review cycle, quarterly for fast moving categories and twice a year otherwise. Update the figures rather than the timestamp, and show a real modified date so freshness can be judged honestly. ai seo services

Be prepared for the internal objection that this sends people to competitors. Some of it will, and those are mostly people who would not have bought from you anyway. The trade is that the page becomes usable as an impartial source, which is worth considerably more than the small number of poorly matched prospects it redirects, and the sales team usually agrees once they see which enquiries stop arriving.

Deciding Whether to Block Anything There is a legitimate argument for restricting training crawlers, particularly for publishers whose archive is the product. That is a commercial and editorial decision and it deserves a real discussion rather than a default.

Nobody outside the labs has the full picture, and anyone claiming otherwise is guessing with confidence. What we do have is a large volume of observable behaviour, published research and the citations that several assistants display openly, and those three together support some reasonably firm conclusions.

This matters more than any subtlety about model training. It means recommendations are built largely from pages that exist right now, which is why a page published this month can influence an answer this month, and why a brand absent from the retrievable web is absent from the answer regardless of how well known it is offline.

The reasonable reading is that ranking gets a page considered while quotability and corroboration decide whether it is used. Treating a strong search position as an entitlement to appear in answers is the mistake that catches out established brands most often.

What robots.txt Controls It is a request, honoured by mainstream crawlers, that certain user agents avoid certain paths. It has no enforcement behind it and it does not secure anything, but the major providers respect it.

The decision that almost never makes sense for a commercial business is blocking the agents that fetch pages when composing answers. That is the mechanism by which you get recommended, and turning it off is the equivalent of declining to be listed anywhere, taken quietly, usually by accident.

Blocking these is therefore not one decision. Turning away a training crawler is a defensible editorial position. Turning away the agent that fetches pages at answer time removes you from answers entirely, and the two are frequently confused.

We also know the picture is unstable. Retrieval strategies are revised without announcement, and a method that explained answers well six months ago may explain them poorly today. Anyone selling certainty here is selling something they do not have.

Corroboration Beats Assertion The single clearest pattern in observed behaviour is that independent agreement outweighs self description. A claim made only on your own site is treated as a claim. The same claim appearing on a review platform, in a trade publication and in a forum thread is treated as a fact about the world.

Why the Format Wins When somebody asks an assistant who they should use, the answer required is a comparison. A page that has already performed that comparison, naming specific options and stating how they differ, maps directly onto the shape of the answer being composed.