How Llms.txt And Robots.txt Affect AI Crawlers

Local businesses have an unusual position here. They are more exposed than most, because a large share of local intent queries are exactly the who should I use questions that assistants answer directly, and they also have a shorter route to fixing it than a national brand does.

One warning about testing. If you fix something and immediately re-run a prompt in the same session, the assistant may repeat its earlier answer from context rather than retrieving afresh. Start a new session, and run the prompt several times, before concluding that nothing changed. answer engine optimization

This matters more than any subtlety about model training. It means recommendations are built largely from pages that exist right now, which is why a page published this month can influence an answer this month, and why a brand absent from the retrievable web is absent from the answer regardless of how well known it is offline.

Write these plainly and prominently. A page that says we serve the wider area and offer competitive pricing contains nothing a model can use. A page that says we cover a fifteen mile radius, charge a fixed call out fee, and can usually attend within four hours can be quoted directly into an answer.

Check your robots file, then check your server logs for the relevant agents and see what status codes they receive. A site that returns a challenge to every non-browser request is invisible to this entire channel, and nobody involved will have thought of it as a marketing decision.

Fragmented identity produces a specific symptom worth recognising: an assistant knows facts about you but attributes them vaguely, or confuses you with a similarly named business. The fix is dull consistency work across every place your name appears.

There is almost always a specific, findable reason for this, and it is rarely that the model dislikes you. Here are the causes worth checking, roughly in the order that they tend to be responsible. answer engine optimization

That is a month of intermittent effort, it costs almost nothing, and in most local categories it is enough to change what an assistant says. Local is one of the few places where the whole discipline is genuinely accessible without an agency. answer engine optimization

The result is a content programme aimed at guesses. Sometimes it works by accident. Usually it produces pages nobody retrieves, and the diagnosis that would have directed the effort correctly costs a fraction of what the content did.

What can legitimately be committed to is process: the prompt set will be run on a schedule, the raw answers will be kept, specific technical fixes will be made by a date, a defined number of third party listings will be corrected. Commitments about inputs are honest. Commitments about outputs are not.

Which to Fix First Work in that order, because the sequence is roughly cheapest to most expensive and each step is wasted without the one before it. There is no value in earning press coverage if the crawler cannot reach the page it points at.

A page worth having states what you do in that area specifically: which neighbourhoods, what travel time, what jobs are common there, what the local constraints are. If you cannot write anything genuinely local about a town, the honest answer is not to publish a page for it.

The Details That Get Quoted Locally Local recommendations turn on practical specifics, and most local sites omit all of them. Your actual coverage radius. Whether you handle emergency call outs and at what hours. Typical price range for a common job. Whether you are licensed, insured and to what level.

In most categories the pages that generate answers are review platforms, directories, forum threads, comparison articles and trade publications. Being absent or wrong on those explains far more absences than anything on a brand's own site, and correcting a listing costs an afternoon.

The shortlist is shorter than a conventional local results page, which raises the stakes on being included. Being fourth on a map still gets calls. Being fourth in a recommendation that names three businesses gets none.

Nobody Independent Talks About You This is the cause most brands resist hearing. Assistants lean heavily on third party sources when making recommendations, because a company describing itself is a weak signal. If no review platform, directory, forum thread, comparison article or publication mentions you, there is nothing to corroborate your claims.

Expect the timeline to be uneven. Crawler access can change what an assistant sees within days, because retrieval happens at answer time. Identity consistency takes longer, since scattered mentions have to be re-crawled before they join up. Third party coverage is slowest of all and is the part you control least directly, which is exactly why it is worth starting on it before you need the result.

Insist on the raw answers. If a report cannot be disagreed with, it is not a report. This single requirement filters out most of the weak offerings in the market without needing any technical knowledge.