Every check, and why it is worth what it is worth

Does robots.txt allow the crawlers behind AI answers to read the page?

Assistants that cite the web fetch through named agents. A blanket rule written for search engines years ago can quietly exclude every one of them, and nothing in your analytics will tell you.

If it failsPublish a robots.txt that names the AI agents you want and allows the paths you want cited. Blocking is a legitimate choice — but make it a decision, not an accident.

Full article: robots.txt for AI agentsOpenAI — GPTBotGoogle — Google-ExtendedRFC 9309 — Robots Exclusion Protocol

Does the server answer when the request carries an agent user agent?

robots.txt is a request; bot management is enforcement. A WAF rule or rate limiter can return 403 to exactly the clients you meant to welcome, and the permission you wrote is then worth nothing.

If it failsAllowlist the agent user agents in your bot management, or verify them by reverse DNS instead of blocking on the string. Then re-test.

Cloudflare — verified bots

Does the content arrive without a consent gate or bot challenge in the way?

An agent has no hands. A modal it cannot dismiss, or a challenge page it cannot solve, ends the session on the first request — and the page behind it may as well not exist.

If it failsServe the content in the initial response and let the consent layer appear over it. Never gate the document itself behind a challenge for a well-behaved crawler.

Is there real text and a heading in the raw HTML, before any script runs?

This is the single most common failure on the agentic web. Most crawlers behind AI answers do not execute JavaScript, and an agent that receives an empty shell has nothing to reason about — no matter how good the page looks in a browser.

If it failsRender the page on the server or prerender it at build time. Aim for a real <h1> and several hundred words of the actual content in the first response.

Full article: Content that survives without JavaScriptweb.dev — rendering on the web

Are there landmarks and a heading hierarchy that does not skip levels?

Landmarks and heading order are how a machine separates the article from the furniture. Without them a summariser quotes your cookie banner with the same confidence as your product description.

If it failsWrap the content in <main>, use <header>, <nav> and <footer> for the frame, keep exactly one <h1>, and never jump from h2 to h4.

Is there parseable JSON-LD, and does its type match what the page is?

Structured data is the one place where you get to state facts rather than imply them: this is a price, this is an opening hour, this is the author. Agents and search engines both take it at face value, which is exactly why it must parse.

If it failsEmit JSON-LD in a <script type="application/ld+json"> block with a type that matches the page — Product and Offer for a product, Service for a service page, FAQPage for questions, LocalBusiness for a location.

Full article: Structured data: the one place you state factsschema.orgGoogle — structured data general guidelines

Does the page return Markdown when a client asks for it?

An agent that wants your text does not want your layout. Answering `Accept: text/markdown` with clean prose cuts tokens, removes navigation noise and makes the quotable part of the page unambiguous.

If it failsServe a Markdown representation for the same URL, vary on Accept, and keep it in sync with the HTML. A .md sibling route is an acceptable second best.

Is there a curated file that tells an agent what this site is and where to look?

A crawl gives an agent everything; llms.txt gives it the part you would point at yourself. It is cheap to publish and it is the only place where you control the summary rather than hoping one is inferred.

If it failsPublish /llms.txt with a one-paragraph description, then linked lists of your canonical pages grouped by purpose. Note that it is a convention, not a standard, and Google Search ignores it.

Full article: llms.txt: what it is, and what it is notllmstxt.org

Is there a valid sitemap, and does the page declare which address is the real one?

Agents deduplicate by canonical URL. Without one, the same page reached through three paths becomes three sources that can disagree, and a citation may point at the copy you did not want indexed.

If it failsPublish a valid sitemap.xml, reference it from robots.txt, and put a self-referencing <link rel="canonical"> on every page.

Is there something on this page an agent could complete on the visitor's behalf?

Agent readiness is not a reading exercise. If the page carries no form, no search and no route to a cart or a booking, there is nothing to finish — and the visit ends as a summary instead of a conversion.

If it failsMake the main action part of the served HTML: a real form, a search input, or a link to the booking or cart route. A button that only exists after hydration does not count.

Do the fields carry labels, names, types and autocomplete tokens?

An agent fills a form by reading it. A field with no label and a name like `field_7` is a guess; the same field with a label and `autocomplete="email"` is a certainty. This is the cheapest large improvement most sites can make.

If it failsGive every field a <label for>, a meaningful name, the right input type, and an autocomplete token from the HTML standard.

Conditional — excluded when it does not apply

Full article: Forms an agent can actually fillHTML standard — autofill tokens

Can every interactive control be identified by name rather than by position?

A <div> with a click handler is invisible to anything that does not render and hover. Agents address controls the way assistive technology does — by role and name — so an unnamed control is an unreachable one.

If it failsUse <button> and <a href> for actions, and give each one text or an aria-label. Reserve div-with-onclick for decoration.

Conditional — excluded when it does not apply

Is the main action free of CAPTCHA or a bot challenge?

A CAPTCHA is a statement that automated visitors are unwelcome. That was a reasonable position when automation meant scrapers; it is an expensive one when it means your customer's assistant trying to buy something.

If it failsMove bot defence off the conversion path: rate limiting, honeypot fields and server-side scoring keep the abuse out without ending a legitimate agent session. If a challenge is unavoidable, keep it off the first step.

Are validation states expressed in markup rather than only in colour?

When a submission fails, an agent needs to know which field and why. A red border communicates nothing; aria-invalid and a described-by message let it correct the input and try again instead of abandoning the task.

If it failsSet aria-invalid on the failing field, link the message with aria-describedby, and keep the text next to the field rather than only in a toast.

Conditional — excluded when it does not apply

Does the page register WebMCP tools at all?

This is the difference between an agent inferring your interface and calling it. Registered tools turn a page into a small, documented API that the visitor's own assistant can operate — without you hosting a server or handing out keys.

If it failsRegister your two or three most valuable actions as tools with document.modelContext.registerTool. Start read-only. The polyfill lets you ship before the browser support is universal.

Full article: What WebMCP is, and what it is notWebMCP specification (W3C WebML CG)Explainer repositoryChrome for Developers — WebMCP

Are existing forms annotated so the browser can synthesise tools from them?

The declarative form is the cheapest entry point into the standard: the browser derives the schema from fields you already have, so a contact or search form becomes callable without new JavaScript.

If it failsAnnotate the forms you already ship. Because the attribute names are still marked TODO in the explainer, verify them against the current specification before rolling this out widely.

Conditional — excluded when it does not apply

Full article: The declarative API: tools from forms you already haveExplainer — declarative API

Are the tools named legally and described well enough to choose between?

The description is the entire interface. An agent picks a tool by reading it, so a name outside the permitted character set fails registration and a four-word description produces confident calls to the wrong function.

If it failsKeep names inside 1–128 characters of A–Z, a–z, 0–9, underscore, hyphen and dot. Write descriptions that say what the tool returns and when to prefer it. Stay well under fifty tools per page.

Does each tool declare a typed JSON Schema with described properties?

Without a schema an agent sends whatever it invented. With one it sends a validated object — and a required field with a one-line description is the difference between a working call and three retries.

If it failsGive every tool an inputSchema with typed properties, a description per property, and an explicit required list.

Are read-only tools marked read-only, and is anything destructive gated behind confirmation?

Annotations are how an agent decides what it may do without asking. Marking your read paths readOnlyHint is what makes them safe to call freely — and the specification's own threat model is the reason anything that writes should pause for a human.

If it failsSet readOnlyHint on every tool that only reads. Set untrustedContentHint where a tool returns text written by other users. Put a confirmation step in front of anything that spends money or changes state.

Full article: The WebMCP security model, and the parts that are still openSpecification — tool annotations

Can another system find your capabilities without a human reading your docs?

WebMCP tools exist only while a page is open. A discovery document is how an agent learns what you offer before it visits — and how your capabilities show up in a directory rather than being found by accident.

If it failsPublish a server card at /.well-known/mcp.json, or an OpenAPI document, and make sure robots.txt does not disallow the path it points at.

Full article: Discovery documents, and the robots.txt trapModel Context ProtocolRFC 9727 — API Catalog

Is there a documented way for an agent to authenticate?

Inside the page the browser session already carries the login, which is WebMCP's quiet advantage. Everything outside the page still needs a stated path — otherwise integrators guess, and guessing means credentials in places you did not intend.

If it failsPublish OAuth discovery metadata, or an auth.md describing how machine clients obtain access.

Conditional — excluded when it does not apply

RFC 8414 — OAuth authorization server metadataRFC 9728 — OAuth protected resource metadata

Can an agent complete a purchase without a human at the keyboard?

A shop that an agent can browse but not pay stops one step short of the point. The rails are young and picking one is a real decision, but the shops that ship first will be the ones agents can actually buy from.

If it failsEvaluate x402, the Agentic Commerce Protocol and the Universal Commerce Protocol against your payment provider, then publish the discovery document for whichever you adopt.

Conditional — excluded when it does not apply

Can you tell a legitimate agent from an impostor, and have you stated your terms?

Once agents transact, user-agent strings stop being good enough — anyone can claim to be one. Signed requests let you welcome the real ones, and content signals let you state what your text may be used for without blocking it outright.

If it failsPublish a Web Bot Auth signature directory, and express usage terms with content signals in robots.txt rather than a blanket disallow.

Cloudflare — Content Signals Policy