Empirical benchmark · live corpus
State of Agent Readiness: What Public Websites Actually Expose to AI Agents
Most claims about the “agentic web” conflate early-adopter demos with the everyday internet, or grade sites by anecdote. Every figure on this page is computed live from our public scan archive on rule set 2026-08-31 — strictly separating an uncurated Tranco top-web sample from declared WebMCP adopters and user-requested scans.
69,187 comparable scans · 802 distinct third-party domains · 2026-08-31 to 2026-11-22 · 107 operator-owned domains excluded
Six citable findings
Each finding states its exact cohort, numerator, and denominator alongside public reports you can verify yourself.
93.5% of mainstream top sites (158 of 169) score Grade D or F, with a median Agent Readiness Score of 36/100.
Across the uncurated Tranco HTML sample, zero sites earn Grade A (0/169) and only 2 (1.2%) earn Grade B (mean score 36.8/100). While many pages are partially readable, reliable task completion and machine protocols remain rare on the general web.
llms.txt adoption (24.3%) outpaces callable MCP endpoints (2.4%) and browser WebMCP tools (1.2%) roughly 10-to-1.
41 of 169 reachable Tranco sites now ship a valid structured /llms.txt (including www.cloudflare.com, www.forter.com, amplitude.com, sentry.io, wordpress.org). Yet only 5 (3%) answer Accept: text/markdown, 4 (2.4%) expose an unblocked MCP discovery card, and 2 (1.2%) register browser WebMCP tools (www.cloudflare.com, www.youtube.com). Websites are summarizing themselves for LLMs far faster than they are letting agents act.
Nearly half of top homepages (49.1%) lose content without JavaScript, and only 5.8% of HTML forms are fully machine-labelled.
33 of 169 Tranco homepages (19.5%) arrive effectively empty without script execution, and another 50 (29.6%) arrive incomplete (why raw HTML matters). Among the 69 homepages serving HTML input forms, only 4 (5.8%) pass form semantics (labels, names, types, and autocomplete), and 1 (1.4%) expose machine-readable validation state.
Even among declared WebMCP early adopters (n = 370), only 41.1% cleanly pass on current document.modelContext registration.
60 listed adopter domains (16.2%) score partial — either still calling deprecated navigator.modelContext (deprecated in Chrome 150) or loading a polyfill without an observable registration. Another 158 (42.7%) expose no registration in raw HTML or up to ten same-origin bundles (gated behind auth, lazy chunks, or removed), and 238 (64.3%) omit readOnlyHint safety annotations.
22.3% of live-probed sites refuse real agent user agents at the HTTP/WAF layer — while early adopters accidentally block their own manifests in robots.txt.
In the Tranco baseline, only 8.9% (15/169) explicitly allow all twelve named AI crawlers in robots.txt. Across 202 organic user-requested scans where live agent user-agent requests ran, 45 (22.3%) blocked at least one agent at the HTTP/WAF layer (26 refused ≥3 of 4 agents). And in the adopter cohort, 17 sites published an MCP card and 20 published an API description only to disallow it in robots.txt.
Repeat measurements (30+ days apart) already show measurable churn among early WebMCP deployments.
Among 132 WebMCP adopter hosts measured at least 30 days apart on the same rule set, 12 sites that previously passed modelcontext-present no longer pass on their latest scan (while 9 newly passed). Over the same span across 154 Tranco hosts, 96 scores did not move at all while 6 sites added llms.txt.
Benchmark your own URL
Where does your site sit between the mainstream median (36/100) and early adopters (63/100)?
Run the same 25-check evaluation against any public URL in ~20 seconds — no account required.
Free instant score in ~20s. No account, no email required. Works on any public homepage, product, or form URL.
Three cohorts, separated on purpose
Pooling all 802 third-party domains into a single headline average would make llms.txt look 50% adopted across the internet — an artefact of including a directory of WebMCP early adopters. Separating the three sampling frames shows the real gap between the mainstream web and early protocol adopters.
| Metric | Mainstream web (Tranco HTML) | Declared WebMCP adopters | Organic user scans |
|---|---|---|---|
| Distinct hosts (n) | 169 | 370 | 260 |
| Median score | 36 / 100 | 63 / 100 | 44 / 100 |
| Mean score (min–max) | 36.8 (7–87) | 63.3 (10–97) | 44.8 (9–100) |
| Grade A (90–100) | 0 (0%) | 24 (6.5%) | 8 (3.1%) |
| Grade A or B (≥75, ready / usable) | 2 (1.2%) | 103 (27.8%) | 23 (8.8%) |
| Grade D or F (<55, fails key paths) | 158 (93.5%) | 108 (29.2%) | 202 (77.7%) |
| Fitness axis (mean earned %) | 52.9% | 75.7% | 61% |
| Adoption axis (mean earned %) | 4.2% | 39.5% | 10.3% |
| Hosts with extracted WebMCP tool names | 0 | 50 | 9 |
All 25 checks: pass, partial, and failure rates
Conditional checks (such as form semantics on pages with no form, or payment rails on non-commerce sites) are excluded from that check's denominator n rather than counted as failures. Every check links to its exact rule in the scoring methodology.
| Check | Pillar | Tranco pass | Tranco partial | Tranco fail | Adopters pass |
|---|---|---|---|---|---|
robots.txt lets AI crawlers inrobots-ai-crawlers · 5 pts · n=169 | Agent Access | 8.9% 15/169 | 76.9% 130/169 | 14.2% 24/169 | 33.8% 125/370 |
Agent user agents are actually servedagent-user-agents-served · 5 pts · n=10 | Agent Access | 90% 9/10 | 10% 1/10 | 0% 0/10 | 93.3% 125/134 |
No wall in front of the contentno-blocking-interstitial · 4 pts · n=169 | Agent Access | 91.7% 155/169 | 5.3% 9/169 | 3% 5/169 | 89.2% 330/370 |
Content survives without JavaScriptcontent-without-javascript · 6 pts · n=169 | Agent Access | 50.9% 86/169 | 29.6% 50/169 | 19.5% 33/169 | 85.1% 315/370 |
Semantic landmarks and heading ordersemantic-structure · 4 pts · n=169 | Machine Readability | 25.4% 43/169 | 32.5% 55/169 | 42% 71/169 | 62.2% 230/370 |
Valid structured datastructured-data · 6 pts · n=169 | Machine Readability | 40.2% 68/169 | 4.1% 7/169 | 55.6% 94/169 | 82.2% 304/370 |
Markdown content negotiationmarkdown-negotiation · 4 pts · n=169 | Machine Readability | 3% 5/169 | 0% 0/169 | 97% 164/169 | 48.4% 179/370 |
llms.txt source filellms-txt · 3 pts · n=169 | Machine Readability | 24.3% 41/169 | 3.6% 6/169 | 72.2% 122/169 | 76.5% 283/370 |
Sitemap and canonical URLsitemap-and-canonical · 3 pts · n=169 | Machine Readability | 37.3% 63/169 | 45% 76/169 | 17.8% 30/169 | 79.2% 293/370 |
A primary action is discoverableprimary-action-present · 5 pts · n=169 | Actionability | 49.1% 83/169 | 7.7% 13/169 | 43.2% 73/169 | 47.6% 176/370 |
Forms an agent can fillform-quality · 7 pts · n=69 | Actionability | 5.8% 4/69 | 81.2% 56/69 | 13% 9/69 | 19.6% 31/158 |
Controls have accessible namesnamed-controls · 5 pts · n=128 | Actionability | 50% 64/128 | 35.9% 46/128 | 14.1% 18/128 | 79.7% 255/320 |
No bot challenge on the conversion pathno-captcha-on-conversion · 5 pts · n=169 | Actionability | 92.3% 156/169 | 3.6% 6/169 | 4.1% 7/169 | 89.2% 330/370 |
Errors an agent can readmachine-readable-errors · 3 pts · n=69 | Actionability | 1.4% 1/69 | 29% 20/69 | 69.6% 48/69 | 5.1% 8/158 |
document.modelContext is in usemodelcontext-present · 6 pts · n=169 | WebMCP Implementation | 1.2% 2/169 | 0% 0/169 | 98.8% 167/169 | 41.1% 152/370 |
Declarative tools on formsdeclarative-forms · 4 pts · n=74 | WebMCP Implementation | 0% 0/74 | 0% 0/74 | 100% 74/74 | 30.5% 51/167 |
Tool names and descriptions+5 bonustool-hygiene · +5 pts · n=169 | WebMCP Implementation | 1.2% 2/169 | 0% 0/169 | 98.8% 167/169 | 56.8% 210/370 |
Typed input schemas+5 bonusinput-schema-typed · +5 pts · n=169 | WebMCP Implementation | 1.2% 2/169 | 0% 0/169 | 98.8% 167/169 | 53.2% 197/370 |
Honest annotations on tools+5 bonustool-annotations · +5 pts · n=169 | WebMCP Implementation | 0.6% 1/169 | 0.6% 1/169 | 98.8% 167/169 | 34.6% 128/370 |
An MCP endpoint an agent can reachmcp-endpoint · 7 pts · n=169 | Integration | 2.4% 4/169 | 1.2% 2/169 | 96.4% 163/169 | 32.7% 121/370 |
A machine-readable description of your APIapi-description · 5 pts · n=169 | Integration | 5.3% 9/169 | 0% 0/169 | 94.7% 160/169 | 39.5% 146/370 |
An agent card other agents can readagent-card · 3 pts · n=169 | Integration | 3% 5/169 | 0% 0/169 | 97% 164/169 | 27.8% 103/370 |
Agent-facing authenticationagent-auth · 4 pts · n=11 | Identity & Commerce | 27.3% 3/11 | 9.1% 1/11 | 63.6% 7/11 | 47.9% 91/190 |
Agentic payment railsagentic-payments · 3 pts · n=25 | Identity & Commerce | 4% 1/25 | 0% 0/25 | 96% 24/25 | 25.1% 49/195 |
Bot identity and content signalsbot-identity · 3 pts · n=169 | Identity & Commerce | 4.1% 7/169 | 24.9% 42/169 | 71% 120/169 | 40.3% 149/370 |
Methodology, sampling frames, and limitations
A benchmark is only as trustworthy as what it refuses to conflate. Four constraints govern how every number on this page is computed:
- Stratified sampling frames: The mainstream baseline is drawn in rank order from Tranco list
74V4X(first 250 domains whose homepage serves HTML over HTTPS; 169 permitted our identified crawler viarobots.txt). The adopter cohort comes fromwebmcp.com's publictype=livedirectory with Tranco overlaps removed (n = 370). Organic scans (n = 260) are third-party URLs submitted by visitors. - Conflict-of-interest exclusion:
webmcp-tool.comand 106 other operator-run domains are filtered out before any cohort statistic is computed. - No user-agent spoofing on uninvited crawls: Our background corpus collector identifies itself as
webmcp-tool.com/1.0, obeysrobots.txt, and never impersonatesGPTBotorClaudeBotagainst sites nobody asked us to test. Consequently,agent-user-agents-servedis excluded on background rows and reported strictly from 202 interactive third-party scans. - Single-URL, unauthenticated boundary: Each crawl inspects the target URL's raw HTTP response,
robots.txt,sitemap.xml,llms.txt, standard/.well-known/*discovery paths, and up to ten same-origin JS bundles. WebMCP tools registered only behind authentication or lazy post-login routes will not appear in an unauthenticated public scan. See how Cloudflare and Ora/Vercel grade the same corpus and browse individual implementations in the WebMCP directory.
How to cite this benchmark
You are welcome to quote any statistic on this page with attribution and a link to this URL. Machine clients requesting this same URL with Accept: text/markdown receive the live data table in clean Markdown.
Wübbe, G. (2026). State of Agent Readiness: Empirical Benchmark of the Public Web (802 domains, ruleset 2026-08-31). WebMCP Tool. https://webmcp-tool.com/researchNeed hands-on help fixing Agent Readiness or shipping WebMCP?
We work with engineering and agency teams to remediate raw-HTML and WAF blockers, expose declarative and imperative WebMCP tools, and ship remote MCP endpoints.