BRIEFING

Machines Reading Machines, Issue 2026-W39: What Do the Crawlers See When They Arrive?

Published 2026-09-27 by Marketingz, Laval.

A weekly look at the part of your website that no customer ever sees: the visitors that are not people. This issue is about what a crawler actually receives when it requests one of your pages, how much of your traffic is crawlers at all, and three changes from the last few weeks that alter who gets in. You will not need a terminal to follow it. Every figure links to a dated public source. Where the newest evidence is older than we would like, or comes from a company with something to sell, that is said next to the number rather than left out. The thread through all of it: machines see what your server sends and what your settings allow, and neither is visible from a browser.

The short version

• The major AI crawlers studied did not run JavaScript. They read the HTML your server sends and nothing that loads afterwards (Vercel, with MERJ, 17 December 2024).
• 53 percent of web traffic in 2025 was automated, and 40 percent was bots classed as malicious (Thales 2026 Bad Bot Report, via Help Net Security, 30 April 2026).
• Since 15 September, new Cloudflare sites that carry ads disallow AI training by default and block AI agents on their ad pages, while Googlebot, Bingbot and Applebot stay allowed for search (Cloudflare, 15 September 2026).
• Schema.org released version 30.1 on 16 September, adding product-passport and environmental-declaration types; Google documented a new VideoObject property on 24 September (schema.org releases; Google Search Central updates).
• The newest crawl-to-referral ratios we could trace to the original publisher are from July 2025. Newer figures are circulating; we could not confirm them.

What does a crawler actually see when it lands on your page?

Only what your server sends in its first reply. The main AI crawlers studied fetch the page's HTML and do not run the JavaScript that builds many modern pages, so anything that appears only after scripts load is invisible to them.

The clearest public measurement is still a study Vercel, a website hosting company, published with the search consultancy MERJ on 17 December 2024. Across a month of traffic on Vercel's network, OpenAI's GPTBot made 569 million requests and Anthropic's Claude crawler 370 million, together about 20 percent of Googlebot's 4.5 billion. GPTBot downloaded JavaScript files in 11.50 percent of its requests and Claude in 23.84 percent, but the authors found that "none of the major AI crawlers currently render JavaScript" (Vercel, with MERJ, 17 December 2024). Google is the exception: Googlebot renders pages before indexing them (Google Search Central, JavaScript SEO basics, updated 4 March 2026). In plain terms, if your prices, opening hours, service list or reviews are loaded by a script after the page opens, a person sees them, Google probably sees them, and most AI assistants' crawlers do not.

The same study found that 34.82 percent of GPTBot's fetches and 34.16 percent of Claude's landed on "page not found" errors, against 8.22 percent for Googlebot. That points to the AI crawlers requesting old or guessed addresses far more often, which is one reason to keep redirects in place for retired pages.

What is uncertain: this is one hosting network's traffic, measured almost two years ago. We found no newer study of comparable size from a primary source. Plenty of 2026 articles repeat the finding, but they cite this study rather than new measurement. It is the best public evidence available, and it is getting old.

How much of the traffic hitting your website is even human?

Less than half, across the large networks that measure it. Thales found 53 percent of 2025 web traffic was automated, and 40 percent was bots it classed as malicious. Cloudflare said in July 2026 that most internet traffic is now non-human.

The Thales 2026 Bad Bot Report, published 30 April 2026, puts automated traffic at 53 percent of what its network observed in 2025, with bad bots at 40 percent and benign automation at 13 percent, which leaves people at 47 percent. It also reports that AI-driven bot activity grew roughly 12.5 times over the year, and that about 9 percent of AI crawler sessions triggered its bad-bot detection (Help Net Security, 30 April 2026). Cloudflare's one-year report, published 1 July 2026, says "more than 50% of traffic on the Internet is now non-human," and that as of June 2026, 52 percent of crawler requests on its network were for AI training, up from 22 percent in spring 2025, and mixed-use crawlers doing several jobs at once accounted for over 36 percent of activity (Cloudflare, 1 July 2026).

For a business owner, the practical meaning is about your analytics. Visit counts, bounce rates and "time on page" are partly measuring machines unless your analytics tool filters them, and filtering is imperfect. Both sources sell bot protection, which is worth knowing when reading their totals. Your own site's mix could be higher or lower than either network's average; these figures describe the networks, not you.

Do AI crawlers send any customers back?

Very few, on the most recent figures we could verify. In July 2025, Cloudflare counted about 38,000 pages crawled by Anthropic for every visitor it referred, about 1,100 for OpenAI and about 195 for Perplexity.

Cloudflare calls this the crawl-to-refer ratio: pages a platform's bots request, divided by the visitors that platform sends back. In a post published 29 August 2025 it reported July 2025 ratios of about 38,066 to 1 for Anthropic (down from 286,930 to 1 in January), 1,091 to 1 for OpenAI, about 195 to 1 for Perplexity (up from 54 to 1) and 40.7 to 1 for Microsoft. Over the preceding twelve months, 80 percent of AI crawling was for training, 18 percent for search and 2 percent for acting on a user's request (Cloudflare, 29 August 2025). Cloudflare has also cautioned that apps often do not pass along where a visitor came from, which can make these ratios look worse than the real referral flow, by an amount Cloudflare says it cannot estimate (Cloudflare, 1 July 2025).

What is uncertain: these figures are fourteen months old. Several 2026 articles quote newer ratios, some very different from these, but we could not trace them back to a dated Cloudflare publication, so they are not repeated here. Treat the direction as solid and the exact ratio as a snapshot.

What changed at Cloudflare on 15 September, and could it hide you from Google?

For new sites on Cloudflare that show ads, the defaults now publish a "Disallow AI Training" signal for the whole domain and block AI agents on the pages that carry ads; search stays allowed. Googlebot, Bingbot and Applebot remain allowed for search, so the default should not remove you from search results.

If your site sits behind Cloudflare, its defaults decide what your site allows without you ever looking. From 15 September 2026, it sorts AI traffic into three jobs: Search, Agent (a bot acting on a person's behalf right now) and Training. New domains that carry ads get "Disallow AI Training" and have agents blocked on ad pages; new domains without ads allow both. Existing customers' settings carry over (Cloudflare, 15 September 2026). The reasoning Cloudflare gives is that an ad signals the owner wants a person to land on that page (Cloudflare, 1 July 2026).

The awkward case is crawlers that do several jobs at once. Cloudflare's July announcement said multi-purpose crawlers such as Googlebot, Applebot and Bingbot "will be blocked by customers who have selected to block Training" (Cloudflare, 1 July 2026). The 15 September post resolves it with a new "Accountable" category: operators who offer a training opt-out, a promise that opting out will not affect search, page-level reporting, and a way to opt out of AI summaries (Cloudflare says it aims to offer its own control over this early in 2027) are allowed for search even when training is disallowed. Cloudflare names Googlebot, Bingbot and Applebot as Accountable, and also lists crawlers from Amazon, Anthropic, Meta and OpenAI, which run search and training as separate bots (Cloudflare, 15 September 2026). The same post quotes Google's commitment that disallowing its separate Google-Extended token "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."

What is uncertain: the sources disagree on who got the new defaults. TechCrunch reported on 1 July that they would also apply to all existing free-plan customers (TechCrunch, 1 July 2026). Cloudflare's own 1 July post says the defaults apply to "all new domains onboarding to Cloudflare" (Cloudflare, 1 July 2026), and its 15 September post describes presets for new domains and migration of existing settings. If your site uses Cloudflare, the safe move is to ask whoever manages it which of the three settings is active, rather than to assume. A local business site with no ads that joins Cloudflare now gets Search, Training and Agent all allowed by default; the blocking defaults apply only to sites that carry ads.

Are the big sites blocking AI crawlers, and which ones?

Yes, and increasingly by default. Reuters and Time switched to blocking all bots except an approved list in May 2026.

A robots.txt file is a short public note on a website telling crawlers what they may read. It is a request, not a lock. Search Engine Journal reported on 9 June 2026 that Reuters and Time adopted "default-deny" lists in May, letting in only approved bots, joining People Inc. and The Atlantic. People Inc. said its list of blocked user agents went from about 2,100 to more than 30,000 after the switch. The same report cites a BuzzStream analysis finding 79 percent of top news publishers block at least one AI training bot, and a TollBit finding that 30 percent of AI bot scrapes in the second half of 2025 ignored explicit robots.txt instructions (Search Engine Journal, 9 June 2026).

What is uncertain: because robots.txt is voluntary, a block records what a site asked for, not what crawlers did.

What changed in structured data this month?

Two things, both small and both about products and video. Schema.org released version 30.1 on 16 September with new product-compliance types, and Google documented a new creator property for video markup on 24 September.

Structured data is a labelled summary of a page, hidden in its code, that tells machines "this is a product, this is its price, this is the business." Schema.org, the shared vocabulary that search engines read, released version 30.1 on 16 September 2026. It adds types for a Digital Product Passport, an Environmental Product Declaration and a Declaration of Conformity, product properties such as recycled-content percentage, importer and authorized representative, a "maximum retail price" price type, and a minimum order value for shipping rates (schema.org release listing). These track European product-compliance rules more than marketing; they matter first to manufacturers and importers selling into the EU. For a Quebec business, that means the 30.1 product-compliance types matter only if you export goods into the EU; a local service business can ignore them.

On Google's side, the Search Central changelog for the period shows a creator property added to VideoObject markup on 24 September, and an 8 September page on how Search features vary by country (Google Search Central, documentation updates). Google's crawler changelog shows one September entry: on 17 September it clarified that its Mediapartners-Google crawler serves several ad products, not only AdSense (Google crawling changelog).

The larger shift happened earlier this year. Google stopped showing FAQ rich results on 7 May 2026 and removed their documentation on 15 June (Google Search Central, documentation updates). If a supplier is still selling FAQ markup as a way to win extra space in Google results, that result no longer exists.

What is uncertain: many 2026 articles claim that AI assistants use structured data to judge whether a page is trustworthy. We found no primary statement from Google, OpenAI or Anthropic that confirms how, or whether, their answer systems weigh it. The claim is plausible and unproven.

What breaks a page for a machine?

Three things, on the evidence above: content that only appears after JavaScript runs, broken or retired addresses, and a blocking setting you did not choose. A fourth is money rather than breakage: markup for search features that no longer exist.

Each one is invisible from a browser, which is why they persist. A page can look perfect to you while serving an AI crawler a blank template (the JavaScript finding above), sending it to an error page because it asked for an old address (the 404 rates above, where about a third of AI-crawler requests on one hosting network hit missing pages), or turning it away at a default you never set (the Cloudflare change above). None of these needs a developer to diagnose; each is a question you can put to whoever runs your site: is the important text in the page the server sends, do old addresses redirect, and which AI-crawler setting is active? A separate cost check: are you paying for markup aimed at a feature Google has retired?

For an owner in Laval or elsewhere in Quebec, the first question is usually the one to start with. If your site was built with a page builder, or shows its hours, prices or menu through a booking or ordering widget, ask whether that text is in the page the server sends or is loaded by a script afterwards. If it is loaded afterwards, the JavaScript finding above suggests an AI assistant asked about your business may not see it at all.

What should you watch for next?

Fresher crawl-to-referral figures, who qualifies under Cloudflare's new rules, any new study of how AI crawlers handle JavaScript, and whether AI companies say anything first-hand about structured data.

• Whether Cloudflare publishes crawl-to-refer ratios newer than July 2025 in a dated post, so the figures quoted around the industry can be checked.
• Whether Cloudflare adds operators beyond the seven it has named as Accountable, and whether any mixed-use crawler is refused the label.
• Whether anyone publishes a JavaScript-rendering study newer than December 2024, and whether any AI crawler has started rendering pages the way Googlebot does.
• The schema.org release after 30.1 and Google's Search Central changelog, for whether Google adds rich-result support for any of the new product types.
• Any first-party statement from an AI assistant's maker on whether structured data affects which pages it cites.

Questions, or a correction: marketingz@campaigncrush.org or (438) 813-2465.