Cloudflare Browser Run tells you why a crawled URL actually failed
The /crawl endpoint now returns a per-URL outcome and a robots.txt diagnostics array, replacing a bare success-or-failure result with an actual reason.
Cloudflare’s Browser Run added diagnostics to the /crawl endpoint’s results on 3 November. Each URL record can now carry an outcome, such as success, robots_disallowed, timeout or page_error, instead of just a status.
A new job-level robots array also reports the robots.txt state for up to 50 origins the crawl touched, with the origin, whether usable rules were found, and the HTTP status Cloudflare got back when fetching it. A sample response in the changelog shows the shape:
{
"robots": [{ "origin": "https://example.com", "state": "available", "status": 200 }],
"records": [{ "url": "https://example.com/docs/", "status": "completed", "outcome": "success" }]
}Why it matters: if you’re running crawls through Browser Run, “it didn’t work” used to mean opening a browser and guessing. Now you can tell, programmatically, whether a URL was blocked by robots.txt, timed out, or errored on the page itself, and build retry or alerting logic around the actual cause instead of a blanket failure.