Anthropic’s native web_fetch tool in the Claude API and Claude Code has made lightweight research a lot easier but many keep treating it like a full browser automation tool and wondering why it fails. I spent few days stress-testing native web_fetch across about 60 different URLs to see where its actual limits are. Here is what works and where it consistently chokes: I) Where native fetch works great: • Static HTML docs, standard blogs and changelogs where you already have the direct URL. • Raw PDF extraction (it parses PDFs into text smoothly without needing PyPDF). • Lightweight lookups where the site doesn’t run aggressive bot mitigation. II) No JavaScript execution (the biggest blind spot) Claude’s native tool only pulls the initial HTTP response so if a site is built on React, Vue, or Next.js client hydration, Claude literally sees an empty <div id=“root”></div> shell. You ask it to summarize a dashboard or modern docs page and it hallucinates or says the page is empty. III) No dynamic URL generation Claude cannot dynamically guess or construct a new URL and fetch it on the fly but it can only request URLs already present in the prompt or conversation history. V) Zero interaction handling If a site requires closing a newsletter modal, clicking a “Show More” tab, expanding accordion menus, or authenticating, native fetch is dead in the water. V) How to handle dynamic sites If you’re building an agent that needs to touch modern web apps, you basically have to route the request through a headless engine and in my case, I wrap those requests in firecrawl instead: it executes the JS, handles anti-bot challenges and returns clean markdown. Its /interact endpoint also lets you keep a live browser session open if the agent needs to click or scroll. here’s more detailed report about it: https://www.firecrawl.dev/blog/claude-web-fetch-vs-firecrawl Native web_fetch is fine for quick static docs and PDFs but if your agent workflow relies on scraping modern SPAs or dynamic pages, relying solely on native fetch is going to lead to constant empty responses. submitted by /u/Winter-Manager996
Originally posted by u/Winter-Manager996 on r/ClaudeCode