To render, or not to render

Reading raw HTML is cheap. Running a full browser to see what JavaScript builds is not — and that price decides how much of the modern web a crawler can actually see.

A non-rendering crawler reads the HTML the server sent and stops. A rendering one runs a headless browser, executes the scripts, waits for the network to settle, and reads the resulting DOM — seeing content that only exists after JavaScript runs, at many times the cost per page.

Most large crawlers do both: a fast HTML pass for everything, a slower render pass for a chosen subset. If your content only appears after rendering, you are betting on making that subset — a bet many pages quietly lose. It also spends more crawl budget.