Since its emergence in the 1990s, JavaScript has become a pillar of the modern web, enabling dynamic and interactive web pages. This evolution has led to the rise of JavaScript engines, browsers, and runtimes that allow the language to be executed outside the browser.
However, the omnipresence of JavaScript presents a challenge for SEO and, in particular, indexing robots (crawlers). Search engines must be able to properly analyze and index websites whose page content depends on script execution. Yet, JavaScript rendering is resource-intensive and not always fully supported by crawlers. When crawling a site, it is crucial to follow all dynamically generated links/URLs from JS code. Some sites load content only after scripts have run, thus changing the URLs accessible to bots.
In this article, we’ll explore the evolution of JavaScript engines, their role in SEO, and available solutions to efficiently retrieve a post-execution JavaScript DOM.
History of Browsers and JavaScript Engines
The web has evolved rapidly since the 1990s, along with browsers and their JavaScript engines. Here’s a brief overview of this evolution:
- 1994 : Netscape Navigator (Mosaic Netscape) and the Birth of JavaScript and SpiderMonkey | At the dawn of early browsers, web pages still severely lacked dynamism and interactivity. For Netscape, it was crucial to maintain its dominant position (nearly 80% market share) against a Microsoft eager to conquer the web. In 1995, Brendan Eich created Mocha for Netscape, which was later renamed LiveScript and finally JavaScript. This was also the birth of the SpiderMonkey engine, originally written in C and later rewritten in C/C++. It’s still used today by Firefox.
- 1996 : JScript (Microsoft) | Microsoft countered with JScript, an alternative version of JavaScript for Internet Explorer 3, resulting in standard fragmentation. This rivalry significantly impacted how websites were developed and rendered. By 2000, Microsoft was set to hold over 90% of the market.
- 2000 : Konqueror and the Introduction of KHTML and KJS (KDE) | The Konqueror browser, developed by KDE, introduced KHTML and KJS, which led to the creation of WebKitand JavaScriptCore, still in use today.
- 2000 : Chakra (old) (Microsoft) | JavaScript engine used from Internet Explorer 9 to 11, different from Edge’s engine.
- 2002 : Safari and JavaScriptCore (Apple) | Apple launched Safari based on a fork of KHTML called WebKit, using JavaScriptCore derived from KJS. Before Safari, MacOS included a specific version of Internet Explorer, which Microsoft gradually abandoned until its discontinuation in 2003.
- 2008 : Chrome and the V8 Era | Google revolutionized the web with V8, an engine optimized for speed and Just-In-Time (JIT) compilation. Chrome was initially based on WebKit, itself from KHTML, but evolved by creating its own fork: Blink. Thanks to its speed, V8 allowed Chrome to execute JavaScript more efficiently and better support crawler indexing.
- 2008 : SquirrelFish (SquirrelFish Extreme aka Nitro) (Apple) | An evolution of JavaScriptCore, optimized for Safari.
- 2010+ : The Rise of JavaScript Engines | Microsoft developed a new Chakra iteration for Edge, while Mozilla continued enhancing SpiderMonkey.
The optimization of JavaScript engines led to the rise of runtimes like Node.js, bringing JavaScript into the backend world.
The Rise of JavaScript Runtimes
Early JavaScript runtimes primarily allowed scripting within more complex applications (the same use browsers originally had), but weren’t really intended for other purposes. As engines improved— especially with JIT in V8 — JavaScript became a full-fledged language with capabilities beyond basic scripting. Already the most used language in web development, it has a particularly large user community. Being integrated into all major browsers by default, it’s far easier to use than alternatives like Flash, Silverlight, or Java applets, all deprecated in the early 2020s in favor of HTML5, JavaScript, or derivatives like TypeScript. Today, these are the only client-side languages capable of dynamically manipulating the DOM of a web page in a browser.
Here’s a quick overview of major JavaScript engines developed over the years:
- 1998 : Rhino (Mozilla) | A JavaScript implementation in Java developed by Mozilla.
- 2009 : Node.js (Ryan Dahl) | Based on V8, it transformed backend development by making JavaScript server-side competitive.
- 2013 : Nashorn (Oracle, Java 8) | JS engine written in Java, integrated into JDK 8 as Rhino’s successor (discontinued in 2021).
- 2014 : GraalJS (formerly GraalVM JavaScript) (Oracle, GraalVM) | A modern JavaScript engine in GraalVM, offering better performance and ECMAScript compatibility.
- 2017 : ChakraCore (Microsoft) | Open-source version of Chakra, used outside browsers (discontinued in 2021).
- 2018 : Hermes (Facebook / Meta) | Lightweight JS engine optimized for React Native.
- 2020 : Deno (Ryan Dahl) | Created by Node.js’s original author, developed in Rust for more security and better TypeScript support.
- 2020 : QuickJS (Fabrice Bellard) | An ultralight C-based engine supporting ES2020, suitable for running JS on embedded systems.
- 2022 : Bun (Jarred Sumner) | A fast runtime developed in Zig using JavaScriptCore, optimized for performance.
These runtimes allow JavaScript code interpretation in various contexts, from backend to embedded applications.
SEO and the Need to Execute JavaScript Outside the Browser
JavaScript plays a crucial role in web page rendering but poses SEO challenges. Although Googlebot, Google’s crawler, can now execute JavaScript using a dedicated Chromium version, the process is inherently slower (up to 10x) and much more resource-intensive. Additionally, rendering time depends more on content (scripts, resources, etc.) than on a basic HTTP request. Crawlers must then justify this effort, treating such a site as “trusted.” Websites heavily reliant on JavaScript or too slow/resource-consuming will likely be penalized. Due to this complexity, many crawlers choose not to—or only partially—execute JavaScript on indexed pages, impacting their search visibility.
Solutions like SSR (Server-Side Rendering) or pre-rendering can help mitigate this issue. Allowing a bot to index a site also contributes to the user’s experience. It’s entirely possible to generate a pre-rendered version of a site only during crawling, as long as the result matches what a user would get after dynamic client-side execution.
Solutions to Retrieve a Post-Execution JavaScript DOM
There are several ways to execute JS from a crawler’s perspective, from the most resource-intensive to the least precise:
1. Traditional Headless Browsers (Chrome, Firefox, WebKit)
Tools like Puppeteer (Chrome) and Playwright (Chrome, Firefox, WebKit) allow JavaScript execution through a dedicated interface using a real browser, offering accurate DOM rendering for indexing bots. Originally designed for UI testing (e.g., via WebDriver), browsers also offer interfaces like Chrome DevTools Protocol (CDP) for controlling and tracing a webpage’s rendering. Although practical, these tools are not always suitable for crawling.
Pros:
- Rendering identical to a standard browser.
- Supports all JS and Web APIs.
- Ideal for SEO and interface testing.
Cons:
- Very resource-intensive.
- Hard to scale across thousands of pages.
2. “Light” Headless Browsers (Splash, HtmlUnit, Lightpanda)
Lighter solutions like HtmlUnit, Splash, and Lightpanda offer a balance between performance and rendering accuracy.
Pros:
- Less resource-intensive than full browsers.
- Can execute JavaScript faster.
- Better suited for scalable scraping and SEO.
Cons:
- Partial JS support: complex interactions may fail.
- Might not support all advanced Web APIs.
3. Dedicated JS Runtimes (JSDOM, Happy-DOM, Linkedom)
For lighter needs, DOM simulators like JSDOM, Happy-DOM, and Linkedom let you manipulate HTML pages without a browser.
Pros:
- Very fast and lightweight.
- Ideal for extracting HTML content and analyzing pages without complex JS.
- Perfect for technical SEO and HTML audits.
Cons:
- Doesn’t support all Web features (e.g., WebGL, Canvas, advanced interactions).
- Not a full browser replacement for interactive rendering.
Summary and Recommendations
| Solution | Rendering Accuracy | Resource Consumption | Recommended Use |
|---|---|---|---|
| Full Headless (Chrome, Firefox) | ✅✅✅ | 🔴🔴🔴 | Advanced testing, accurate SEO |
| Light Headless (Splash, HtmlUnit, Lightpanda) | ✅✅ | 🟡🟡 | Scraping, scalable crawling |
| JS Runtimes (JSDOM, Happy-DOM) | ✅ | 🟢🟢🟢 | Fast data extraction |
Conclusion
Choosing a solution to retrieve a post-execution JavaScript DOM shouldn’t be random. The best approach is to first analyze the technologies used on a page to understand its actual rendering needs.
By identifying in advance the JavaScript libraries and frameworks (React, Vue, Angular, etc.), as well as prerequisites for their execution, one can choose the most suitable rendering engine. This strategy helps optimize resource consumption while ensuring a final DOM that reflects the true page content.
Rather than adopting a costly, one-size-fits-all method, it’s more efficient to contextualize each analysis based on the page’s specifics—just like an optimized workflow where each task is handled with the exact tools and resources needed. A crawler could then determine in advance whether a simple HTML analysis is sufficient, whether it needs light JavaScript execution via a runtime like JSDOM, or if a full headless browser is necessary for reliable rendering. In an ideal world, a web page could declare its own prerequisites or list used interfaces to help target the most appropriate processing method.
By adopting this approach, not only can crawl costs be reduced, but SEO efficiency can be maximized by using intensive resources only when truly needed. This would likely also reduce complexity and boost browser performance.
Ce projet a été financé par l’État dans le cadre de France 2030
