Search Focus Are A SEO Company That Can Increase Your Online Sales Revenue Through Driving Qualified Targeted Visitors To Your Website
AI Crawlers Cannot Read Your JavaScript: Is Your Website Invisible to AI?
A website can look complete, work perfectly and even perform well in Google while remaining almost empty to an AI crawler.
This is something I now consider whenever I review a website that relies heavily on JavaScript. The visitor may see headings, service information, product specifications, comparison tables, prices, reviews and internal links. However, the crawler may receive little more than a page title and an empty <div> waiting for JavaScript to populate it.
That difference is becoming more important as people increasingly use ChatGPT, Claude, Perplexity and other AI tools to research businesses, compare services and find answers.
When important website content only appears after JavaScript has run, some AI systems may never receive enough information to understand, retrieve or cite the page properly.
The Problem in One Sentence
Many AI crawlers fetch the HTML returned by the server but do not execute the JavaScript required to build the completed page.
When I assess this issue, I normally compare four different versions of the same page:
| Version of the Page | What It Contains |
|---|---|
| Initial HTML | The code returned immediately by the server |
| Rendered Document Object Model | The completed page after JavaScript has run |
| Browser View | What the visitor sees on screen |
| AI Crawler View | Often only the initial server response |
Large-scale testing published by Vercel and MERJ found that several major AI crawlers did not render JavaScript. The crawlers tested included OpenAI’s GPTBot, OAI-SearchBot and ChatGPT-User, along with ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider.
Some of the crawlers downloaded JavaScript files, but the researchers found no evidence that they executed those files to construct the page.
Googlebot works differently. Google documents a separate rendering stage in which a headless version of Chromium can execute JavaScript and process the rendered HTML. Even so, Google continues to recommend server-side rendering or pre-rendering because other crawlers may not have the same capabilities.
This is why I never assume that strong Google visibility automatically means that a page is equally accessible to AI search systems.
Raw HTML and Rendered HTML Are Not the Same
Consider a basic client-side application.
The server may initially return something like this:
<!doctype html>
<html lang="en">
<head>
<title>Commercial Scaffolding Services</title>
</head>
<body>
<div id="app"></div>
<script src="/assets/app.js"></script>
</body>
</html>
A normal browser downloads app.js, executes it and inserts the page content into the empty application container.
After rendering, the visitor might see:
<main>
<h1>Commercial Scaffolding Services</h1>
<p>
We provide designed scaffolding systems for commercial
construction, refurbishment and maintenance projects.
</p>
<h2>Our Commercial Scaffolding Services</h2>
<ul>
<li>Temporary Roofs</li>
<li>Access Scaffolding</li>
<li>Loading Bays</li>
<li>Edge Protection</li>
</ul>
</main>
To the visitor, this looks like a useful and complete service page.
However, a crawler that does not execute the script may only receive:
<div id="app"></div>
From my point of view, this is the central problem. The business has created the content, published it and made it visible to users, but the crawler cannot access it in a meaningful form.
Why Inspect Element Can Give the Wrong Impression
One of the most common mistakes I see is checking a page through the browser’s developer tools and assuming that the HTML shown there is what every crawler receives.
The Elements or Inspector panel normally shows the live, rendered Document Object Model. It includes content added or changed by JavaScript after the initial page loaded.
That makes it useful for inspecting layouts and debugging scripts, but it does not necessarily show the original server response.
When I want to understand what a non-rendering crawler may receive, I start with View Page Source.
I then search the source code for:
- The Main H1 Heading
- The Opening Paragraph
- Product or Service Descriptions
- Prices and Specifications
- Internal Links
- Comparison Table Content
- Review Text
- Structured Data
- Canonical Tags
- Important Calls to Action
When content is clearly visible in the browser but cannot be found in the page source, I know JavaScript is probably responsible for adding it.
That does not guarantee that the page will be invisible to every search or AI platform. It does mean the page’s accessibility depends on whether each crawler can render it correctly.
Google Rendering Does Not Prove AI Crawler Accessibility
Google has spent years developing its Web Rendering Service.
Its documented process broadly involves three stages:
- Googlebot Fetches the URL.
- Google Processes the Initial HTML and Discovers Links.
- The Page Enters a Rendering Queue Where JavaScript Can Be Executed.
Once the necessary resources are available, Google uses a headless version of Chromium to render the page and process the completed HTML.
Most AI crawlers have been developed for different purposes. Some gather potential training material, some create search indexes and others retrieve pages in response to specific user requests.
Rendering JavaScript at scale also requires substantially more computing power than downloading ordinary HTML. A crawler must load scripts, process network requests, execute code and wait for the page to reach a usable state.
A basic HTML request is considerably faster and less expensive.
For that reason, I do not rely on the following assumption:
Google can index the content, so every AI platform can read it.
Google visibility and AI-search visibility overlap, but they are not the same thing.
AI Crawlers Do Not All Serve the Same Purpose
The term “AI crawler” covers several different types of activity.
I think this distinction is particularly important because a website owner may want their content to appear in AI search results while still choosing to restrict its use for model training.
Training Crawlers
Training crawlers collect material that may be used to develop or improve foundation models.
Examples include:
- GPTBot
- ClaudeBot
OpenAI advises publishers to disallow GPTBot on pages they want to exclude from potential model training.
Anthropic also identifies ClaudeBot separately from its search and user-request bots. It states that its bots honour standard robots.txt directives.
Search and Indexing Crawlers
These crawlers discover and analyse content so it can be retrieved through an AI search product.
Examples include:
- OAI-SearchBot
- Claude-SearchBot
- PerplexityBot
OpenAI states that publishers should allow OAI-SearchBot when they want their content to be included in ChatGPT search summaries and snippets.
Anthropic describes Claude-SearchBot as a crawler used to improve the relevance and accuracy of search responses.
Perplexity describes PerplexityBot as a crawler designed to surface and link to websites in its search results. It also states that this particular bot is not used to gather material for training foundation models.
User-Initiated Fetchers and Agents
A user-fetch agent accesses a page because somebody has asked an AI system a question or instructed it to interact with a website.
Examples include:
- Claude-User
- Perplexity-User
- Browser-Based ChatGPT Agents
These should not be confused with background search crawlers.
Some browser-based agents can interact with JavaScript websites. OpenAI, for example, advises developers to use clear WAI-ARIA labels and roles so browser agents can interpret interactive components.
However, the ability of a browser agent to interact with a rendered website does not prove that an indexing crawler will execute the same JavaScript.
I separate them in the following way:
| Agent Type | Main Purpose | Rendering Should Be Assumed? |
|---|---|---|
| Training Crawler | Collect Potential Training Material | No |
| Search Crawler | Discover and Index Content | No |
| User Fetcher | Retrieve Content for a Specific Request | No |
| Browser Agent | Interact With a Website | Possibly, Depending on the Agent |
| Googlebot | Search Crawling and Indexing | Yes, Subject to Rendering Conditions |
Which Websites Are Most at Risk?
JavaScript itself is not the problem.
I use JavaScript on websites regularly, and it remains essential for creating useful forms, calculators, filters and interactive features.
The risk appears when essential public content depends entirely on client-side JavaScript.
Single-Page Applications
React, Vue and Angular applications can return a minimal application shell before constructing the page inside the visitor’s browser.
When client-side rendering is used without server-side rendering, static generation or pre-rendering, the initial HTML may contain very little useful information.
The framework is not necessarily the issue. The way the website has been configured and deployed is what matters.
Product and Ecommerce Pages
Ecommerce websites frequently load important information asynchronously, including:
- Product Descriptions
- Prices
- Stock Information
- Variations
- Specifications
- Reviews
- Delivery Details
- Comparison Tables
When an AI crawler receives only a generic product template, it may not have enough information to distinguish one product from another.
I therefore look beyond whether the URL can technically be crawled. The crawler also needs access to the facts that make the page useful.
Comparison and Calculator Pages
Interactive calculators and comparison tools often place their most valuable content inside JavaScript components.
The form may work perfectly for a visitor, but the crawler may not receive:
- Example Prices
- Calculation Factors
- Explanatory Text
- Comparison Criteria
- Typical Outcomes
- Supporting Guidance
I generally keep the interactive calculation client-side while ensuring that the purpose, inputs, methodology and representative results are explained in accessible HTML.
This gives the crawler useful context even when it cannot complete the interaction itself.
Service and Location Pages
Some websites request service or location content from an application programming interface after the page template loads.
This can create hundreds of accessible URLs that all return approximately the same empty server response.
The title may change from one location to another, but the service descriptions, local information and calls to action remain hidden until JavaScript runs.
When I audit location-page templates, I check several URLs rather than assuming that one functioning browser page proves the entire template is accessible.
Documentation and Knowledge Bases
Documentation websites often use JavaScript to load articles, code samples, navigation trees and accordion content.
This is particularly important because detailed documentation is exactly the type of material an AI system may want to retrieve when answering a technical question.
A documentation page that only delivers an empty shell may be visually impressive but difficult for a non-rendering crawler to use.
Content Hidden Behind Interaction
Important content may only be requested after the visitor:
- Opens an Accordion
- Selects a Tab
- Scrolls Down the Page
- Clicks a “Load More” Button
- Chooses a Product Variation
- Enters a Postcode
- Accepts a Cookie Banner
I distinguish between content that already exists in the source but is hidden with CSS and content that is only fetched after an interaction.
The second type creates the greater risk because the crawler may never receive it at all.
Warning Signs I Look For
This type of problem rarely creates a clear warning in an analytics dashboard.
A website can continue receiving Google traffic, making missing AI visibility difficult to identify.
The strongest warning signs I look for include:
- Important Text Is Missing From View Source
- The Page Becomes Empty When JavaScript Is Disabled
- Multiple URLs Return Nearly Identical Raw HTML
- Titles or Canonicals Are Added Only After Rendering
- Internal Links Use Click Handlers Instead of HTML Anchors
- Product Data Arrives Through Browser-Side API Requests
- Content Appears Only After Scrolling or Clicking
- Server Logs Show AI Crawler Visits but Little Related Visibility
- An AI Tool Can Find the URL but Cannot Describe the Page Accurately
Another sign is inconsistent retrieval.
An AI platform may mention a business using information from directories, review sites or third-party publications but fail to use information from the company’s own website.
In my view, this may indicate that the website is discoverable but its main content is difficult to extract.
How I Test What an AI Crawler Can Read
No single test perfectly reproduces every AI platform.
I normally combine several checks to understand what the server returns and how much of the page depends on JavaScript.
1. Check the Page Source
I open the page normally and select View Page Source.
I then search for a distinctive sentence from the main body content.
Searching only for the title is not enough. A page can return the correct title while the rest of its content remains missing.
I check whether the source contains:
- The Full Main Heading
- Several Complete Paragraphs
- Important Entity Names
- Prices or Product Specifications
- Links to Related Pages
- Structured Data
- Contact Information
When these elements are absent, I inspect how and when they are inserted.
2. Fetch the Page With cURL
A command-line request shows the HTML returned without loading the page in a browser.
curl -L https://www.example.co.uk/example-page/
I can also save the response to a file:
curl -L https://www.example.co.uk/example-page/ \
-o page-source.html
I then search for a known phrase:
grep -i "known sentence from the page" page-source.html
To test whether the server treats a recognised crawler differently, I can change the user agent:
curl -L \
-A "OAI-SearchBot" \
https://www.example.co.uk/example-page/
This does not reproduce every part of a genuine crawler request, but it can reveal whether the server, firewall or rendering service applies different rules to bots.
3. Disable JavaScript
I disable JavaScript in the browser and reload the page.
I then ask three straightforward questions:
- Can I Still Identify the Page’s Subject?
- Can I Read Its Main Information?
- Can I Follow Links to Related Pages?
The page’s interactive features do not necessarily need to work without JavaScript.
However, I want the server-delivered version to contain enough information to explain what the page is about and where the visitor can go next.
4. Compare the Source With the Rendered DOM
Where necessary, I compare:
- The Original HTTP Response
- The Rendered HTML After JavaScript Has Run
A large difference is not automatically a problem. Navigation menus, interactive controls and visual components may reasonably be added after loading.
I concentrate on differences involving the main text, metadata, structured data and internal links.
5. Review Server Logs
Server logs can show whether recognised AI user agents are requesting important pages.
I search for names such as:
GPTBot
OAI-SearchBot
ChatGPT-User
ClaudeBot
Claude-SearchBot
Claude-User
PerplexityBot
Perplexity-User
User-agent strings can be copied or spoofed, so I would also verify published IP address ranges where accurate crawler identification is important.
Logs help answer two different questions:
- Are AI Crawlers Reaching the Website?
- What Response Are They Receiving?
A successful 200 status code does not prove that the crawler has received the page’s meaningful content. The response could still contain little more than an empty application shell.
The Fastest Practical Solution
The fastest dependable solution is to include the page’s essential content in the initial HTML response.
That does not mean I remove JavaScript or avoid using interactive components.
JavaScript can still control:
- Filters
- Calculators
- Forms
- Accordions
- Animations
- Product Selectors
- Live Availability
- Personalised Features
My priority is to stop JavaScript from becoming the only source of information that crawlers need.
For most websites, I would work through the following order:
- Render the Main Content on the Server.
- Include Crawlable Internal Links in the Initial HTML.
- Place Important Metadata in the Server Response.
- Add JavaScript Interactivity Through Hydration.
- Test the Raw Response After Every Major Deployment.
Comparing the Main Technical Fixes
| Method | How It Works | Best Use | Main Limitation |
|---|---|---|---|
| Server-Side Rendering | Generates HTML for each request | Dynamic Public Pages | More Server Processing |
| Static Site Generation | Creates HTML During the Build | Articles, Services and Documentation | Rebuilds May Be Needed |
| Incremental Static Regeneration | Refreshes Static Pages Periodically | Large or Frequently Updated Sites | Framework-Specific Setup |
| Hydration | Adds Interactivity to Existing HTML | Interactive Marketing and Product Pages | Requires Careful Implementation |
| Pre-Rendering | Produces Static HTML Ahead of Crawling | Existing Client-Side Applications | Can Add Operational Complexity |
| Dynamic Rendering | Serves a Rendered Version to Selected Bots | Temporary Legacy Workaround | Bot Detection and Maintenance |
Google now describes dynamic rendering as a workaround rather than a recommended long-term solution.
Where possible, I favour server-side rendering, static rendering or hydration because they provide a more consistent version of the page to both visitors and crawlers.
Server-Side Rendering
With server-side rendering, the server executes the application logic and returns completed HTML.
The browser can then hydrate the page so interactive components continue to work.
This gives the crawler immediate access to the content while preserving the website’s interactive features.
Static Generation
Static generation creates complete HTML files before a visitor or crawler requests them.
I find this particularly suitable for:
- Blog Posts
- Service Pages
- Location Pages
- Documentation
- Guides
- Category Introductions
- Evergreen Product Information
The page can still use JavaScript, but its central information does not depend on it.
Incremental Static Regeneration
Incremental regeneration combines static output with periodic updates.
This can work well when a website contains thousands of public pages that change more frequently than a traditional static build can comfortably manage.
Hydration
Hydration begins with meaningful server-rendered HTML and then connects JavaScript functionality to it.
I generally prefer this to beginning with an empty container and generating the entire page in the browser.
Dynamic Rendering
Dynamic rendering detects selected crawlers and sends them a pre-rendered version of the page.
It can provide a temporary solution for an older client-side application while a more permanent fix is developed.
However, it introduces several risks:
- New Crawlers Must Be Identified and Added
- User Agents Can Change or Be Spoofed
- Rendered Versions Can Become Stale
- Users and Crawlers May Receive Different Content
- Rendering Services Can Fail
- Testing Becomes More Complicated
I would only use it as a temporary workaround.
The crawler and visitor versions should also contain equivalent information. Serving substantially different content can create cloaking concerns.
What I Want to See in the Initial HTML
A crawler does not need every visual effect or interactive component.
It does need enough information to identify, interpret and connect the page.
For a typical commercial page, I want the initial response to contain the following.
Clear Page Identification
I check for:
- A Descriptive Title Element
- A Relevant Meta Description
- One Clear Main Heading
- A Canonical URL
- The Correct Language Attribute
Substantive Main Content
The initial response should answer the visitor’s main questions.
For a service page, that may include:
- What the Service Is
- Who It Is For
- Where It Is Available
- What Is Included
- Typical Costs or Pricing Factors
- Relevant Qualifications or Standards
- The Next Step
Crawlable Internal Links
Important links should use normal anchor elements:
<a href="/commercial-scaffolding/">
Commercial scaffolding
</a>
I avoid relying entirely on elements such as:
<div onclick="openServicePage()">
Commercial scaffolding
</div>
The first version provides a real destination in the HTML. The second depends on JavaScript executing correctly before the destination can be discovered.
Structured Data
Where appropriate, I include structured data in the server-delivered HTML rather than injecting it only after the page has loaded.
Google may process JavaScript-generated structured data after rendering, but a non-rendering AI crawler may never receive it.
This is particularly important for information such as products, services, organisations, articles, prices and availability.
Meaningful Text Around Interactive Tools
I do not like calculator pages that consist of little more than input fields and a button.
I would normally include text explaining:
- What the Calculator Estimates
- Which Factors Affect the Result
- What Assumptions It Makes
- Whether Figures Include VAT
- Why the Final Quote May Differ
- What the Visitor Should Do Next
This supports the visitor while also providing useful context to crawlers that cannot operate the calculator.
WordPress Websites Are Not Automatically Safe
A conventional WordPress page normally returns its written content in the initial HTML.
However, I have learned not to assume that every WordPress website is safe from JavaScript rendering problems.
Issues can still appear when content is loaded through:
- AJAX Requests
- Faceted Search Plugins
- Product Variation Scripts
- Popup Builders
- JavaScript Tabs
- External Review Widgets
- Headless WordPress Frontends
- Custom React or Vue Components
- “Load More” Pagination
- Third-Party Booking Systems
Page builders can also place important information inside components that only load after an interaction.
I always prefer to inspect the finished response instead of relying on what the platform or plugin is expected to do.
React, Next.js, Vue and Nuxt Require Deployment Checks
Modern frameworks can produce highly accessible HTML, but their capabilities depend on how they are configured.
A Next.js website may use:
- Static Generation
- Server-Side Rendering
- React Server Components
- Client-Side Rendering
- A Combination of Methods
The framework name alone does not tell me what a crawler receives.
The same applies to Vue and Nuxt. A correctly rendered Nuxt page may return complete HTML, while a client-only Vue application may return little more than an empty mounting point.
I therefore test real production URLs after deployment.
Development previews and browser-based checks do not always reveal what the production server, cache or content delivery network returns.
Robots.txt Controls Access, Not Rendering
Allowing an AI crawler through robots.txt does not give that crawler the ability to execute JavaScript.
It simply gives the crawler permission to request the relevant paths.
A business may decide to allow search-related crawlers while blocking crawlers associated with model training.
For example:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
A similar distinction can be made for Anthropic’s bots:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
I see these as publishing decisions rather than universal recommendations.
Blocking a search crawler may reduce the likelihood that first-party website content appears accurately in that platform’s results.
Allowing it means the crawler can request the page, but the server still needs to return content that the crawler can understand.
I also check firewalls and bot-management tools. A crawler can be permitted in robots.txt while still being blocked by a security service or content delivery network.
Why I Do Not Treat llms.txt as a Complete Solution
An llms.txt file is intended to give AI systems a simplified description or map of a website.
It may become a useful supporting format, but I do not view it as a replacement for accessible pages.
A crawler still needs to retrieve and understand the original source material.
An llms.txt file cannot repair:
- Empty HTML Responses
- Blocked Crawlers
- Missing Internal Links
- Client-Only Product Data
- Incorrect Status Codes
- Inaccessible Documentation
- JavaScript-Generated Canonicals
- Content Hidden Behind Authentication
I would treat it as an optional supplementary layer rather than a solution to JavaScript rendering problems.
My Practical Repair Process
I normally prioritise pages according to their commercial and informational importance.
Step 1: Select Critical Templates
I begin with:
- Main Service Pages
- High-Traffic Blog Posts
- Product Pages
- Comparison Pages
- Pricing Pages
- Location Landing Pages
- Documentation
- Calculators
When thousands of pages share the same template, I do not need to test every URL individually. I select representative examples and then investigate the underlying template.
Step 2: Compare Source and Rendered Content
I record which elements are missing from the initial source.
I then separate essential content from optional interface components.
Step 3: Identify the Rendering Method
I determine whether the page uses:
- Server-Side Rendering
- Static Generation
- Client-Side Rendering
- Hybrid Rendering
- Asynchronous Browser Requests
Step 4: Move Essential Content Into the Server Response
I start with the H1, primary copy, important facts, internal links, metadata and structured data.
Interactive enhancements can remain client-side.
Step 5: Recheck Bot Controls
I review:
robots.txt- Meta Robots Directives
X-Robots-TagHeaders- Firewall Rules
- Content Delivery Network Bot Settings
- Authentication Requirements
- Cookie or Geographic Restrictions
Step 6: Test the Production Website
I do not rely solely on a staging environment.
Caching, edge functions, deployment settings and firewalls can all change what the live website returns.
Step 7: Monitor Logs and Visibility
I track recognised crawler activity, AI referral traffic and whether important pages begin appearing more accurately in AI-generated answers.
OpenAI states that referrals from ChatGPT search include a utm_source=chatgpt.com parameter, which can help identify visits in analytics platforms.
How I Prevent the Problem Returning
Rendering problems can reappear after a redesign, plugin update, framework migration or deployment change.
For that reason, I prefer to include raw HTML checks in the release process rather than treating this as a one-off technical SEO task.
My basic prevention checklist confirms that:
- Important Content Exists in the Initial Response
- Internal Links Have Real
hrefAttributes - Titles and Canonicals Are Server-Rendered
- Structured Data Does Not Depend Entirely on JavaScript
- Critical Pages Work Without Browser-Side API Calls
- Content Is Not Locked Behind Unnecessary Interaction
- Search Crawlers Are Not Accidentally Blocked
- Firewall Rules Do Not Contradict
robots.txt - Production Responses Match the Intended Templates
- New Components Are Tested With JavaScript Disabled
Automated tests can also fetch representative URLs after each deployment and check for required content.
For example, a service-page test could fail when the response does not contain:
<h1>
It could also check for a known service description, a canonical tag and a minimum amount of body text.
This does not measure the quality of the content, but it can identify an empty-shell deployment before it affects hundreds or thousands of pages.
Accessible HTML Is Becoming More Important, Not Less
I do not believe AI visibility requires creating a separate website for machines.
The more reliable approach is to return a useful HTML document before optional browser enhancements are applied.
That principle supports several objectives at once:
- Search Engine Accessibility
- AI Crawler Accessibility
- Faster Initial Page Loads
- Better Website Resilience
- Improved Accessibility
- More Reliable Link Discovery
- Easier Technical Testing
JavaScript remains an essential part of the modern web. The mistake is not using it.
The mistake is allowing it to become the only route to information that should have been available in the original document.
A visitor may be willing to wait for an application to load.
A non-rendering crawler will not.
When I work on a website, I want the initial HTML to contain the real heading, the real copy, the real links and the real page-specific facts.
That creates a much safer foundation because the website no longer depends on every search engine and AI platform behaving like a complete browser.
As AI-driven search continues to develop, I expect this distinction between what a visitor sees and what a crawler receives to become an increasingly important part of technical SEO.
