AI Crawlers and JavaScript: Why ChatGPT May See an Empty Page

Short answer: With AI crawlers and JavaScript the rule is simple: a bot that does not run scripts only sees the HTML your server sends. If your text is built in the browser, that HTML can be an empty shell. In a measurement by Vercel and MERJ from December 2024, none of the major AI crawlers executed JavaScript, while Google's crawler and AppleBot did. That is a snapshot from 2024, so test your own pages and put the main content into the HTML itself: render it on the server or generate it ahead of time.

How does a crawler get your page?

Every fetch starts the same way: a request to your server, answered with HTML. What happens next decides how much of your page arrives.

A browser loads the scripts the HTML points to, runs them and shows the result. A crawler may stop after the first HTML. Or it renders: it runs the scripts, waits, and reads only afterwards.

If the server sends the text, both paths lead to the same content. If the page is assembled in the browser, the first HTML contains hardly any. Google's documentation calls this the "app shell" model. A bot without JavaScript receives roughly this:

<body>
  <div id="root"></div>
  <script src="/app.js"></script>
</body>

Headline, text, product list and links appear only after /app.js has run. Not every crawler runs it, and that is the whole issue.

What Google does with JavaScript

Google is the easy case. Its documentation on JavaScript SEO basics says:

• Google Search runs JavaScript with an "evergreen" version of Chromium.
• A page goes through three phases: crawling, rendering and indexing.
• A page with status 200 is put into a queue for rendering, which can take seconds or longer. For a page with a non-200 status, such as a 404, rendering can be skipped.
• Title, meta description and canonical may be set with JavaScript.

The same page also gives the reason this article exists. Google writes that server-side rendering or pre-rendering is still a good idea, because it makes the page faster for users and crawlers and because not all bots can run JavaScript. "Google indexes my page" therefore does not mean that every bot can read it.

AI crawlers and JavaScript: what is actually known

The best-known measurement comes from Vercel and MERJ, published in December 2024 under the title The rise of the AI crawler. Their finding: none of the major AI crawlers executed JavaScript. The study names GPTBot, OAI-SearchBot, ChatGPT-User and ClaudeBot among them. Googlebot (listed in the study for Gemini) and AppleBot rendered JavaScript.

Two cautions belong next to that result. First, it is a snapshot from 2024, and we have no newer measurement to cite. Crawlers change, so treat it as a warning and not as a law. Second, it tells you what a bot reads, not whether a page is quoted in an answer. Nobody can promise a citation, with or without JavaScript.

The bots themselves have different jobs. This is what the operators say about them:

The sources are OpenAI, Anthropic and Perplexity. Perplexity also has a separate bot for fetches on a user's request. If the bots behind AI search answers read only HTML, a page with an empty shell gives them nothing to quote. Which of them render scripts is exactly what the 2024 study measured, and what you should check on your own pages.

Who is affected?

You are affected if the text or the links of your pages do not exist in the HTML your server sends:

• Pages that are assembled completely in the browser. The server answers with a shell and a script bundle, and the script writes headline, text and navigation.
• Content that scripts load afterwards. Product lists, prices, reviews, tabs or "load more" sections that come from an API call after the page has loaded.
• Links that exist only after scripts have run. A crawler without JavaScript can only follow links that are in the HTML. SEOgrinder's own site crawl runs into this too: it notes that links which arise only in the browser cannot be followed.
• Title, description, canonical and Open Graph tags that a script sets. Google accepts this. A bot that skips scripts never sees them.

You are not affected if the server already sends the text. Interactive extras, such as a menu, a chart or a calculator, can still be built in the browser. SEOgrinder flags pages that exist only in the browser as "The page is only assembled in the browser", because many AI crawlers run no JavaScript and see an almost empty page there.

Three tests in one minute

You do not need a tool to find out. All three tests answer the same question: is my text in the HTML the server sends?

1. View the page source. Open your page and choose "View page source" (Ctrl+U, or Cmd+Option+U on a Mac). Do not use "Inspect": the inspector shows the page after the scripts have run. Search the source for a sentence you can read on the page.
2. Fetch the page with curl. In a terminal, run curl -sL https://example.com/your-page | grep -io "a short phrase from your page", using a short phrase from your own page. If the phrase is printed, it is in the HTML, unless it appears only inside a script element, which does not count. If nothing is printed, check the spelling and special characters first; otherwise it is not in the HTML.
3. Switch JavaScript off. Open the developer tools, open the command menu (Ctrl+Shift+P, or Cmd+Shift+P on a Mac), type "Disable JavaScript", select it and reload the page. If the main content is gone, a bot without JavaScript sees the same.

How to read the result:

Two limits: curl shows what your server sends to curl, not what it sends to a particular bot, for example when a firewall treats bots differently. And to see what Google has rendered, use the URL inspection in Google Search Console.

How to fix it: put the content into the HTML

The goal is that headings, body text and internal links are visible in the source. There are two common ways to get there:

• Server-side rendering (SSR). The server builds the finished HTML for each request.
• Pre-rendering, also called static generation (SSG). The pages are generated ahead of time, for example at build time, and the server simply delivers the files.

Both put the content into the first response. Beyond that:

• Start with the pages that matter. A full rewrite is rarely necessary. Begin with the pages that should rank or be quoted.
• Put the essentials into the first HTML. That means title, meta description, canonical, Open Graph tags, the main text and navigation links written as real <a href> elements.
• Give crawlers and visitors the same content. Google lists "dynamic rendering", where bots get a pre-built version, only as a workaround. Showing crawlers something different from what visitors see is cloaking.
• Check the result in the source. Repeat the three tests after the change.

None of this is a ranking guarantee. It removes a technical obstacle, nothing more.

How SEOgrinder detects pages built in the browser

SEOgrinder looks at the HTML your server delivers, not at what a browser makes of it. Enter an address, start the analysis, and the tasks appear in the Tasks tab, sorted by impact. Three findings belong to this topic:

• The page is only assembled in the browser. The finding states how many words, headings and internal links the delivered HTML holds, and notes that Google renders such pages with a delay and that many AI crawlers see an almost empty page.
• "N pages are only assembled in the browser". The site crawl reports how many of the crawled pages deliver an almost empty HTML shell. Pages set to noindex and sign-in or sign-up pages are not counted.
• Sign-in or sign-up pages that need JavaScript. This is usually fine, because such pages work for visitors without server-side rendering. If one is meant to rank, render it on the server.

After a fix, press Check again: reports are cached for up to 24 hours, and this button forces a fresh run (a few times per hour). Hand over to AI turns the report into a Markdown work order for a coding agent such as Claude Code, Cursor or GitHub Copilot, with one verifiable target condition per finding.

What SEOgrinder does not do: it does not measure whether ChatGPT, Gemini or Perplexity actually name your brand. It checks the technical basis, which means whether the content can be read at all.

Common mistakes

• Testing with "Inspect" instead of "View page source". The inspector shows the result after the scripts have run, which is exactly what a bot without JavaScript does not get.
• Concluding "Google indexes it, so every bot can read it". Google runs JavaScript, and its own documentation says that not all bots do.
• Setting title and meta tags only with a script. Google allows it, but a bot that does not run scripts never sees them. Put them into the first HTML.
• Links that are only click handlers. Use real <a href> links so that a crawler can follow them without running scripts.
• Making dynamic rendering the permanent plan. Google calls it a workaround.
• Answering every unknown address with the shell and status 200. Client-rendered apps often do this, and such pages can be treated as soft 404. Our article on soft 404 errors explains the fix.
• Allowing bots in robots.txt and turning them away at the firewall. The text exists, but the bot gets an error. SEOgrinder has a finding for this case: "Server may turn away allowed AI crawlers". See also our article on robots.txt for AI crawlers.

FAQ

Does Google see my JavaScript page?

Usually yes, but later. Google runs JavaScript with an evergreen Chromium, and pages with status 200 wait in a queue for rendering. Indexing is never guaranteed. The URL inspection in Google Search Console shows what Googlebot sees.

Is a sitemap enough?

No. A sitemap tells crawlers which addresses exist. It does not put text into the HTML of those pages.

Do ChatGPT and Claude run JavaScript?

In the 2024 measurement by Vercel and MERJ, the crawlers of OpenAI and Anthropic named in it did not. We have no newer measurement to cite, so test your own pages instead of relying on either answer.

Do I have to rewrite my whole site?

Not necessarily. Server-side rendering or pre-rendering for the pages that should rank or be quoted is the usual first step. Interactive parts can stay in the browser.

Your next step

Pick one important page and run the three tests. If the text is missing from the source, run the free scan in SEOgrinder and open the Tasks tab. Fix the pages that matter most by rendering them on the server or generating them ahead of time, then press Check again.