A recent investigation has uncovered a problem for websites relying on JavaScript for structured data.
This data, often in JSON-LD format, is difficult for AI crawlers to access if not in the initial HTML response.
Crawlers like GPTBot (used by ChatGPT), ClaudeBot, and PerplexityBot can’t execute JavaScript and miss any structured data added later.
This creates challenges for websites using tools like Google Tag Manager (GTM) to insert JSON-LD on the client side, as many AI crawlers can’t read dynamically generated content.
Key Findings About JSON-LD & AI Crawlers
Elie Berreby, the founder of SEM King, examined what happens when JSON-LD is added using Google Tag Manager (GTM) without server-side rendering (SSR).
He found out why this type of structured data is often not seen by AI crawlers:
- Initial HTML Load: When a crawler requests a webpage, the server returns the first HTML version. If structured data is added with JavaScript, it won’t be in this initial response.
- Client-Side JavaScript Execution: JavaScript runs in the browser and changes the Document Object Model (DOM) for users. At this stage, GTM can add JSON-LD to the DOM.
- Crawlers Without JavaScript Rendering: AI crawlers that can’t run JavaScript cannot see changes in the DOM. This means they miss any JSON-LD added after the page loads.
In summary, structured data added only through client-side JavaScript is invisible to most AI crawlers.
Why Traditional Search Engines Are Different
Traditional search crawlers like Googlebot can read JavaScript and process changes made to a webpage after it loads, including JSON-LD data injected by Google Tag Manager (GTM).
In contrast, many AI crawlers can’t read JavaScript and only see the raw HTML from the server. As a result, they miss dynamically added content, like JSON-LD.
Google’s Warning on Overusing JavaScript
This challenge ties into a broader warning from Google about the overuse of JavaScript.
In a recent podcast, Google’s Search Relations team discussed the growing reliance on JavaScript. While it enables dynamic features, it’s not always ideal for essential SEO elements like structured data.
Martin Splitt, Google’s Search Developer Advocate, explained that websites range from simple pages to complex applications. It’s important to balance JavaScript use with making key content available in the initial HTML.
John Mueller, another Google Search Advocate, agreed, noting that developers often turn to JavaScript when simpler options, like static HTML, would be more effective.
What To Do Instead
Developers and SEO professionals should ensure structured data is accessible to all crawlers to avoid issues with AI search crawlers.
Here are some key strategies:
- Server-Side Rendering (SSR): Render pages on the server to include structured data in the initial HTML response.
- Static HTML: Use schema markup directly in the HTML to limit reliance on JavaScript.
- Prerendering: Offer prerendered pages where JavaScript has already been executed, providing crawlers with fully rendered HTML.
These approaches align with Google’s advice to prioritize HTML-first development and include important content like structured data in the initial server response.
Why This Matters
AI crawlers will only grow in importance, and they play by different rules than traditional search engines.
If your site depends on GTM or other client-side JavaScript for structured data, you’re missing out on opportunities to rank in AI-driven search results.
By shifting to server-side or static solutions, you can future-proof your site and ensure visibility in traditional and AI searches.
Featured Image: nexusby/Shutterstock
