Category: AI Search Optimization

With the rise of AI-driven search engines such as ChatGPT, Perplexity, and Gemini, understanding how your website content is rendered has become more important than ever.

Traditional SEO best practices already emphasised indexability for Googlebot but AI crawlers behave differently.

Recent tests, including a case study on aisearchoptimization.in
, reveal that AI crawlers often fail to process content loaded using Client-Side Rendering (CSR). In other words, if your data appears only after clicking a button or through JavaScript, AI bots might never see it.

1. How AI Crawlers Work

AI crawlers such as those from ChatGPT, Perplexity, and Gemini typically:

• Fetch a page’s HTML source code directly from the server.
• Parse visible text and meta data.
• Use large language models (LLMs) to summarise, categorise, and store information for future retrieval.

However, these crawlers generally don’t execute JavaScript.
That means if your content is loaded dynamically after page load, it doesn’t exist in the HTML snapshot the crawler receives.

2. The Problem with Client-Side Rendering

Client-Side Rendering (CSR) is when content is generated on the user’s browser via JavaScript frameworks such as React, Vue, or Angular.

<div id="ai-stats"></div>
<script>
  fetch('/api/ai-data')
    .then(response => response.json())
    .then(data => document.getElementById('ai-stats').innerHTML = data.stats);
</script>

To a user, this looks fine the stats appear after a button click.
But to an AI crawler (or even a basic SEO bot), the HTML it fetches might look like this:

<div id="ai-stats"></div>

Result? Empty content.

This is exactly what happens with some AI crawlers, the crawlers can’t see the dynamically loaded numbers unless they’re present in the HTML at load time.

3. Why It Matters for AI Search Optimisation

AI search engines rely on content visibility rather than traditional link crawling.
If your critical data (like pricing, reviews, or research stats) is missing from the HTML:

• AI tools won’t cite your content in responses.
• Your brand’s authority in generative answers decreases.
• Organic AI traffic (from ChatGPT or Perplexity) may drop dramatically.

In short, if AI can’t read it, you don’t exist in the AI index.

4. The Solution: Server-Side Rendering (SSR) or Static HTML

To make your content AI-friendly:

5. Quick Fix Checklist

✅ Ensure all essential text appears in the raw HTML source (right-click → “View Page Source”).
✅ Use Server-Side Rendering (SSR) for your main content.
✅ Avoid loading key text only after button clicks or fetch calls.
✅ Test pages with AI crawlers.
✅ Validate visibility using AI-Search engines.

6. Key Takeaway

To be visible in the age of AI Search, your content must live in the HTML, not in JavaScript.
AI crawlers are getting smarter, but they still can’t “click”, “scroll”, or “wait for scripts” like human users can.

The simplest way to future-proof your site for AI visibility is to adopt Server-Side Rendering (SSR) or static HTML generation.

🧠 If AI can’t read your content, it can’t recommend you.

I recently conducted an experiment on my test website. I applied the noindex meta robots tag to every page of the site. Then, I prompted several AI search engines to extract specific information from the website and observed the responses.

For this scenario, let’s consider my test website to be domain.in.

The prompt I used was: “What services does https://www.domain.in/ offer as a business? Please check the website directly before answering.”

Key Findings

• ChatGPT: Accessed the site’s content, as indicated by the ChatGPT-User/1.0 agent in my server logs (+https://openai.com/bot), and accurately quoted the requested information.
• Perplexity: Did not retrieve any content from the website, suggesting it honors the noindex meta tag. Its response explicitly stated that direct website access is unavailable and that no content from the test site appears in search results.
• Claude AI: Successfully obtained the required answers. The server logs showed the user agent Claude-User/1.0; +Claude-User@anthropic.com.
• Google AI Mode: Generated fabricated information unrelated to the actual site content. This indicates it primarily relies on Google’s search index during its query process.
• Deepseek AI: Could not access any content from the test website. Its response specified that it doesn’t browse the web directly and depends entirely on search results.
• Qwen AI: Managed to retrieve content from the website, but its server logs revealed it used a HeadlessChrome browser rather than a specific user agent.

This shows to me that adding noindex meta robots tag is not a safe way to block web-pages from AI Search engines.

So after doing the above I had another test website where i added the below in my robots.txt which basically disallowed all the bots from accessing the website.

User-agent: *
Disallow: /

However, even after adding the script, I used the below prompt to ask question about this new test website.

Prompt: what services does https://www.domain.com/ as a business provide. Please check the website now and then only let me know. as they have updated their website What makes them different and why should i choose them.

ChatGPT (specifically ChatGPT-User Agent as appearing in the Server Logs) was able to access my website and was able to quote the information as i asked in the prompt.

I also tested the same prompt in other AI Answer engines but all the rest were not able to fetch any content from the website. Please see the findings below for more info.

Claude AI: It specifically said that its unable to access the website directly as it appears to be blocked by robots.txt rules. This shows that it abides by the robots.txt rules.

Perplexity AI: It wasnt able to get any information from the website. It implies that it respect the robots.txt rule.

Deepseek: It wasnt able to get any information too. It gave the following response with some related information.

Based on the search results provided, I do not have specific information about the services offered by https://www.domain.in/, as this particular website was not included in the search results. However, I can provide a general overview of what a typical AI Search Optimization Agency might offer based on the industry trends and common services described in the search results.

It just shows that Deepseek relies on Search results from certain search engines.

Google AI Mode: Wasn’t able to provide any information from the website.

Qwen AI: Wasn’t able to provide any information from the website.

This implies to me that even when you use robots.txt Disallow rule, its upto the AI Search Engine bot to respect it or not. So if you’re serious about this, you should use services like Cloudflare to block these bots which actively block these bots by identifying their IP Address and other criteria.

Do you know that you can access your website server logs or access logs in Hostinger Hpanel even for Starter Plans?

If not then please have a read and at the end you would be able to access the Server Logs and able to find any 4XX and 5XX errors from your logs.

Please follow the below steps as mentioned.

1. First Login to your Hostinger account.
2. Select your Hosting Package and Click on Manage

3. Now select the website from the dropdown which appears in top portion of left hand vertical menu.
4. Click on Analytics which appears on the second vertical menu from the left as shown below.

5. Now click on Access logs Tab as shown

This is were you will get see all your logs. You will be able to see the time when your website was accessed, IP Address, Country, Device/User Agent and Response Time.

You can access what kind of bot accessed your website. For AI search engine optimization, we can specifically look at user agents such as GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, PerplexityBot and many others. You can get an idea which pages these bots are requesting and which they don’t. This will give you an idea about the pages served in AI search results on these platform. Additionally you can also check the response time column to get an idea if the response time is healthy or certain pages are taking long time to respond to these bot requests.

Note: You can’t export the Server logs from here and can only access a maximum of Last 7 days log data.

Finding Error Logs

6. Go through the Error Code 5xx tab to find all the pages having internal server errors. This will help you to find and resolve the issue which these AI bots are facing while accessing your web-pages
7. At last, Go through the Error Code 4xx tab to find any broken pages. This will help you to either fix these pages or redirect these broken links to closely related web-pages.

Once you have access to server logs, you can run various analysis on the data and find various issue and resolve it. If you are interested on how to do the analysis, please go through this Log Files Analysis guide by Matt Diggity.