25
Dec
What if AI crawlers could bypass downloading entire websites and only receive structured JSON data instead? Serving AI bots this way could drastically reduce inefficiencies, improve response times, and optimize resource use. Imagine delivering just the essential data—product names, prices, descriptions—without unnecessary UI, animations, or assets. This idea was inspired by Vercel’s blog post on AI crawlers and their growing influence on SEO. Here are some highlights that led to this thought: AI crawlers are gaining ground:GPTBot, Claude, AppleBot, and PerplexityBot made 1.3 billion fetches last month, accounting for 28% of Googlebot’s activity. JavaScript rendering limitations:Only Google’s Gemini and AppleBot…