Prithvi Earth, Structure
The crawl and render layer
Before any model can quote you, a crawler has to fetch, render and parse your page. Prithvi is the ground everything else stands on.
What machines check
- AI crawlers such as GPTBot, ClaudeBot, PerplexityBot and Google-Extended allowed in robots.txt
- Key content in server-rendered HTML, not hidden behind JavaScript
- Valid Organization, Service, Article and FAQ structured data

