User-agent: * Allow: / # Block authenticated and admin paths from crawl budget Disallow: /v2/dashboard/ Disallow: /admin/ # AI crawlers — explicitly welcome User-agent: ClaudeBot Allow: / User-agent: GPTBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Applebot-Extended Allow: / Sitemap: https://agenttext.com/sitemap.xml