SEO GEO AI Discovery
AI Discovery And Crawler Policy
Public policy for how iGaming Suite exposes product knowledge to search engines, AI search crawlers and browser agents while protecting authenticated workspaces and customer data.
The policy follows official crawler distinctions from Google, Bing, OpenAI, Anthropic and Perplexity, and treats optional files such as llms.txt as agent conveniences instead of ranking promises.
Who it is for Search engines, AI retrieval systems, browser agents, developers and partners.
What it does
- Search and retrieval crawlers can discover public pages
- Training crawler policy is documented separately
- Private app paths are disallowed from public crawling
- Structured public content must remain truthful and reviewed
Inside the Suite
A real product view
Crawler policy by function
The policy separates public search/retrieval discovery from private product access. Search and retrieval crawlers can read public pages, docs, research, glossary, robots.txt, sitemap.xml, llms.txt and ai-manifest.json. Authenticated Suite routes, API responses, customer data, admin tools, tokenized capture links and unreviewed directory records remain private.
- Search and retrieval: public pages should be crawlable and indexable where the content is meant to be found.
- Training use: training crawler policy is documented separately and can be changed without opening private routes.
- User-initiated agents: agent fetchers may read public docs, but cannot bypass Suite authentication or API-key scope.
- Security challenge risk: public discovery URLs must not depend on JavaScript challenges that non-browser crawlers cannot pass.
What llms.txt means here
The llms.txt file is useful as a compact public map for agents, but it is not treated as a Google ranking lever. The durable discovery work is the crawlable HTML, internal links, canonical URLs, sitemap coverage, truthful entity definitions and source-backed content.
- Do maintain llms.txt as a helpful public navigation artifact.
- Do not claim that llms.txt guarantees AI citations or Google visibility.
- Do use ai-manifest.json to list public routes, entities and private boundaries in a machine-readable way.
Why the policy is public
AI search teams, SEO professionals and browser-agent builders need a public policy they can cite. This page explains how iGaming Suite allows public discovery while refusing private data exposure, cloaking and unreviewed automated action.
- Search and retrieval crawlers can read public pages.
- Private Suite workspaces, APIs and user data are excluded.
- The policy gives future automations a boundary to test against.
Product surfaces
These are the product names, modules and data objects used inside iGaming Suite today.
- robots.txt
- sitemap.xml
- llms.txt
- ai-manifest.json
FAQ
Is llms.txt a Google ranking factor?
No. iGaming Suite keeps llms.txt as an agent-readable map, but the Google-facing foundation is crawlable helpful content, indexing eligibility, canonical URLs, snippets and sitemap coverage.
Can AI crawlers access private Suite data?
No. Public crawler access applies only to public pages and machine-readable discovery files. Suite workspaces, API data, customer records, sessions and admin controls remain private.
What is the biggest deployment risk for AI discovery?
A crawler or agent receiving a browser-only security challenge on public URLs. That must be checked in Vercel Firewall/Bot Management and Search Console URL Inspection.