The Scrunch Site Audit tool helps you evaluate how well your website is optimized for AI visibility. It runs a series of checks across three categories—Access Controls, Content Delivery, and Content Quality—to identify any barriers that might prevent AI systems from understanding and retrieving your content effectively.
Here’s a breakdown of each category and the checks within them:
1. Access Controls
Is your site configured to allow AI bots to access your web pages at all?
These checks ensure that your site isn’t unintentionally blocking AI bots at the domain, page, or server level:
Is your site configured to allow AI bots to access your web pages at all?
A high-level check confirming whether AI bots can reach your content.Does your site have a
robots.txtfile?
A missingrobots.txtfile can create ambiguity about how bots should interact with your site.Does your
robots.txtallow all key AI bots to access your site?
Even if arobots.txtfile exists, it must explicitly allow known AI user agents.Does your
robots.txthave specific rules allowing all key AI bots?
Not strictly required, but recommended—especially for larger organizations where internal teams may manage content and access differently. Clear allow-lists help avoid unintentional blocking.When any key AI bot requests this page, do they get a successful response?
This checks whether bots like ChatGPT-User or PerplexityBot receive a 200 OK response when accessing a specific page.If an AI bot was blocked, was it because of an identifiable bot-blocking solution that can be reconfigured?
This identifies tools or configurations (e.g., bot management software) that may be unintentionally blocking access.
2. Content Delivery
Is your content delivered with technology and speed that AI bots can process?
These checks focus on the page’s load time and how content is rendered:
Does your page load in under 5 seconds?
AI bots tend to deprioritize or skip slow-loading pages. Aim for quick, lightweight delivery.Is there meaningful content on your page that can be delivered without JavaScript?
AI bots may not execute JavaScript reliably. If important content is only available via JavaScript, it may not be seen.
3. Content Quality
Is your content complete, and in a format that is optimized for AI bots?
These checks evaluate whether your content is accessible, relevant, and within AI-readable limits:
Is the page short enough to be completely read by AI?
AI models typically operate within a token limit (e.g., ~128,000 tokens total across all sources in a query). We recommend keeping your primary content within ~10,000 tokens (~7,500 words) to ensure it fits in the AI’s context window.Are the page title and description relevant to page content?
Clear and descriptive metadata helps AI systems understand what your page is about, which improves its chances of being selected and referenced in responses.Is the non-JavaScript version of the page substantially the same as the JavaScript version?
Many AI bots do not execute JavaScript, so they only see the raw HTML. If your key content (like product descriptions, headlines, or body text) is only rendered via JavaScript or renders at the User-level, it is likely invisible to AI. Make sure the core content is present in the HTML source so that bots - and search engines - can access it without relying on scripts.
Pro Tip
AI systems look for clear, relevant, and accessible content - not clickbait. Think of your site not just as something to attract human visitors, but also as a source of answers for AI agents. The Site Audit helps you speak both languages.
Reminder: Site Audit is a One-Time Snapshot
The Site Audit tool performs a one-time check of your page at the moment it’s run. If you make updates to your site’s content, metadata, access controls, or speed, be sure to rerun the audit in Scrunch to see how those changes impact your results. This ensures you’re always working from the most current data.