Overview
The Agent Traffic tool in Scrunch lets you monitor how much access your site is getting from LLM bots—including ChatGPT, Perplexity, Gemini, Grok, and others.
If your website is proxied by Cloudflare, you can use Cloudflare Workers to send traffic logs to Scrunch. The Worker runs at the edge and forwards request details without slowing down your site.
Cloudflare CDN and Cloudflare Workers are offered separately by Cloudflare, and are not included in your Scrunch plan. Cloudflare offers generous free plans for smaller websites. For higher-traffic websites, please note that there may be additional costs associated with deploying Agent Traffic via Workers. See our additional guidance below.
What You’ll See
Once your Cloudflare Worker is connected, the Bot Traffic dashboard will show:
Total Bot Traffic in the last period
Bot traffic over time
Traffic distribution between Retrieval, Indexer, and Training LLM Bots
Comparison between the current period and the last period (%)
Top bot agents and when they were last seen
Top content pages accessed by LLM bots
Recent bot requests
A date filter to see data from the last 24 hours, last 7 days, or last 30 days
Scrunch AI's Agent Traffic feature allows customers to granularly track
which AI platforms are consuming their content (and for what purpose) to enable better understanding of how their content:
will be surfaced in AI platforms like ChatGPT
drives AI responses to relevant questions
and ultimately how it influences AI to describe and recommend their brand,
products and services and click through to their site(s).
Adding Your Website
Open the Scrunch app.
Go to the Agent Traffic menu.
You’ll see the list of your websites connected to Agent Traffic.
Click + Connect Site.
Select Cloudflare as the platform.
You’ll see an instructions page with your Webhook URL, Site ID, and API Key.
Your page will look like this one:
ℹ️ Each site has its own endpoint and key. Don’t reuse them across different sites or integrations.
Integrating Cloudflare Workers
Step 1: Ensure your website is proxied by CloudFlare
EdgeWorkers will run only if your website is proxied by CloudFlare. You can check if the website is proxied by the DNS menu on the domain level.
Then, check if your website is proxied or not. Your website DNS is normally served by A or CNAME rules
⚠️ If the website is not proxied (orange cloud), then the next steps won't work
Step 2: Create a Worker
Log in to your Cloudflare dashboard.
Navigate to Workers & Pages → Create Application
Select Start with Hello World!
Give it a name (e.g.
bot-tracking-worker) and then click on Deploy
Example:
If everything went well, the page will look like this:
Now, click on Edit code and follow the next steps.
Step 3: Add Worker Code
The Worker intercepts requests, collects traffic data (user agent, path, status code, etc.), and sends it to Scrunch asynchronously.
Replace the default Worker code with the following snippet
function getUserAgent(request) {
return (
request.headers.get("X-Original-UA")?.toLowerCase() ||
request.headers.get("User-Agent")?.toLowerCase() ||
"unknown"
);
}
async function logToScrunchWebhooks(data, env) {
try {
const response = await fetch(env.SCRUNCH_WEBHOOK_URL, {
method: "POST",
headers: {
"Content-Type": "application/json",
"X-Api-Key": env.SCRUNCH_API_KEY,
},
body: JSON.stringify(data),
});
if (!response.ok) {
console.error(
"Failed to log to Scrunch webhooks:",
response.status,
response.statusText
);
}
} catch (error) {
console.error("Error logging to Scrunch webhooks:", error);
}
}
export default {
async fetch(request, env, ctx) {
const url = new URL(request.url);
const startTime = Date.now();
const userAgent = getUserAgent(request);
const logAndReturn = (response) => {
ctx.waitUntil(
logToScrunchWebhooks(
{
timestamp: new Date().toISOString(),
site_id: env.SCRUNCH_SITE_ID,
domain: url.hostname,
user_agent: userAgent,
url: url.toString(),
path: url.pathname,
method: request.method,
status_code: response.status,
response_time_ms: Date.now() - startTime,
ip_address: request.headers.get("CF-Connecting-IP") || "unknown",
},
env
)
);
};
try {
const forwardedRequest = new Request(request.url, {
method: request.method,
headers: request.headers,
body: request.body,
redirect: request.redirect,
signal: request.signal,
});
const response = await fetch(forwardedRequest);
logAndReturn(response);
return response;
} catch (error) {
const errorResponse = new Response("Internal Server Error", {
status: 500,
});
logAndReturn(errorResponse);
return errorResponse;
}
},
};Click on Deploy
Step 4: Set Environment Variables
In your Worker settings, go to Settings → Variables.
Add the following environment variables
SCRUNCH_SITE_ID=<your-site-id>
SCRUNCH_WEBHOOK_URL=<your-webhook-url>
SCRUNCH_API_KEY=<your-api-key>
Step 5: Deploy the Worker
Click Deploy.
Once deployed, your Worker will begin forwarding traffic logs to Scrunch.
Step 6: Configure Custom Domain
To place your Worker in front of your site's domain and begin recording bot traffic for the site, you'll need to configure Custom Domains on the Worker in Cloudflare.
Go to Workers & Pages → your-worker → Settings → Triggers → Custom Domains.
Add your domain using wildcards
For example:
testing.com/*
Step 7: Verify Traffic
Wait up to 5 minutes for your site to show as “Active” in Scrunch.
If you don’t see traffic, test with:
curl -v -H "User-Agent: ScrunchAI-Testbot" https://yourdomain.com
This will send a sample request to confirm logs are flowing.
👉 Once configured, your Cloudflare Worker will continuously forward logs to Scrunch, giving you real-time visibility into how LLM bots access your website.
Troubleshooting and Tips
Don’t see any traffic?
Make sure your Worker is deployed and active.
Double-check environment variables (
SCRUNCH_SITE_ID,SCRUNCH_WEBHOOK_URL,SCRUNCH_API_KEY).Check your Worker logs in Cloudflare for any errors.
Wait up to 5 minutes—Scrunch may take a short time to detect your configuration.
Tips for better results:
Repeat the process if you manage multiple sites.
Always use the credentials shown in your Scrunch app for each site. Don't reuse credentials across sites.
Limitations and Costs with Agent Traffic on Cloudflare Workers
Agent Traffic via Cloudflare Workers works by placing a small Worker ("reverse proxy") in front of your entire website. This means that, by default, all requests to your website domain will go through the Worker, regardless of origin. (Scrunch's Agent Traffic will log and track requests only for LLM bot user agents.)
It's important to understand that Cloudflare Workers is a cloud computing product that uses usage-based billing, and therefore using Workers can incur additional costs outside of your Scrunch plan.
Cloudflare provides a generous free tier for Workers which covers 100,000 requests per day. For small sites, the entire traffic footprint (humans and bots) of the site will often fall within the 100,000 request limit. For moderately larger sites, Workers Free will likely not provide enough capacity for your site's traffic.
If this is the case, there are several solutions you can pursue:
Upgrade to Cloudflare Workers Paid
The simplest solution is to upgrade to a Cloudflare Workers Paid plan, which is $5 per month. Workers Paid includes 10 million requests per month, or approximately 333 thousand requests per day, and allows for overage billed at $0.30 per million requests.
For example, if your website receives 10 million requests per day, the total cost of Cloudflare Workers will be approximately $90 per month.
If your website receives between 100,000 and 1 million requests per day, your total monthly cost should be less than $20.
Use Cloudflare Logpush (Cloudflare Enterprise plan only)
For Cloudflare Enterprise customers, we recommend using our Cloudflare Logpush integration, which has no additional costs on either the Cloudflare or Scrunch side and does not require a Worker of any kind.
Cloudflare has announced their intention to make their enterprise-only features, presumably including Logpush, available on all plans in the future. However, Logpush is still Enterprise-only for now.
Add Agent Traffic to underlying website platform (Wordpress)
For customers on WordPress as their underlying site technology, Scrunch also supports capturing agent traffic via a WordPress plugin. There are no additional costs associated with the plug-in and performance / server load impact from the plugin is typically minimal to none.
Scrunch plans to natively support additional website / CMS platforms for Agent Traffic in the future.
Switch to Snippet-based routing and route traffic selectively to the worker (Cloudflare Pro plan+)
Rather than using Cloudflare Workers Routes to place the Worker in front of your existing website, you can remove the Worker Custom Domain and use Cloudflare Snippets to route traffic more selectively -- for example, based on user agent, request path, type of content requested, etc.
Cloudflare Snippets are only available on paid Cloudflare plans (starting at $20/mo); however there are no additional costs or usage-based billing associated with Snippets. If you would like to implement Snippet-based routing, contact Scrunch Customer Support to receive a template Snippet you can adapt for your needs.
Important things to know:
Scrunch can’t offer direct technical support for debugging Cloudflare Snippets to customers. We recommend this option only for customers who are reasonably comfortable with Cloudflare and the idea of defining traffic routing rules.
Scrunch cannot update Snippet code or rules on your behalf. You will need to be responsible for making any updates to change how traffic is sent to the Agent Traffic worker (e.g. to capture or exclude particular user agents.)