Overview
The Bot Traffic Analytics tool in Scrunch lets you monitor how much access your site is getting from LLM bots—including ChatGPT, Perplexity, Gemini, Grok, and others. Akamai Data Stream 2 broadcasts your traffic logs directly to Scrunch.
If your website is proxied by Akamai, then Data Stream 2 is the perfect tool to broadcast traffic and get metrics from LLM bots in real time.
What You’ll See
Once your Akamai Data Stream 2 is connected, the Bot Traffic dashboard will show:
Total Bot Traffic in the last period
Bot traffic over time
Traffic distribution between Retrieval, Indexer, and Training LLM Bots
Comparison between the current period and the last period (%)
Top bot agents and when they were last seen
Top content pages accessed by LLM bots
Recent bot requests
A date filter to see data from the last 24 hours, last 7 days, or last 30 days
Scrunch AI's Bot Traffic Analytics feature allows customers to granularly track
which AI platforms are consuming their content (and for what purpose) to enable better understanding of how their content:
will be surfaced in AI platforms like ChatGPT
drives AI responses to relevant questions
and ultimately how it influences AI to describe and recommend their brand,
products and services and click through to their site(s).
Adding Your Website
Open the Scrunch app.
Navigate to the Sites menu.
You’ll see the list of websites already connected to Bot Traffic Analytics.
Click + Connect Site at the top.
Select Akamai as your CDN platform.
A dedicated instructions page will appear, showing your Webhook URL and API Key.
Your page will look like this one:
ℹ️ Each site has its own endpoint and key. Don’t reuse them across different sites or integrations.
Integrating Akamai Data Stream 2
Step 1: Configure DataStream 2
Log in to your Akamai Control Center with DataStream access.
Go to Common Services → DataStream and create a new stream.
Provide a Stream Name and continue.
Requirements: Akamai must be serving as a reverse proxy in front of your site, and your account must include DataStream 2.
Step 2: Select Data Parameters
In the Data Sets tab, include all available fields.
At a minimum, Scrunch requires:
Timestamp / Request Time
User Agent
Host
Path
Method
Status Code
Optionally, include:
Client IP / Client ASN (helps confirm bot identity)
Query String (if your content is query-driven)
JA4 / TLS Fingerprint (future-proofing)
⚠️ Omitting required fields will prevent proper bot detection.
Step 3: Configure Destination (Webhook)
In the Destination tab, choose Custom HTTPS Endpoint.
Set Log Format to
JSON.Enter the Webhook URL and X-Api-Key header value from your Scrunch app.
Example:
Webhook URL: https://webhooks.scrunchai.com/v1/sites/<SITE_ID>/platforms/akamai/web-traffic
Header: X-Api-Key = <YOUR_API_KEY>
ℹ️ Akamai does not allow Authorization headers for DataStream, so always use X-Api-Key.
Step 4: Review & Activate Stream
On the Summary step, review your configuration: URL, header, dataset fields, and frequency.
Save and activate the stream.
Akamai will send a one-time verification request to Scrunch.
Step 5: Enable DataStream Behavior on Property
Open Property Manager for your site.
Add the DataStream behavior (v2).
Save and activate your property version.
Step 6: Verify Traffic
Wait up to 5 minutes for your site to show as “Active” in Scrunch.
If you don’t see traffic, test with:
curl -v -H "User-Agent: ScrunchAI-Testbot" https://yourdomain.com
This will send a sample request to confirm logs are flowing.
👉 Once configured, your site will continuously stream traffic logs to Scrunch, giving you real-time visibility into how LLM bots interact with your content.
Troubleshooting and Tips
Don’t see any traffic after integrating?
Ensure the Webhook URL and X-Api-Key match what’s in your Scrunch app.
Verify the DataStream behavior is active in Akamai Property Manager.
Double-check the stream is in Active state in the Akamai console.
Confirm you included all required fields (see Step 2).
Wait 5–10 minutes after activation.
Tips for better results:
Exclude paths for static assets (CSS, JS, images) if you want cleaner data.
Include paths that serve PDFs—AI bots often request them.
Keep stream frequency at 1–5 minutes for near real-time monitoring.
If you manage multiple sites, repeat the process for each site in Scrunch.