Skip to main content

Connecting your website to Bot Traffic Analytics using Vercel

How to stream traffic logs from Vercel to Scrunch’s Bot Traffic Analytics using Drains

Updated over 2 weeks ago

Overview

Scrunch’s Bot Traffic Analytics shows how LLM bots hit your site (e.g., ChatGPT, Perplexity, Gemini, Grok)—including volume over time, top pages, and recent requests. If your site is hosted by Vercel with an Enterprise or Pro subscription, you can use Drains to send request logs straight to Scrunch over HTTPS, and no code is needed.


What You’ll See

Once your Drain is connected, the Bot Traffic dashboard will show:

  • Total Bot Traffic in the last period

  • Bot traffic over time

  • Traffic distribution between Retrieval, Indexer, and Training LLM Bots

  • Comparison between the current period and the last period (%)

  • Top bot agents and when they were last seen

  • Top content pages accessed by LLM bots

  • Recent bot requests

  • A date filter to see data from the last 24 hours, last 7 days, or last 30 days

Scrunch AI's Bot Traffic Analytics feature allows customers to granularly track which AI platforms are consuming their content (and for what purpose) to enable a better understanding of how their content:

  • will be surfaced in AI platforms like ChatGPT

  • drives AI responses to relevant questions

  • and ultimately, how it influences AI to describe and recommend their brand,
    products and services, and click through to their site(s).


Adding Your Website

  1. Open the Scrunch app

  2. Go to the Sites menu

  3. You’ll see the list of your websites connected to Bot Traffic Analytics

  4. Click + Connect Site

  5. Select Vercel as the platform

  6. You’ll see an instructions page with your Drain destionation URL and API key header

Your page will look like this one:

ℹ️ Each site has its own drain destination URL and API key header. Don’t reuse them across different sites or integrations.


Integrating Vercel Log Drains

Step 1: Choose data to drain

  • Log in to your Vercel account

  • Navigate to Settings Drains and click on Add Drain

  • Choose Custom HTTP endpoint

  • On the next step, select Logs and click Next

Step 2: Configure the drain

  • Give your drain a name ("Scrunch Bot Traffic Monitoring" for example), select the project you'd like to add monitoring for and for Sources, select Functions, Edge Functions, Static Files and Rewrites

  • Confirm which environments you'd like to enable Bot Traffic Monitoring for

  • Sampling Rate should be left at 100% so Scrunch AI can properly process all website activity

  • Click Next

Step 3: Configure the destination

  • For URL, paste your scrunch Drain destination URL

  • Ensure Encoding is set to JSON

  • Enable the Custom Headers toggle

  • In the header configuration input, paste in scrunch's API key header as-is

  • Click Create Drain

Step 4: Verify Traffic

  • Wait up to 5 minutes for your site to show as “Active” in Scrunch.

  • If you don’t see traffic, test with:

curl -v -H "User-Agent: ScrunchAI-Testbot" https://yourdomain.com

This will send a sample request to confirm logs are flowing.

👉 Once configured, your Vercel Log Drain will continuously forward logs to Scrunch, giving you real-time visibility into how LLM bots access your website.


Troubleshooting and Tips

Don’t see any traffic?

  • Ensure the Log Drain is not paused

  • Revisit the projects, sources and environments settings in your Vercel dashboard

  • Wait up to 5 minutes—Scrunch may take a short time to detect your configuration.

Tips for better results:

  • Repeat the process if you manage multiple sites

Did this answer your question?