Skip to main content

Connecting your website to Bot Traffic Analytics using Cloudflare Enterprise Logpush

How to stream traffic logs from Cloudflare (CDN) to Scrunch’s Bot Traffic Analytics using Logpush → HTTPS with the required fields

Updated over 2 weeks ago

Overview

Scrunch’s Bot Traffic Analytics shows how LLM bots hit your site (e.g., ChatGPT, Perplexity, Gemini, Grok)—including volume over time, top pages, and recent requests. If your site is proxied by Cloudflare with an Enterprise subscription, you can use Logpush to send request logs straight to Scrunch over HTTPS, and no code is needed.


What You’ll See

Once your Logpush is connected, the Bot Traffic dashboard will show:

  • Total Bot Traffic in the last period

  • Bot traffic over time

  • Traffic distribution between Retrieval, Indexer, and Training LLM Bots

  • Comparison between the current period and the last period (%)

  • Top bot agents and when they were last seen

  • Top content pages accessed by LLM bots

  • Recent bot requests

  • A date filter to see data from the last 24 hours, last 7 days, or last 30 days

Scrunch AI's Bot Traffic Analytics feature allows customers to granularly track which AI platforms are consuming their content (and for what purpose) to enable a better understanding of how their content:

  • will be surfaced in AI platforms like ChatGPT

  • drives AI responses to relevant questions

  • and ultimately, how it influences AI to describe and recommend their brand,
    products and services, and click through to their site(s).


Adding Your Website

  1. Open the Scrunch app

  2. Go to the Sites menu

  3. You’ll see the list of your websites connected to Bot Traffic Analytics

  4. Click + Connect Site

  5. Select Cloudflare Enterprise as the platform

  6. You’ll see an instructions page with your Webhook URL, Site ID, and API Key

Your page will look like this one:

ℹ️ Each site has its own endpoint and key. Don’t reuse them across different sites or integrations.


Integrating Cloudflare Enterprise Logpush

Step 1: Create a Logpush Job

  • Log in to your Cloudflare dashboard

  • Access a domain

  • Navigate to Analytics & Logs → Logpush and click on + Create a Logpush job

  • Choose HTTP Destination

  • On the next page, paste your Scrunch Webhook URL and append header params:

    https://webhooks.scrunchai.com/v1/sites/{SITE ID}/platforms/cloudflare-enterprise/web-traffic?header_X-Api-Key={TOKEN}

Step 2: Selecting fields

  • On the dataset, pick HTTP Requests

  • Give a name to your Logpush job (without spaces or special characters)

  • Select the required fields to proceed with Scrunch Bot Analysis. The required fields are:

    • EdgeStartTimestamp

    • ClientRequestHost

    • ClientRequestUserAgent

    • ClientRequestURI

    • ClientRequestPath

    • ClientRequestMethod

    • EdgeResponseStatus

    • EdgeEndTimestamp

    • ClientIP

  • Click on Submit and wait to receive the next batch from CloudFlare Enterprise Logpush

Step 3: Advanced Settings

  • Click on Advanced Options and ensure the timestamp format is rfc3339

  • Sampling should be left at 100% so Scrunch AI can properly process all website activity

Step 3: Verify Traffic

  • Wait up to 5 minutes for your site to show as “Active” in Scrunch.

  • If you don’t see traffic, test with:

curl -v -H "User-Agent: ScrunchAI-Testbot" https://yourdomain.com

This will send a sample request to confirm logs are flowing.

👉 Once configured, your Cloudflare Logpush will continuously forward logs to Scrunch, giving you real-time visibility into how LLM bots access your website.


Troubleshooting and Tips

Don’t see any traffic?

  • Ensure the Logpush job is enabled

  • Revisit the required fields on the Logpush job configuration and confirm if all required fields are checked

  • Wait up to 5 minutes—Scrunch may take a short time to detect your configuration.

Tips for better results:

  • Select all fields from CloudFlare Enterprise Logpush

  • Repeat the process if you manage multiple sites

Did this answer your question?