Table of Content:
1. Create an IP Access rule
Cloudflare uses several very aggressive methods to block bots. Therefore, Cloudflare also requires several whitelist rules to be defined so that we can crawl your site again.
Log in to the Cloudflare dashboard, and select your account and domain.
Navigate to Security > WAF > Tools.
Under IP Access Rules, enter the following details:
For Value, enter an IP address, IP range, country code/name, or Autonomous System Number (ASN). For details, refer to Parameters.
Select an action.
For Zone, select whether the rule applies to the current website only or to all websites in the account.
(Optional) Enter a note for the rule (for example,
Payment Gateway).
Select Add.
2. Create a new SKIP-rule
In the second step, we need to make sure that the IPs of the Ryte crawler are not checked by the firewall (in Cloudflare it's not called firewall but WAF) to avoid being blocked by automated processes.
Go to Security > WAF > Custom rules.
Create a custom rule by selecting Create rule (or edit an existing custom rule).
Define the rule name (like: "Whitelist RyteBot").
If incoming requests match should look like this (enter all provided IPs here):
IP Source Address + equals + 123.4.5.6 (example IP)
or
IP Source Address + equals + 123.4.5.6 (example IP)
or
IP Source Address + equals + 123.4.5.6 (example IP)
or
IP Source Address + equals + 123.4.5.6 (example IP)
or
IP Source Address + equals + 123.4.5.6 (example IP)
Under Then take action, select Skip from the dropdown.
Skip all possible components to fully whitelist the Ryte-Crawler:
WAF components to skip
✅ All remaining custom rules
✅ All rate limiting rules
✅ All managed rules
✅ All Super Bot Fight Mode Rules
More components to skip
✅ Zone Lockdown
✅ User Agent Blocking
✅ Browser Integrity Check
✅ Hotlink Protection
✅ Security Level
✅ Rate limiting rules (previous version)
✅ Managed rules (previous version)
Deploy (save) your changes.
This is how your SKIP-Rule should look like in the end:
Sources:
3. Position the SKIP-rule correctly
If there are several rules in the Custom Rules overview, it is important that the newly created rule is at the top in position 1
To achieve this, the rule can be moved up using the arrows.
Afterwards it should look like this
4. Most important step!
After you have set up the rules, you need to get back to your Ryte contact person. This is because we need to activate the proxy server (so that the static IPs are used at all for our crawler).
After we have activated the proxy, your website is ready to be analyzed by our bot again
Feel free to contact support@ryte.com if you have any questions. We are happy to help.
Didn't find what you need? Maybe you can find it on the official Cloudflare website here.