Table of Content:
1. Create a new WAF rule
Cloudflare uses several very aggressive methods to block bots. Therefore, Cloudflare also requires several whitelist rules to be defined so that we can crawl your site again.
First, we need to let the IPs bypass through the firewall (in Cloudflare it's not called firewall but WAF) to avoid being blocked by automated processes. To do this, click on Security in the left sidepanel and then on WAF. There you can create a new firewall rule.
2. Set up the BYPASS rule
In the new window you first have to name your new rule (We recommend "Bypass RyteBot IPs"). Then you have to add the first IP source address. After that you can click on "or" on the right to define another IP Source. Enter the next IP. Repeat the process until you have added all the IP addresses provided by the Ryte Support Team.
IMPORTANT! To work around some peculiarities of Cloudflare, please add this IP address IN ADDITION to the IPs that the Ryte Support Team provided you:
At the Action sector, please select "BYPASS" and also check all possible bypass options in the box on the right. Click on Save / Deploy to add the rule.
3. Create another subsequent WAF rule
Now we need one more rule to allow our bot through your settings and through future rules.
4. Set up the ALLOW rule
Enter here again the IP addresses sent to you by the Ryte Support Team (this time please do not add the IP 127.0.0.1)
At the bottom of Action you select "ALLOW". Then save again and Cloudflare is set. But don't forget the ...
5. Most important step!
After you have set up the rules, you need to contact the Ryte Support again. This is because we still need to enable a proxy so that the static IP addresses are used at all.
After we have activated the proxy your website is ready to be analyzed by our bot again
Feel free to contact email@example.com if you have any questions. We are happy to help.