WordFence for WordPress is a big topic at Ryte. We are looking for a solution.
We are using dynamic IPs. They are changing very fast and it's possible to crawl your website with the same user agent but with different IP-addresses.

The Ryte Crawler is using the user-agent "Rytebot" by default.

Unfortunately, there is no way to whitelist a user agent in WordFence, only IP addresses. The number of our crawler IP-addresses is enormous, the IP addresses themselves are changing constantly.

One possible whitelisting solution is to whitelist the whole IP-range of Amazon Webservice (AWS) Europe in Ireland. But this would open the doors for many other crawlers as well.

You can deactivate the blocking of fake google crawlers in WordFence, but just for a few hours until we have finished crawling your domain and turn it on again afterwards.

Did this answer your question?