All Collections
Crawler
Why is my website not crawling?
Why is my website not crawling?

If your website is not crawlable check out this content

Updated over a week ago

There could be several different reasons for this.

If your site is protected by a firewall;

In this case, you will see a warning as in the picture.

Please follow the instructions to fix the problem.

How to Add Screpy to Your Firewall

Screpy is a service that allows you to crawl your website for changes and updates, giving you all the data necessary to keep your website running smoothly. In order to make sure that your firewall won't act as an obstacle to Screpy's crawling, you will need to add the IP addresses of the Screpy servers in the settings of your firewall provider.

Setting Up Your Firewall

Before setting up your firewall, you need to make sure that you have all the IP addresses for the Screpy servers. You can easily access this information from the Screpy dashboard. Once you have all the IP addresses you need, enter them into the settings of your firewall provider to allow Screpy access to your website.

Implementing the Settings on All Your Servers

If you have more than one server, make sure you set up the Screpy-specified IP addresses on all of your servers. This way, they won't detect Screpy as a threat and will allow it to crawl your site seamlessly.

Conclusion

By following the steps above, you can easily add Screpy to your firewall and make sure that it has access to your website without any disruption or security threats.

IP

Location

49.13.56.96

Helsinki, Finland

49.13.118.216

Helsinki, Finland

If you have exceeded the crawl limit;

Understanding Crawl Limits in Screpy

If you are using the website audit tool, Screpy, you may come across a message saying that your crawl limit has been reached. This message can be confusing and hinder your progress, so understanding the cause of this issue and how to fix it is important.

What are Crawl Limits?

Crawl limits in Screpy refer to how many sites you are able to crawl and save data from within a given period of time. Depending on the plan you are on, these limits are very different. Limits on free plans are much lower than those of the more premium ones.

What If My Limit Is Reached?

If you are unable to add a new site and are receiving a message stating that your crawl limit has been reached, then we recommend checking your plan badge for confirmation. You can do this by clicking the badge next to the Screpy logo in the upper left corner of your dashboard.

If your browsing limit has indeed been reached, there are two solutions you can explore. You can either upgrade your plan to a higher level with increased limits or wait until your existing limits renew each month. Keep in mind that you must be on a plan in order to use Screpy.

If you think you are experiencing any other issues with the application, please reach out to our support team for further assistance.




















Did this answer your question?