All Collections
Project Settings
Project set up
Why was my project not crawled completely?
Why was my project not crawled completely?
Berislav avatar
Written by Berislav
Updated over a week ago

When your project was not crawled completely, please have a look into the Project Settings -> Project Setup to find out what's missing from the crawl:

In this example we see that some URLs were ignored, this can be due to unintended crawl settings.

In case you activated Obey robots.txt all "disallowed" URLs will be ignored, deactivated subdomains will lead to ignored URLs to your subdomains and so on. Please also have a look at the other settings such as Blacklist/Whitelist and crawl subfolder.

Did this answer your question?