A website with 1 million pages (or more) needs to meet these criteria:
The website must meet our minimum technical requirements
The website must group its pages into subdirectories
The subdirectories must be logically structured
Minimum technical requirements
The website you want to monitor needs to meet these technical requirements:
Google Search Console contains the domain property you want to monitor
Your account has owner or delegated owner access to the domain property
Important pages you want us to monitor are in XML Sitemaps
XML sitemaps submitted to Google Search Console
Group pages into subdirectories
Your website needs to group pages into logical subdirectories.
For example, an e-commerce website like Decathlon UK website groups its pages into subdirectories.
Subdirectories | URLs |
1,500 | |
2,500 | |
12,000 | |
1,000 | |
4,000 | |
1,800 | |
2,600 | |
1,200 | |
800 | |
157,000 |
Our tool uses URL prefix web properties to help you maximise the URL Inspection API limits.
To create URL prefix web properties, your website must group its pages into subdirectories. These subdirectories can then be turned into URL-prefix web properties.
Learn more about how Indexing Insight works.
Logically structured subdirectories
Finally, the website needs to group its pages into a logical hierarchical structure.
The website needs to have 100+ subdirectories that group pages into hierarchical structures:
Level 1: https://example.com/level-1/
Level 2: https://example.com/level-1/level-2/
For example, Crate and Barrel website has well over 1 million pages, but the website divides hundreds of subcategories into hierarchical subdirectories.
Subdirectories | Level | URLs |
1 | 884 | |
2 | 1 | |
2 | 28 | |
2 | 78 | |
1 | 465 | |
2 | 40 | |
2 | 22 |
The hierarchical subdirectory structure allows you to create hundreds of web properties at scale.
When you create more web properties, you can fetch more pages daily. This allows you to monitor pages within the recommended 30-day window.
For one client, we can monitor up to 1 million pages with an inspection cycle of 16 days.
We’re currently fetching 200,000 URLs per day using hundreds of web properties (that were created at scale in the tool).
The inspection cycle must stay under 30 days.
After testing the data, we found that the longer it takes to inspect a page, the less accurate the index state of each page. This inaccuracy can cause teams to make the wrong decision.
Learn more about the inspection cycle.
Next Steps
If you’re interested in using Indexing Insight:
Learn if the tool is the right fit for your website.
Use the inspection cycle calculator
Book a demo call with our team.