Indexing Insight helps monitor Google indexing through URL Inspection API.
Before implementing our tool, you should evaluate if your website meets our requirements for optimal performance.
Websites perfect for Indexing Insight
A website that has these 3 characteristics get the most value from our tool:
Logical directory structure: A website organizes pages into subdirectories.
Fast inspection cycle: The inspection cycle must stay under 30 days.
Balanced URL distribution: A website follows 2 rules to balance URL distribution.
Group pages into subdirectories.
Your website needs to group pages into logical subdirectories.
For example, an e-commerce website like Decathlon UK website groups its pages into the following subdirectories.
Subdirectories | URLs |
1,500 | |
2,500 | |
12,000 | |
1,000 | |
4,000 | |
1,800 | |
2,600 | |
1,200 | |
800 | |
157,000 |
Our tool uses URL prefix web properties to help you maximise the API limits.
To create URL prefix web properties, your website must group its pages into subdirectories. These subdirectories can then be turned into URL-prefix web properties.
For more detail learn more about how Indexing Insight works.
Fast Inspection Cycle
The inspection cycle must stay under 30 days.
After testing the data, we found that the longer it takes to inspect a page, the less accurate the index state of each page. This inaccuracy can cause teams to make the wrong decision.
Our team have created the traffic light system to help highlight the accuracy of the data.
Data Quality Risk | Inspection Cycle | Description |
None | 1 day | No chance of a monitored page’s indexing state being a false positive or negative. |
Low | 2-7 days | Low chance of a monitored page’s indexing state being a false positive or negative. |
Moderate | 8 - 30 | Moderate chance of a monitored page’s indexing state being a false positive or negative. |
High | 30+ | High chance of a monitored page’s indexing state being a false positive or negative. |
This is because indexing is a continuous process that constantly changes.
To improve the accuracy of your data, we recommend keeping the inspection cycle (maximum days) for your web properties between 1 and 30 days.
For moderate or high-risk data, we recommend verifying the indexing status directly in Google Search Console and monitoring the pages over time.
Balanced URL Distribution
A website must keep the URL distribution balanced to keep inspection cycles fast.
There are 2 rules to keep the URL distribution balanced:
A website with multiple large subdirectories contains a maximum of 60,000 URLs.
A website with a single large subdirectory contains a maximum of 180,000 URLs.
A website with multiple large subdirectories
A website with multiple large subdirectories contains a maximum of 60,000 URLs.
For example, the Rvshare website has multiple category directories with less than 60,000 URLs. So, the inspection cycle for most web properties is 1-2 days.
Subdirectory URL | URLs | Daily limit | Inspection Cycle |
300 | 2,000 | 1 | |
131 | 2,000 | 1 | |
322 | 2,000 | 1 | |
400 | 2,000 | 1 | |
900 | 2,000 | 1 | |
5,500 | 2,000 | 3 | |
5,000 | 2,000 | 3 | |
100 | 2,000 | 1 | |
21,000 | 2,000 | 5 | |
52,000 | 2,000 | 9 |
A website with a single large subdirectory
A website with a single large subdirectory contains a maximum of 180,000 URLs.
For example, the Decthlon UK website has as single large product directory. There are under 160,000 URLs in this site section. And the other subdirectories contain less than 60,000 URLs.
For any other SEO tool it would take 79 days to fetch the indexing data, but with Indexing Insight, it only takes 26 days.
Subdirectory URL | URLs | Daily limit | Inspection Cycle |
1,500 | 2,000 | 1 | |
2,500 | 2,000 | 2 | |
12,000 | 2,000 | 6 | |
1,000 | 2,000 | 1 | |
1,200 | 2,000 | 1 | |
800 | 2,000 | 1 | |
157,000 | 2,000 | 26 |
Types of websites that may not fit Indexing Insight
There are 3 types of websites that are not ideal for our tool:
Limited subdirectories: A website with pages grouped into 2-3 subdirectories.
Flat website architecture: A website that doesn't have subdirectories.
Extremely large directories: A subdirectory with more than 160,000 URLs.
Limited Subdirectory Structure
A website with pages grouped into 1-3 subdirectories with large numbers of URLs.
This means your indexing data takes over 30 days to be monitored and has a high chance of false positives.
Web Properties | URLs | Daily limit | Inspection Cycle |
500,000 | 2,000 | 84 days | |
150,000 | 2,000 | 75 days | |
10,000 | 2000 | 5 days |
Flat Website Architecture
The website does not group pages into subdirectories.
This means our tool cannot maximise the URL Inspection API limits by creating URL-prefix web properties.
For example, if you monitored a site with 1 million pages but no subdirectories, scanning every page would take us 500 days (even with us maximising API limits).
The chance of false positives in this data is extremely high.
Web Properties | URLs | Daily limit | Inspection Cycle |
1,000,000 | 2,000 | 500 days |
Extremely Large Directories
A website has more than 180,000 pages in one large subdirectory.
For example, Zapier.com’s famous apps directory has 7 million URLs. Monitoring all these pages in this directory would take our tool 3 years (even with us being 3x faster).
Web Properties | No. of URLs | Daily limit | Inspection Cycle |
7,000,000 | 2,000 | 1,166 days |
However, if Zapier were to focus its monitoring on other subdirectories on the website, this would be possible. We could monitor multiple subdirectories with less than 60,000 URLs.
Web Properties | No. of URLs | Daily limit | Inspection Cycle |
1,150 | 2,000 | 1 day | |
8,000 | 2,000 | 4 days | |
98 | 2,000 | 1 day |
Next Steps
If you’re still interested in Indexing Insight, please follow these steps:
Grab a copy of our calculator.
Follow the steps in the How to Calculate your Inspection Cycle
Either purchase a plan that is recommended OR contact our team for a demo