Skip to main content
All CollectionsTroubleshooting
How to monitor 1 million pages
How to monitor 1 million pages

Learn how we can help you monitor up to 1 million important URLs

Updated over 2 weeks ago

A website with 1 million pages (or more) needs to meet these criteria:

  1. The website must meet our minimum technical requirements

  2. The website must group its pages into subdirectories

  3. The subdirectories must be logically structured

Minimum technical requirements

The website you want to monitor needs to meet these technical requirements:

Group pages into subdirectories

Your website needs to group pages into logical subdirectories.

For example, an e-commerce website like Decathlon UK website groups its pages into subdirectories.

Our tool uses URL prefix web properties to help you maximise the URL Inspection API limits.

To create URL prefix web properties, your website must group its pages into subdirectories. These subdirectories can then be turned into URL-prefix web properties.

Learn more about how Indexing Insight works.

Logically structured subdirectories

Finally, the website needs to group its pages into a logical hierarchical structure.

The website needs to have 100+ subdirectories that group pages into hierarchical structures:

For example, Crate and Barrel website has well over 1 million pages, but the website divides hundreds of subcategories into hierarchical subdirectories.

The hierarchical subdirectory structure allows you to create hundreds of web properties at scale.

When you create more web properties, you can fetch more pages daily. This allows you to monitor pages within the recommended 30-day window.

For one client, we can monitor up to 1 million pages with an inspection cycle of 16 days.

We’re currently fetching 200,000 URLs per day using hundreds of web properties (that were created at scale in the tool).

The inspection cycle must stay under 30 days.

After testing the data, we found that the longer it takes to inspect a page, the less accurate the index state of each page. This inaccuracy can cause teams to make the wrong decision.

Learn more about the inspection cycle.

Next Steps

If you’re interested in using Indexing Insight:

Did this answer your question?