Skip to main content

How to Exclude URLs from Crawling with LinkStorm

Learn how to prevent specific pages or sections of your website from being crawled using URL exclusion patterns.

Updated over 2 weeks ago

The URL Exclusion settings allow you to prevent specific pages or entire sections of your website from being crawled and included in link opportunity analysis.

When to use URL exclusions

Use URL exclusions when you want LinkStorm to ignore:

  • Language or region sections (e.g. /fr/*, /es/*)

  • Archive or low-value content (e.g. /archive/*, /tag/*)

  • Admin or system paths (e.g. /wp-admin/*)

  • Subdomains (e.g. store.example.com)

How to add exclusion patterns

  1. Go to your website's Settings page.

  2. Click URL Exclusions.

  3. Type a pattern in the input field and click Add.

  4. Click Save changes.

πŸ’‘ Tip: Click Scan Sitemap to automatically detect directory structure from your XML sitemap β€” then tick the sections you want to exclude.

Pattern syntax

Patterns are matched against the full URL of each page. Use * as a wildcard.

Pattern

What it excludes

/blog/*

All pages under /blog/ (e.g., /blog/post1, /blog/sub/page)

/seo*

Pages starting with /seo (e.g., /seo, /seo-basics, /seo-local)

/category/

Only the exact page /category/ β€” not pages under it

/category/*

All pages under /category/ (e.g., /category/seo, /category/marketing)

*.pdf

All PDF files on the site

/blog/*-ai-*

Blog pages with -ai- in the URL (e.g., /blog/intro-ai-tools, /blog/best-ai-models)

blog.example.com

All pages on the blog subdomain (auto-becomes blog.example.com/*)

blog.example.com/drafts/*

Only /drafts/* on the blog subdomain

⚠️ Patterns * and /* are not allowed as they would exclude your entire site.

When do the exclusion pattern take effect

Already crawled pages are not immediately removed. They will be removed during the next crawl cycle.

When you delete a pattern, pages that no longer match the exclusion rule will be re-included on the next crawl cycle.

Limits

  • Maximum 50 patterns per website

  • Maximum 200 characters per pattern

Did this answer your question?