dlvr.it's Feed Fetcher is how dlvr.it retrieves RSS and Atom feeds on behalf of your users. When someone adds a feed to their dlvr.it account, Feed Fetcher periodically checks that feed for new content and keeps it up to date automatically.
This article is aimed at site and server owners who have questions about Feed Fetcher's behavior.
How do I stop dlvr.it Feed Fetcher from accessing my feeds?
You can block Feed Fetcher using your site's robots.txt file.
Note: dlvr.it intentionally ignores the catch-all User-Agent: * section of robots.txt. This is by design — a dlvr.it user explicitly chose to add your feed, and we don't want general search engine rules to accidentally block your readers from accessing it. To block dlvr.it specifically, you need to target the dlvr.it agent directly.
To block a specific feed path:
User-Agent: dlvr.it Disallow: /path/to/your/feeds
To block all content on your site:
User-Agent: dlvr.it Disallow: /
To block most of your site but allow specific feeds:
User-Agent: dlvr.it Disallow: / Allow: /path/to/some/feeds
To allow everything (a plain Disallow with no path means "allow all"):
User-Agent: dlvr.it Disallow:
How often will Feed Fetcher check my feeds?
For most sites, Feed Fetcher checks feeds no more than once per hour on average. Sites that publish content very frequently may be checked more often.
You may occasionally see what looks like more frequent requests in your logs — this is usually due to network delays rather than actual extra fetches.
If your feed advertises a push hub, dlvr.it will subscribe to receive updates as they happen and reduce polling to just once a day. See the push technology question below for more detail.
Why is Feed Fetcher requesting URLs that don't exist on my server?
Feed Fetcher only retrieves feeds that dlvr.it users have explicitly added to their accounts. If you're seeing requests for URLs that don't exist, it's likely that a user entered an incorrect or outdated feed URL. The request is coming from a real user — it just points to a location that no longer exists.
Why is Feed Fetcher accessing an internal or "private" server?
Same reason as above — a dlvr.it user added a feed URL that points to your server. They may know about the server directly, or may have entered the URL by mistake. Feed Fetcher only acts on URLs that users provide.
Why am I seeing requests from multiple dlvr.it machines, all with the same user agent?
dlvr.it Feed Fetcher runs across multiple servers to handle scale and improve performance. All of them identify themselves with the same user agent: dlvr.it/1.0.
Can you give me a list of IP addresses Feed Fetcher uses?
The IP addresses used by Feed Fetcher change over time, so a fixed IP list isn't a reliable way to identify or filter Feed Fetcher traffic. The most reliable identifier is the user agent string: dlvr.it/1.0.
Why is Feed Fetcher downloading the same page more than once?
In most cases, Feed Fetcher only downloads each file once per retrieval cycle. However, if the same feed appears under multiple IDs in your system, it may be fetched more than once. We're continuously improving our deduplication logic to minimize this.
Do you support push technology?
Yes. If your feed advertises a push hub, dlvr.it will subscribe to it and receive content updates as they're published — rather than polling on a schedule. This reduces the number of requests to your server to approximately once per day.
Still have questions? Reach out to our support team — we're happy to help address any feed fetcher questions.