Robots.txt is a text file with instructions for web crawlers or search engine robots that tells them which pages they should and should not crawl.

It's designed to communicate with these bots, guiding them with instructions on which parts of the website are open for indexing and which should be excluded. These instructions are made specific by ‘allowing’ or ‘disallowing’ the behaviour of certain – or all – bots.

By defining specific rules and directives within the robots.txt file, website administrators can control how search engines and other automated tools access and interact with their site's content.