What does robots.txt mean?
Marlon Dean Neuhuber avatar
Written by Marlon Dean Neuhuber
Updated over a week ago

The robots.txt defines how a search engine should handle your site. So it contains regulations on which pages should be taken care of, which should be ignored and which bots are allowed to access your page. Additionally, it gives references on where to find the sitemap(s) of your domains.

Therefore, it can be seen as a very important file, with a huge relevance for SEO.

There are two main positions of your page, where the robots instructions can be found.

1. The robots.txt file which can be found on www.YOURDOMAIN.com/robots.txt
โ€‹2. robots instructions in your meta tags of every page HTML.

Please analyze both areas and see if there they are set to "index" or "noindex" and make sure the setting is as its should be.

Here is some general information on this topic: Robots.txt

Here is an interesting link about Common SEO Mistakes and How You Can Fix Them.

Did this answer your question?