Once your robots.txt file has been created according to requirements and made live, if not already, it’s important to check to make sure it’s valid.

Google has a free tool to do just that, which can be found here:
​https://www.google.com/webmasters/tools/robots-testing-tool

Simply submit a URL to the robots.txt Tester tool, and it will flag up any errors or warnings.