If an AI crawler can't reach your site, that assistant can't cite you, no matter how good your content is. Checking crawler access is one of the first things to fix if visibility is flat.
robots.txt
A file at the root of your domain (yourdomain.com/robots.txt) that tells crawlers which pages they may fetch. Crawlers request it before anything else. Without one, they use their own defaults and you can't point them to your sitemap.
A rule that blocks an AI crawler, such as those used by ChatGPT, Claude, Perplexity or Google, stops that crawler from reading your pages.
llms.txt
An emerging convention: a file at yourdomain.com/llms.txt that tells AI assistants what your site is and which pages matter most.
How Citera helps
Analytics → Organic traffic shows which AI crawlers your robots.txt and llms.txt allow.
The Website audit checks AI crawler access, llms.txt, structured data and sitemaps.
Citera can show your live files, or draft a robots.txt or llms.txt from the pages it crawled.
Placing the files
These files live at your domain root, outside what Citera publishes to, so you add them yourself:
Webflow and Shopify have a robots.txt editor in site settings. Wix generates one you can override under Marketing & SEO.
llms.txt on hosted builders: Webflow, Wix and Shopify don't allow arbitrary root files. Add a redirect from /llms.txt to a hosted copy (on Shopify, a redirect to a file page).
Your own server: put the file in the web root.
Sitemap
Add a Sitemap line to robots.txt that points to the full address of your sitemap (usually yourdomain.com/sitemap.xml, written with https:// in front), and submit the sitemap in Google Search Console under Sitemaps.