llms.txt, llms-full.txt, sitemap.xml, and robots.txt
Each file is checked for unregistered packages, dead hostnames, and hidden content. Takes about 30 seconds. Free for most sites.
Fast, and then careful
Drop in any public website URL. We handle the rest — no plugins, no code, no OAuth dance.
Our crawler maps your site, pulls out the content a human actually sees, and builds all four files to spec — then checks every reference in them.
You get four files plus a severity report on anything inside them that doesn't resolve: a package nobody registered, a domain that lapsed, text hidden from human readers.
What you get
Two get read by every crawler on the web. Two get read by coding agents and RAG pipelines. We build all four — and then we check what's inside them, which is the part nobody else does.
llms.txtA short map of your site for agents. Coding assistants like Cursor and Claude Code fetch this on demand when you point them at your docs - File example:
llms-full.txtThe full text of every page in one document. Built for RAG pipelines and agents that want the whole picture, not just a link list - File example:
sitemap.xmlReal lastmod dates read from your server, not today's date stamped on every URL. This one gets read by every search engine there is - File example:
robots.txtBuilt from what's actually on your site: platform detected, sitemaps confirmed live before we list them, retrieval bots kept separate from training bots - File example: