Robots.txt vs llms.txt: Different Jobs, and What 47 Real llms.txt Files Contain
Robots.txt can block AI crawlers and llms.txt cannot. I requested llms.txt from 69 well-known sites and graded the 47 real files for format, size and where their links lead, then set that against what Google and Chrome say about the file.