Home Tags Robots.txt

Tag: robots.txt

Google aims to standardise robots.txt, 25 years on

Google has declared it is open sourcing its robots.txt parser software, in parallel with an effort to standardise […]