Robots.txt Turns 30: Google Highlights Hidden Strengths

Google’s Gary Illyes highlights robots.txt file’s error tolerance and unexpected features as it marks 30 years of aiding web crawling and SEO.