Podcast discussion: past, present, and future of robots.txt
engine
11:31 am on Dec 29, 2021 (gmt 0)
The Google Search Team podcast, this issue is a about the past, present and future of robots.txt, and is well worth a listen. "They discuss human initiated crawl requests vs. bot initiated crawl requests, the robots.txt file, and more!"
