Webwork - 6:02 pm on Dec 17, 2010 (gmt 0)
Question 1: Here's the robots.txt file I'm currently using. To be perfectly honest I cannot explain the reason(s) behind every exclusion like a pro . . :(
Anyone see anything wrong with this list?
Question 2: Is a robots.txt file, for a Wordpress site, not really all that important for indexing or SEO purposes? I see so many versions of robots.txt, even amongst "the pros". (I also wonder if I'm actually seeing what the bots are seeing or if the pros cloak their robots.txt file.)
Is it more a matter of "Don't be a dumbass by excluding bots from sections/content that OUGHT to be indexed"?