Although I use robtos.txt, I've been reading a lot about SEO recently and I haven't heard much about the file.
Is it really necessary for good placement on search engines?
topr8
11:41 am on Oct 31, 2005 (gmt 0)
it has no bearing what so ever that i know of regarding your SERPS
Dijkgraaf
9:00 pm on Oct 31, 2005 (gmt 0)
About the only SEO aspect might be to stop a duplicate page problem (maybe caused by invalid inbound URL's). Apart from that there is nothing you can do in robots.txt to improve your ranking.
makzan
6:57 pm on Nov 1, 2005 (gmt 0)
So is there any point in having the file unless you do want some pages blocked?
Lord Majestic
7:02 pm on Nov 1, 2005 (gmt 0)
Zero length robots.txt would eliminate 404 errors in your log file related to lack of that file.
