Welcome to WebmasterWorld Guest from 174.129.151.95

Forum Moderators: goodroi

Message Too Old, No Replies

Do i need the txt file?

if i want to allow everyone

   
2:13 am on Jul 2, 2004 (gmt 0)

10+ Year Member



If i want to allow all spiders to my site, do i really need the robots.txt file and robot meta-tags on my site?
2:16 am on Jul 2, 2004 (gmt 0)

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member



No you don't but you might want to put a blank file there just so you don't get so many 404's
12:55 am on Jul 5, 2004 (gmt 0)

10+ Year Member



and ogletree that blank file has to be named "robot.txt".

thanks
Navdeep s

12:59 am on Jul 5, 2004 (gmt 0)

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member



that blank file has to be named "robot.txt"

A common mistake (or perhaps just a typo). the filename should be robots.txt, and it must be placed in the document root directory.