Welcome to WebmasterWorld Guest from 54.158.195.220

Forum Moderators: goodroi

Message Too Old, No Replies

Do i need the txt file?

if i want to allow everyone

     
2:13 am on Jul 2, 2004 (gmt 0)

New User

10+ Year Member

joined:Mar 21, 2003
posts:32
votes: 0


If i want to allow all spiders to my site, do i really need the robots.txt file and robot meta-tags on my site?
2:16 am on July 2, 2004 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member ogletree is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Apr 14, 2003
posts:4269
votes: 21


No you don't but you might want to put a blank file there just so you don't get so many 404's
12:55 am on July 5, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Nov 30, 2003
posts:126
votes: 0


and ogletree that blank file has to be named "robot.txt".

thanks
Navdeep s

12:59 am on July 5, 2004 (gmt 0)

Senior Member from CA 

WebmasterWorld Senior Member encyclo is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Aug 31, 2003
posts:9068
votes: 4


that blank file has to be named "robot.txt"

A common mistake (or perhaps just a typo). the filename should be robots.txt, and it must be placed in the document root directory.