homepage Welcome to WebmasterWorld Guest from 54.227.67.210
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Do i need the txt file?
if i want to allow everyone
markbiz




msg:1525459
 2:13 am on Jul 2, 2004 (gmt 0)

If i want to allow all spiders to my site, do i really need the robots.txt file and robot meta-tags on my site?

 

ogletree




msg:1525460
 2:16 am on Jul 2, 2004 (gmt 0)

No you don't but you might want to put a blank file there just so you don't get so many 404's

Navdeep




msg:1525461
 12:55 am on Jul 5, 2004 (gmt 0)

and ogletree that blank file has to be named "robot.txt".

thanks
Navdeep s

encyclo




msg:1525462
 12:59 am on Jul 5, 2004 (gmt 0)

that blank file has to be named "robot.txt"

A common mistake (or perhaps just a typo). the filename should be robots.txt, and it must be placed in the document root directory.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved