homepage Welcome to WebmasterWorld Guest from 54.237.184.242
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Is robot.txt a must for good SEO?
--whats the purpose of robot.txt
redleaf




msg:1526612
 7:07 am on Nov 8, 2004 (gmt 0)

i am sorry if i am asking stupid question(as u can see, i am a total newbie in this!), but can anyone please help? isnt that avoid putting the file that you dont want the robot to crawl onto the host a better approach than applying the robot.txt?
Thanks a lot.

 

submitx




msg:1526613
 7:44 am on Nov 8, 2004 (gmt 0)

It is mostly used to tell crawlers what files not to crawl. It's the best way to keep spiders out.

ncw164x




msg:1526614
 2:50 pm on Nov 8, 2004 (gmt 0)

It's not a must for seo but it is better to have one even if it's a blank file to stop serving 404's to the spiders

The robots.txt file is not intended for controlling access, it main purpose is to disallow access to certain parts of your site and only if the robot visiting obeys the Robots Exclusion Protocol.

It's only the main search engine spiders which normally obey this protocol, others just ignore it and continue to spider your site, it could be an email harvester looking for email addresses or a rogue spider from some unknown source.

The only way to block these is either via a .htaaccess file or with the httpd.conf file if you have root access using apache

For the perfect .htaccess ban list
[webmasterworld.com...]

redleaf




msg:1526615
 12:24 am on Nov 9, 2004 (gmt 0)

ncw164x,
Thanks man, you the man!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved