homepage Welcome to WebmasterWorld Guest from 54.205.207.53
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
needed?
is it completely neccisary
natty

10+ Year Member



 
Msg#: 407 posted 1:07 pm on Jun 15, 2004 (gmt 0)

hi all,

i understand this is to direct the crawlers/spiders to only crawl certain pages in your site.

what is the advantage of this..?
ium not sure i want to stop any of the pages being crawled..
presumably this depends on quite what content and pages i have on offer..
could someone explain a little..
many thanks

n

 

KS_Katz

10+ Year Member



 
Msg#: 407 posted 1:23 pm on Jun 15, 2004 (gmt 0)

If you have no pages that you want to exclude, the only thing a robots.txt file will stop is 404 errors from the spiders. (Not a big deal IMO)

PCInk

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 407 posted 1:34 pm on Jun 15, 2004 (gmt 0)

You can block certain spiders altogether that you may not want to crawl your site. (For example, foreign search engines or image spiders - such as Google's image). This can save bandwidth, particularly if you have a lot of pages or large images.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved