homepage Welcome to WebmasterWorld Guest from 54.205.168.88
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Why block a Spider?
Bobby79




msg:398513
 11:58 am on May 17, 2001 (gmt 0)

Why would anyone block spiders, i thought they where good?

 

Hope




msg:398514
 12:40 pm on May 17, 2001 (gmt 0)

There are several reasons you would block a spider. Some spiders take up too much of your bandwidth. Sometimes you don't want spiders in some areas of you site. Your site may only be relevant for one country and you don't want the site listed if search engines of other countries.

wangdy




msg:398515
 12:50 pm on May 21, 2001 (gmt 0)

Hi,
Although this might be off topic... how do you go about creating a spider? Just curious..
Thanks!

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved