homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

Banning various spiders?

10+ Year Member

Msg#: 501 posted 9:51 pm on Mar 13, 2001 (gmt 0)

Not to get off the subject but I have found that once you become a little more "high profile" thru your positioning efforts you actually can become a "target" by either jealous or mischeivious competitors.

Your next move is to start using .htaccess to block particularly pesty folks from their covert operations. I had one guy who was somehow repeatedly requesting the same page all day long which added up to 5 megs or more a day. That irritated me. So I added the following line to my .htaccess file:

deny from 100.100.100.

No more wasted bandwidth on account of someone elses stupidity.

I found out about this guy by looking thru the reports generated by Weblog. I personally like this script the best and have it set up on a cron to run everyday at the same time. There are about 1,000,000 scripts that do this out there but just pick one you like and use it everyday. Keep up with whats going on in your site.

By eliminating the riff raff...and if you keep up with it you will....you can reserve your resources for the real people. Then when the next threat pops up you'll be right on top of the situation.


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved