homepage Welcome to WebmasterWorld Guest from 54.198.25.229
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Apache Guardian
BoneHeadicus

10+ Year Member



 
Msg#: 185 posted 3:54 am on Nov 25, 2000 (gmt 0)

I have been using this script for a couple of weeks now and it really seems to help identify mischeivious people as well as spiders that are looking for missing files (robots.txt). I put a meta refresh in the header to redirect a human user to the home page.
Everyday it seems I get a visit from some new spider that I would probably miss otherwise. Here's the link
Apache Guardian [xav.com]
Does anybody use a similar, perhaps better system?

 

littleman

WebmasterWorld Senior Member littleman us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 185 posted 8:26 pm on Nov 25, 2000 (gmt 0)

That script is very handy and probably will suite you fine. I do something similar with all my server requests. I have every request logged, good or bad, into a central searchable database. I'm a log junky.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved