homepage Welcome to WebmasterWorld Guest from 54.234.147.84
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
oBot change of User-agent format
another means of blocking oBot in htaccess
privacyman




msg:406544
 11:49 am on Oct 2, 2004 (gmt 0)

Browsing log file of one site that I manage I just discovered that oBot got past my blocking of it in my .htaccess file. Originally I used just the second entry which was written at ^oBot [OR] with other items following, but now I will use two lines as I want to block it starting as User-agent or anywhere in the string thus I have preceeded it with a literal space, and now hope that I will block it whether it uses either format. Log file entry also included herein.

213.252.152.10 - - [28/Sep/2004:03:06:17 -0700] "GET /robots.txt HTTP/1.1" 200 698 "-" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0; obot)"

RewriteCond %{HTTP_USER_AGENT} \ oBot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^oBot [OR]
# ... other entries followed

Hope this helps other people. Note that for the second entry it "could" also have the NC option to catch both upper and lower case.

 

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved