homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

looks like a business directory/networking bot

 6:37 am on Jul 12, 2008 (gmt 0)

64.13.130.XX - - [11/Jul/2008:12:20:03 -0400] "GET / HTTP/1.1" 301 254 "-" "SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)"
64.13.130.XX - - [11/Jul/2008:12:20:03 -0400] "GET /robots.txt HTTP/1.1" 200 3631 "-" "SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)"

Looks like an upstart out of Silicon Valley looking to be the next Facebook or something. The web site is not very complete, which always makes me a bit uneasy as to their intentions.

I'll try "SpokeSpider/1.0" and see if it obeys robots.txt next time through.



 4:07 am on Aug 13, 2008 (gmt 0)

Pesky bugger.

71.6.45.zzz - - [12/Aug/2008:17:30:51 -0500] "GET /robots.txt HTTP/1.1" 200 4740 "-" "SpokeSpider/1.0 (http://support.spoke.com/webspider/) Mozilla/5.0 (not really)"

Requested robots and home page six times in succession over four seconds.

Spider (like crawler) is taboo!


 2:22 am on Aug 14, 2008 (gmt 0)

I've been so worn out guessing the exact right user agents to ban I've just gone to whitelisting, any more. Besides, most of the oddball and/or new bots don't give a rip about robots.txt, or behave like uncaged baboons, so it's just easier to say no, no, and NO to them. They do nothing for me other than suck bandwidth and fill up my log files with whacked requests. So, it's adios to them. No kiss on the cheek, no Christmas cards.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved