homepage Welcome to WebmasterWorld Guest from 107.20.34.144
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Blekko how to ban
viggen

10+ Year Member



 
Msg#: 4546109 posted 12:32 pm on Feb 16, 2013 (gmt 0)

..can you guys help, Blekko is killing my site, doesnt obey robots txt, and appears with hundreds of bots at the same time doing all sorts of weird things...

IP range is always 199.87.252.xxx and 199.87.253.xxx

 

wilderness

WebmasterWorld Senior Member wilderness us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4546109 posted 4:06 pm on Feb 16, 2013 (gmt 0)

Their complete range is
199.87.248.0 - 199.87.255.255
CIDR199.87.248.0/21.

1) Do you have an existing and/or functioning htaccess?
(there are none of this forums threads in your user profile).

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4546109 posted 8:00 pm on Feb 16, 2013 (gmt 0)



For Blekkobot|ScoutJet I block:


64.13.128.0 - 64.13.191.255
64.13.128.0/18

199.87.248.0 - 199.87.255.255
199.87.248.0/21

dstiles

WebmasterWorld Senior Member dstiles us a WebmasterWorld Top Contributor of All Time 5+ Year Member



 
Msg#: 4546109 posted 8:08 pm on Feb 16, 2013 (gmt 0)

Never found blekko to be a problem. I allow their bots on IP/User-Agent match.

keyplyr

WebmasterWorld Senior Member keyplyr us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 4546109 posted 12:41 am on Feb 17, 2013 (gmt 0)



dstiles, many of us object to Blekko caching our content (even against NOINDEX directives) then displaying these cached copies to the public. They also sell SEO data of all your pages to anyone who subscribes to their SEO service.

Even after I blocked them I had to send Cease & Desist notifications to get them to remove my property from their servers.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved