homepage Welcome to WebmasterWorld Guest from 54.211.95.201
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
ScooperBot
Pfui




msg:4355056
 2:03 am on Aug 25, 2011 (gmt 0)

"CustomScoop, a leader in Media Intelligence, delivers customizable media monitoring technology and analysis..."

...and runs a bot that does not heed the robots.txt files it pretends to read:

server41.customscoop.com
ScooperBot www.customscoop.com

08/24 1n:43:30 /robots.txt
08/24 1n:43:31 /
08/24 1n:43:32 /robots.txt

Scoop this:

User-agent: *
Disallow: /

 

Mokita




msg:4355108
 7:08 am on Aug 25, 2011 (gmt 0)

Pfui wrote:
server41.customscoop.com


Seems to be hosted on Rackspace: 64.49.241.205

CustomScoop RACKS-8-1297789565269605 (NET-64-49-241-192-1) 64.49.241.192 - 64.49.241.223
Rackspace Hosting RSPC-NET-3 (NET-64-49-192-0-1) 64.49.192.0 - 64.49.255.255

They also use at least one GoDaddy IP: 72.167.3.1

CIDR 72.167.0.0/22

(Both ranges I already had blocked, from long ago)

Pfui




msg:4357618
 2:46 pm on Sep 1, 2011 (gmt 0)

Same server farm, same double-tap feint:

server34.customscoop.com
ScooperBot www.customscoop.com

09/01 0n:59:57 /robots.txt
09/01 0n:59:58 /302
09/01 0n:59:59 /robots.txt

FWIW

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved