homepage Welcome to WebmasterWorld Guest from 54.166.108.167
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Wotbox Using Questionable Tactics?
Just Wot Are They Doing?
incrediBILL




msg:4473715
 1:06 am on Jul 9, 2012 (gmt 0)

Something from the UK:

81.144.138.34
"Wotbox/2.01 (+http://www.wotbox.com/bot/)"

ROBOTS.TXT: YUPPERS

They claim to crawl from these IPs:

81.144.138.34
81.144.138.40

However, I found some of my tagged content in their results pages that indicates they not only crawled from other IPs but used a stealth user agent to boot.

A couple of the other IPs I encountered were:

83.146.12.115
83.146.13.85
83.146.13.229

I almost thought it was a legit search engine startup until I ran across that garbage.

Oh well, you've been BUSTED!

BOTS BEWARE: if you put your hand in the cookie jar we're stamping it with codes that identify how the content was crawled.

 

keyplyr




msg:4473718
 1:32 am on Jul 9, 2012 (gmt 0)


Thanks. I didn't have it blocked or allowed. Can't remember if I've ever seen it.

dstiles




msg:4473923
 9:52 pm on Jul 9, 2012 (gmt 0)

Slap in the middle of a BT broadband range, albeit a static range, as far as I can tell. Company called ayima according to DNS.

Actually, I have several bad hits plus another small sub-range blocked in 81.144/16 but 34 was the worst at 64 hits in a couple of months. Not sure if the others are "oops" or real bot-ish but highest number of hits on any one IP was 3 so benefit given.

There is actually a fair amount of bad behaviour within the static BT range 81.128/11 - sadly, being in the UK with UK customers, I can't block them all. :(

Nothing showing here in the 83.146/16 range.

g1smd




msg:4473927
 10:26 pm on Jul 9, 2012 (gmt 0)

Ayima are a well known SEO outfit in the UK. Most of their staff are easily reachable on Twitter. Several are often at SES and SMX.

News to me that they have a bot.

slipkid




msg:4502365
 8:06 am on Oct 1, 2012 (gmt 0)

Got hit today.

Robots.txt Yes (Twice!)

81.144.138.34

"Wotbox/2.01 (+http://www.wotbox.com/bot/)"

Spidered entire site from above IP address.

dstiles




msg:4502655
 7:58 pm on Oct 1, 2012 (gmt 0)

Block 81.144.138.32 - 81.144.138.63 - that will fix them until they get another set of IPs.

keyplyr




msg:4502716
 10:19 pm on Oct 1, 2012 (gmt 0)

I allow wotbox.com as I have a lot of pages with high listings, although since I added them to my whitelist I have seen only a trickle of traffic directly from their SERP.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved