homepage Welcome to WebmasterWorld Guest from 54.167.238.60
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
Forum Library, Charter, Moderators: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification Forum

    
Target Domain in Useragent string
new to me
trintragula




msg:4641912
 2:37 pm on Feb 2, 2014 (gmt 0)

I got visited by this bot:

NetzCheckBot/1.0 (security seal; <<my domain name>>.netzcheck.com; bot@netzcheck.com)

from OVH France (yeah, I know): 5.135.8.81

They registered their domain 2 weeks ago.
Showed up on ProjectHoneyPot last week.
Showed up on BotsVsBrowsers on the 29th.
Visited me yesterday, and were automatically blocked by my filters.

But they put the domain of the site they visit in their UA, which is the only reason I list it here, as that's something out of the ordinary. I have seen other sites do it in the past.
I'm really not into blacklisting, but behaviour that is out of the ordinary (like the above) is probably worth noting.

If nothing else, if you automatically report new UAs anywhere, you might want to watch for your domain name in them - otherwise you might publicly identify yourself as a source of such reports. Newsblur is conspicuously absent from botsvsbrowsers, so I assume BvB are filtering those out.

 

incrediBILL




msg:4641938
 5:44 pm on Feb 2, 2014 (gmt 0)

But they put the domain of the site they visit in their UA, which is the only reason I list it here, as that's something out of the ordinary.


Actually there are a ton of sites out there trying to leverage domain names like the 49ers digging for gold back in the day. I started to put out a list of all of them once upon a time and found them like cockroaches popping up faster than I could list them all.

Some of them either share a database or scrape each other plus it looks like a bunch of them are all the same operator with a pile of different host domains trying to clog the SE so that every time you look for a domain name or certain keywords they choke the list of results.

Damned annoying litter is what it is and I wish Google would kill them all as they're worse than useless, literally internet trash scattered about much like the paper fliers strewn all over sidewalk on The Strip in Vegas.

FWIW, some UAs are crafted to be found in the SE index from sites that don't protect their log analysis pages and those pages end up getting indexed in Google and provide marketing and links for these nitwits.

Ugh.

Makes you feel dirty enough you need a shower after finding it in your log file ;)

lucy24




msg:4641975
 10:23 pm on Feb 2, 2014 (gmt 0)

I don't think it's really got anything to do with the UA. It's more like a variation on referer spam. They want you to go to the named URL and look at their site analysis service. More often it's www.example.com/your-name-here/ ... but if they think wildcard subdomains are more impressive, well, let 'em think so. As long as they stay in OVH where they belong.

trintragula




msg:4642079
 12:31 pm on Feb 3, 2014 (gmt 0)

Interesting:

So putting the target domain name in a UA serves at least 3 distinct purposes:

1. Making the UA less likely to be on a list of block-worthy UAs (because it will only be seen on a unique site)
2. Spamming the recipient (attempting to get them to visit the site to follow up the 'analysis')
3. Acting as a beacon for finding out who reports UAs on public lists.

Dastardly... though probably too rare to be worth blocking automatically.

Every time I've looked at BotsVsBrowsers 'Recent Additions' list recently, there has been UA spam there - different spam from the same sources. Birds nesting on the scarecrow.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Search Engine Spider and User Agent Identification
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved