Blocking BacklinkWatch.com

useragent known for this bot?

[cabowabo]

Does anyone know the useragent of this bot? I want to be able to analyze how much it is accessing a client's site. All searches returned nothing.

[not2easy]

Is "BacklinkWatch-com" the UA or referrer? It helps to see the entire UA but because of the ".com" part, it looks like referrer spam.

[cabowabo]

No, that's the site. They are masking their UA under a different name so they won't get blocked. I appreciate the quick response!

[not2easy]

The place to find the UA is in the access logs. Since you said these are all coming from one domain, then you wouldn't need to know the UA, you can do a whois lookup and find the IP and add the server to your IP deny list.

I was trying to understand how you know where it is coming from without checking the access logs which would tell you whether it is actually from that one domain or part of a botnet with referrer spam. That was the reason for asking what I asked. If you block the domain IP or their server IP it won't necessarily stop or prevent seeing the domain in your logs if the domain is not the source of the traffic.

[lucy24]

They are masking their UA under a different name so they won't get blocked.

Are you talking hypothetically, or describing something that has actually happened? If they formerly visited your site, before getting blocked by UA, then all you have to do is block the IP instead. Unless, that is, they're a distributed crawler, which is no fun. But then, a crawler that has neither a distinct IP nor a recognizable UA probably deserves to be blocked regardless.

Sometimes identifying a given site's robot is a no-brainer: plug your URL into their box, make a note of the time, and check your logs for unfamiliar visitors over the next few seconds. (Yet another reason to have a test site, heh heh.) That's not how this particular site works, though.

I gotta say that very few robots change UAs just to get past lockouts; after all, they've no way of knowing what aspect of their request led to the 403. All they see is the response. I tend to associate UA-faking with humans pretending to be robots (including things like site-scraping utilities that give the option of making up a UA). Most of the time they just keep hammering away regardless. How do you know they've been visiting?

[keyplyr]

@cabowabo - why don't you just ask them what their UA is? They might tell you :)

They host at:
virtacore.com
74.204.160.0 - 74.204.191.255
74.204.160.0/19

However they may crawl from different ranges

[Andy Langton]

I doubt they crawl backlinks in general - they probably have a data source. If you mean the crawling they initiate to check anchor text when a user enters a site, the last UA I have is

Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9

No robots.txt, multiple requests/second.

[cabowabo]

I appreciate the responses and I think they are probably getting the data from somewhere else and not doing the direct crawl. Here is the issue. One of my clients only wanted the purest of pure white hat links. He pays a lot for all the research that goes into getting these links. The problem is, he has a competitor that monitors him and piggybacks on these links where he can. While we have blocked the major SEO tools from crawling his site, the smaller ones, like BacklinkWatch, have all of his links listed.

I am sure other high profile clients ask for this service, or would appreciate this type of service. I've exhausted my tool belt of tricks, and was hoping someone had an ace they could slip me under the table. ;-)