Welcome to WebmasterWorld Guest from 54.82.10.219

Forum Moderators: Ocean10000 & keyplyr

Message Too Old, No Replies

Blocking BacklinkWatch.com

useragent known for this bot?

     
5:02 pm on Jun 26, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts: 725
votes: 0


Does anyone know the useragent of this bot? I want to be able to analyze how much it is accessing a client's site. All searches returned nothing.
5:19 pm on June 26, 2016 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4085
votes: 257


Is "BacklinkWatch-com" the UA or referrer? It helps to see the entire UA but because of the ".com" part, it looks like referrer spam.
6:25 pm on June 26, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts: 725
votes: 0


No, that's the site. They are masking their UA under a different name so they won't get blocked. I appreciate the quick response!
7:44 pm on June 26, 2016 (gmt 0)

Administrator from US 

WebmasterWorld Administrator not2easy is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Dec 27, 2006
posts:4085
votes: 257


The place to find the UA is in the access logs. Since you said these are all coming from one domain, then you wouldn't need to know the UA, you can do a whois lookup and find the IP and add the server to your IP deny list.

I was trying to understand how you know where it is coming from without checking the access logs which would tell you whether it is actually from that one domain or part of a botnet with referrer spam. That was the reason for asking what I asked. If you block the domain IP or their server IP it won't necessarily stop or prevent seeing the domain in your logs if the domain is not the source of the traffic.
10:41 pm on June 26, 2016 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15313
votes: 708


They are masking their UA under a different name so they won't get blocked.

Are you talking hypothetically, or describing something that has actually happened? If they formerly visited your site, before getting blocked by UA, then all you have to do is block the IP instead. Unless, that is, they're a distributed crawler, which is no fun. But then, a crawler that has neither a distinct IP nor a recognizable UA probably deserves to be blocked regardless.

Sometimes identifying a given site's robot is a no-brainer: plug your URL into their box, make a note of the time, and check your logs for unfamiliar visitors over the next few seconds. (Yet another reason to have a test site, heh heh.) That's not how this particular site works, though.

I gotta say that very few robots change UAs just to get past lockouts; after all, they've no way of knowing what aspect of their request led to the 403. All they see is the response. I tend to associate UA-faking with humans pretending to be robots (including things like site-scraping utilities that give the option of making up a UA). Most of the time they just keep hammering away regardless. How do you know they've been visiting?
1:31 am on June 27, 2016 (gmt 0)

Moderator This Forum from US 

WebmasterWorld Administrator keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 891


@cabowabo - why don't you just ask them what their UA is? They might tell you :)

They host at:
virtacore.com
74.204.160.0 - 74.204.191.255
74.204.160.0/19

However they may crawl from different ranges
11:28 am on June 27, 2016 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member andy_langton is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Jan 27, 2003
posts: 3332
votes: 140


I doubt they crawl backlinks in general - they probably have a data source. If you mean the crawling they initiate to check anchor text when a user enters a site, the last UA I have is
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071025 Firefox/2.0.0.9


No robots.txt, multiple requests/second.
2:44 pm on June 29, 2016 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:June 12, 2003
posts: 725
votes: 0


I appreciate the responses and I think they are probably getting the data from somewhere else and not doing the direct crawl. Here is the issue. One of my clients only wanted the purest of pure white hat links. He pays a lot for all the research that goes into getting these links. The problem is, he has a competitor that monitors him and piggybacks on these links where he can. While we have blocked the major SEO tools from crawling his site, the smaller ones, like BacklinkWatch, have all of his links listed.

I am sure other high profile clients ask for this service, or would appreciate this type of service. I've exhausted my tool belt of tricks, and was hoping someone had an ace they could slip me under the table. ;-)
 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members