Welcome to WebmasterWorld Guest from 35.175.120.174

Forum Moderators: Ocean10000

Message Too Old, No Replies

Scopia Crawler

     
9:41 am on Aug 19, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893




UA: "Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)"
Protocol: HTTP/1.0
robots.txt: yes
Host: hetzner.de or hetzner.co.za
144.76.0.0 - 144.76.255.255
144.76.0.0/16

Info page is htpasswd. Tries to crawl like a bot and not a GET script.
4:38 pm on Aug 19, 2015 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
posts: 2067
votes: 2


Appears to be widely-based at Hetzner...

136.243.92.97
Mozilla/5.0 (compatible; Scopia Crawler 1.2; +http://www.scopia.co)

136.243.0.0 - 136.243.255.255 [136.243.0.0/16]

Project Honey Pot [projecthoneypot.org...] says prior UAs include v.1.1 and 1.2, and:

Scopia Crawler 1.0 (+http://www.scopia.co)
7:52 pm on Aug 19, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15943
votes: 890


Wouldn't it be nice if all unwanted robots lived at Hetzner so you never even had to think about them? Got a strong notion the only AfriNIC ranges I've ever had occasion to block are hetzner.za

:: wandering off to refresh memory on what country "sa" is ::
5:52 pm on Aug 20, 2015 (gmt 0)

Senior Member from GB 

WebmasterWorld Senior Member dstiles is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 14, 2008
posts:3289
votes: 19


I read a report today saying AT and DE NICs were not only registering loads of spam and mailicious package (virus) site domains but were finding reasons not to suspend them when complained of. Not sure how that implicates hetzner other than it's DE and hosts untold nasties.

[spamhaus.org...]
10:47 pm on Aug 20, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15943
votes: 890


other than it ... hosts untold nasties

By weird coincidence, it is not 24 hours since I was reminded in a different venue of the one beginning "Other than that, Mrs. Lincoln..."

Incidentally, I did look it up. "sa" is Saudi Arabia. So I need not worry about accidentally blocking someone I'd be perfectly happy to see.
2:38 am on Aug 21, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member keyplyr is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 26, 2001
posts:12913
votes: 893


Poor Lawrence :(
6:05 pm on Aug 21, 2015 (gmt 0)

Senior Member from US 

WebmasterWorld Senior Member lucy24 is a WebmasterWorld Top Contributor of All Time 5+ Year Member Top Contributors Of The Month

joined:Apr 9, 2011
posts:15943
votes: 890


Almost all human browsers' user-agent strings now start with some form of "Mozilla" (4 or 5, depending on browser). So robots use it in hopes that it will bypass UA-based lockouts.