Forum Moderators: open

Message Too Old, No Replies

SWEbot

         

tangor

11:03 pm on Jan 31, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Hit hard on section of site, ripped same 200 pages, plus images, 10 times in a row in one visit

UA: (added spaces)
Mozilla/5.0 (compatible; SWEBot/1.0; +http://swebot . net)

IP: 109.123.117.nnn

Robots?: NO

keyplyr

6:42 pm on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Same UA, same IP

robots.txt: yes. obeyed

iamzippy

7:52 pm on Mar 14, 2012 (gmt 0)

10+ Year Member



I have that same bot coming from 159.253.143.nn on 13/Mar/2012:22:36:12 +0100. It couldn't handle a non-www-to-www 301 for robots.txt even after a re-try, so it resorted to a "GET /http HTTP/1.0". WTF? Not the sharpest tool in the box, is it?

dstiles

9:10 pm on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Both server ranges already blocked.

109.123.64/18 is GB UK2
159.253.128/19 is NL softlayer

keyplyr

11:05 pm on Mar 14, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month





dstiles, I think you're ranges are too wide.

Softlayer
159.253.145.128 - 159.253.145.191
159.253.145.128/26

UK2 hosting:
109.123.64.0 - 109.123.64.255
109.123.64.0/24

dstiles

10:03 pm on Mar 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Several DNS maintainers play that trick. They hope we'll miss it - or something. It's worth scrolling down the DNS returned records to see what the full range is: RIPE puts the nominal "full range" at the end, sometimes with separate breakdowns for lesser sub-ranges.

inetnum: 159.253.128.0 - 159.253.159.255
netname: NL-SOFTLAYER-EU-20110921
descr: SoftLayer Dutch Holdings BV
country: NL

route: 109.123.64.0/18
descr: UK2.NET announcement

keyplyr

11:54 pm on Mar 15, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month




Those wide ranges are listed everywhere, but as "country ranges." I prefer to be more surgical.

dstiles

10:24 pm on Mar 16, 2012 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Whatever country they are (and my DNS checks are shown above) they are server ranges and as such get blocked.

UK2 is a known UK server supplier - my own servers are rented from them. The fact they provide individual "names" for the various /24s is irrelevant.

In the Soflayer case, there are at least some /29s assigned to customers for use on server-type systems. This is not unusual: I have an allocation, for example, of /28 on my web server.

At the end of the day, though, it is unusual for such IP ranges to be used for legitimate browsing. At best they provide proxies, many of which are dodgy to say the least (not suggesting the above two companies condone this!).