Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- WBSearchBot


MxAngel - 1:31 pm on Jan 26, 2012 (gmt 0)


Mozilla/5.0 (compatible; WBSearchBot/1.1; +http://www.warebay.com/bot.html)

85.17.29.107

canonical name: hosted-by.leaseweb.com
addresses: 127.0.0.1

inetnum: 85.17.28.0 - 85.17.30.255
netname: LEASEWEB
descr: LeaseWeb
descr: P.O. Box 93054
descr: 1090BB AMSTERDAM
descr: Netherlands

route: 85.17.0.0/16
descr: LEASEWEB
origin: AS16265
remarks: LeaseWeb

CentralOPS: [centralops.net...]

Dang .. didn't have that one blocked :(

Robots.txt: NO

990 unique URL's in 1 hour.

Requested URL's that don't exist, even tried remote file inclusion in the URL ...

Tried to change the language on the site, a feature not implented. Performed an exotic search too.

About 1 year ago I changend the format of the URL's, from a non HTML extension to a HTML extension and it still did request the "old format" for certain URL's. Almost like it did scrape them from elsewhere ...

Anyway, just like Purebot it gained a lifetime ban from our sites.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4410502.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com