Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: martinibuster
NO. Two days ago it took 194 pages each visit. Yesterday it took 254 pages per visit! (If only humans loved my stuff that much...) I know they have a way to ban the bot from a site, but I'm curious, has anybody else observed this sort of geometric crawling habit of their bot?
From what I recall, linksmanager doesn't obey robots.txt, so you'd best ban it, if you don't want it to crawl your site, by something other than robots.txt.
With that in mind, I'd say the bot isn't too sophisticated - a 'psychobot' may be an appropriate description, though I highly doubt it's smart enough to warrant the classification of it's intelligence by psychology. :)