homepage Welcome to WebmasterWorld Guest from 54.166.108.167
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Marketing and Biz Dev / Link Development
Forum Library, Charter, Moderators: martinibuster

Link Development Forum

    
Linksmanager bot crawling
Is this thing a pyschobot or what?
steveb




msg:425507
 2:05 pm on Nov 6, 2002 (gmt 0)

I'm getting crawled by a linksmanager bot seven times each day... not six, not eight, but seven. This started about three weeks ago. I must have exchanged links with somebody using this service. The bot started visiting every day... and each day each "visit" crawled more of my site. My site is 150 or so pages. This bot has gone from taking 30 pages a visit to 33 to 35 and so on. A a few days ago it took 150 each visit. I figured it finally found my whole site.

NO. Two days ago it took 194 pages each visit. Yesterday it took 254 pages per visit! (If only humans loved my stuff that much...) I know they have a way to ban the bot from a site, but I'm curious, has anybody else observed this sort of geometric crawling habit of their bot?

 

jeremy goodrich




msg:425508
 5:22 pm on Nov 6, 2002 (gmt 0)

There was a discussion here a while back, including the author of that program.

From what I recall, linksmanager doesn't obey robots.txt, so you'd best ban it, if you don't want it to crawl your site, by something other than robots.txt.

With that in mind, I'd say the bot isn't too sophisticated - a 'psychobot' may be an appropriate description, though I highly doubt it's smart enough to warrant the classification of it's intelligence by psychology. :)

steveb




msg:425509
 5:04 am on Nov 7, 2002 (gmt 0)

I saw the other thread, and at this point I'm not worried about banning it since I got plenty of bandwidth to spare, but I was just curious about this growing hunger for pages it has, and if anyone else has had this phenomenon. I only get my statistics once a day so I'm curious to see how many pages it will gobble this past 24 hours.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Link Development
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved