Fast Meta Crawler

[TheFlipguy]

This crawler has been visiting my website non stop since more than 24 hours now. The User Agent shows:

FAST MetaWeb Crawler (helpdesk at fastsearch dot com)

It requests one page every other minute so its not really putting any load on my server but should I be worried about it as it has been requesting pages from my website without stopping for more than 24 hours now.

Flip

[GaryK]

Hi Flip. Are the pages it's taking all unique? Or is it crawling the same pages repeatedly? I first saw this ua in June of 2005. Its last visit was August 13, 2006. It's always been respectful of robots.txt and takes pages at a reasonable pace.

[bobsi18]

Hmmm, I'm new to all this, but I've also got this one crawling my site - as far as I can see, they are all unique pages, it's been crawling for about 4 days straight now, sessions usually last over 12 hours, aorund about 18 hours. It seems to be obeying my robots file. Is this a bad thing, the fact that it's been on my site so long? Should I try and kick it off (though the robots file), or just leave it?

TIA
~bobsi18~

[GaryK]

If you have a large site and the bot is being respectful it's not unusual for it to spend a long time on your site. As for whether it's a good thing or not that's always a decision that should be made by each webmaster. If it turns out the bot brings you lots of traffic it's probably worth it. If not then it might be ban-worthy. Again, that's a decision only you can make since you know your site better than anyone.