I see Yahoo, Google, MSN coming into our site, however they are not staying on our site. They just come in for a particular url and then they leave our site.
Is thier something wrong with our site that they are leaving before actually crawling more pages on our site?
treeline
8:38 pm on May 17, 2006 (gmt 0)
Keep track of the total collection of pages they look at. Sometimes bots grab a whole slew of pages at once, which can bog down your server. Other times, trying to be polite, they come back over and over snatching a page here, a page there. Generates lots of robots.txt hits.
