Forum Moderators: open

Message Too Old, No Replies

Bots Coming Into My Site

How do I know if they are crawling?

         

F_Rose

4:34 pm on May 16, 2006 (gmt 0)

10+ Year Member



I see Yahoo, Google, MSN coming into our site, however they are not staying on our site. They just come in for a particular url and then they leave our site.

Is thier something wrong with our site that they are leaving before actually crawling more pages on our site?

treeline

8:38 pm on May 17, 2006 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Keep track of the total collection of pages they look at. Sometimes bots grab a whole slew of pages at once, which can bog down your server. Other times, trying to be polite, they come back over and over snatching a page here, a page there. Generates lots of robots.txt hits.