Forum Moderators: open

Message Too Old, No Replies

IS the deep crawl over with?

or am I getting hit by the fresh crawl?

         

kstprod

2:18 pm on Nov 8, 2002 (gmt 0)

10+ Year Member



Still a bit shady on understanding Google, so bear with me please :)

On the 4th of this month, my new site got deep crawled by googlebot over about a 2 day period, grabbing all of my pages. I was/am VERY happy about that. :)

Now I've noticed that today, she's come back again for my index page so far. Maybe she'll get more, maybe not. But my question is this, is this the Fresh Bot? All it says in my logs is "crawler12.googlebot.com", my logs don't show Google's IP's, so how do I know? I've gotten all these guys from the deep crawl in the beginning of the month, crawler1, 3, 4, 7, 8, 9, but this one from today is crawler12. Any indication of anything?

Can someone help explain all the different ones? I know by previous posts here that you can spot Fresh Bot by IP, but since my logs don't show Google's IP's, is there another way to tell?

Also, what is the process on Google's end, after you get hit by one or the other, or both?

Thanks guys,

Karen

Slade

4:34 pm on Nov 8, 2002 (gmt 0)

10+ Year Member



Deep Crawler's IPs resolve to crawlXX.google.com
Freshbot IPs resolve to crawlerXX.google.com

This is my speculation based on my logs... And I got deep crawled as late as 8am today.

kstprod

4:55 pm on Nov 8, 2002 (gmt 0)

10+ Year Member



Ah-hah! I missed the difference in my logs. It seems that I am getting Fresh Bot visit afterall. Thanks for letting me know!

Will go find more info on FreshBot now :)

Karen

athinktank

7:05 pm on Nov 8, 2002 (gmt 0)

10+ Year Member



My site, PR3, is getting its deep crawl now. So, it's still out there getting data.

carfac

7:07 pm on Nov 8, 2002 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Looking at my logs for the deep crawl... I did not get crawled too deeply... only about 10,000 (or 10%) of my pages got crawled. But, during the same period, about 10% of the googlebot hits were fresh (and 90% deep). I get 100-200 freshbot hits a day...., but more during the deep crawl. weird!

dave