Forum Moderators: open

Message Too Old, No Replies

Googlebot/2.1 question

         

nalavanje

3:04 am on Jan 22, 2004 (gmt 0)

10+ Year Member



I launched my site 1 week ago. I can't wait for Gogglebot to crawl my site. I found this in my logs yesterday and today. Could somebody explain me what is going on?
Shouldn't I see more logs from googlebot than this?

64.68.82.38 - - [21/Jan/2004:18:41:00 -0800] "GET /robots.txt HTTP/1.0" 200 28 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.82.38 - - [21/Jan/2004:18:41:00 -0800] "GET / HTTP/1.0" 200 18877 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

lazurus

4:29 am on Jan 22, 2004 (gmt 0)



Googlebot works in what appears to be strange ways. It will be back and each time it will grab more or less pages.

gerwin

5:33 am on Jan 22, 2004 (gmt 0)

10+ Year Member



Googlebot i simply getting the robots.txt and you're main site. It will be 1-2 months before you entire site may be crawled and indexed by the bot. But what is you're question?

werty

5:45 am on Jan 22, 2004 (gmt 0)

WebmasterWorld Administrator 10+ Year Member Top Contributors Of The Month



nalavanje, googlebot came and checked your robots.txt file to see if it had permission to crawl your site, it did, then grabbed your index page (/)

Watch google and if you have the toolbar installed watch your grey bar turn into white. You can also do an allinurl:www.yourdomain.com search to see if your page has been 'dropped' into googles index. In the last month I have seen sites do this in 72-96 hours.

Depending on the ammount of incoming link you have going to your site you should see a deep crawl(all of your next level pages crawled) within weeks...I know it sounds kind of fast, but this is what has been happening on my sites(the last 3 we launched were deep crawled and indexed within 160 hours each)

nalavanje

6:03 am on Jan 22, 2004 (gmt 0)

10+ Year Member



Very cool.
Thank you all.