googlebot

Why do I see this in my log file?

[spy007]

Hello
For the past couple of days I have numerous entries like this in my log file,
64.68.84.139 - - [03/Dec/2003:00:06:15 +0000] "GET /robots.txt HTTP/1.0" 200 284 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
64.68.84.139 - - [03/Dec/2003:00:06:15 +0000] "GET / HTTP/1.0" 200 7143 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"
There must be 20 or 30 just like this. Googlebot never requests anything except my robots.txt. Does this mean there is a problem with my robots.txt or is this normal?

[Goober]

Spy007,

What does your robots.txt file say? Is there something in there that is keeping googlebot away?

It's getting your robots.txt file, then stopping.

Goober

[spy007]

Well all I have is,

User-agent: *

And then five files written as
Disallow: /files.html

one directory written as
Disallow: /directory/

and three directory files written as
Disallow: /directory/files.html

[jdMorgan]

> Googlebot never requests anything except my robots.txt

In the second line of the log sample you posted, Googlebot requested and successfully retrieved your home page.

Given time and sufficient incoming links to your site, the 'bot will go deeper.

Jim

[spy007]

Hi Goober & jdMorgan,
Thanks for your replies. The reason I am concerned is that up till now Google has been indexing my whole site and ranking me around the 5th page for my keywords. When googlebot used to visit it would request every page on my site. I recently changed my index.html to index.shtml and put the following in my .htaccess file:

DirectoryIndex index.shtml index.html

I noticed that I seemed to have gotten dropped from google in the weeks after this, but I was hoping that it would just re-index my index.shtml. Now this morning I have about a dozen more requests from googlebot requesting my robots.txt and then leaving right away without visiting any other pages. I just included the following in my .htaccess:

Redirect permanent /index.html [mysite.com...]

Any thoughts on this?