Googlebot Starange behaviour

Googlebot not crawling

[saha_m]

I am new to this , I found a starange behaviour of googlebot. For last one month i can notice googlebot visit in my logs as follows.

GET /robots.txt HTTP/1.0
GET / HTTP/1.0

Googlebot is not crawling my site after that. What will be the reason.

Please help

[Netizen]

Do you have a robots.txt file and has it changed recently?

It could be that Google is just checking to see if your home page has changed.

[swampy webber]

It's pretty common for Googlebot to check the home page only.

[Essex_boy]

Its BECOMING the norm that Googlebot only goes to your home page, id suggest submiting all your pages via the google submit page.

[MS_Excel]

There is no need to submit all your pages to Google. Set up a basic HTML site map for Google to follow and have a link to it from your home page.

What do you have a robot file? They are only needed to stop Google

[andy_boyd]

Another helpful thing is getting other relevant sites to link to pages deep within your site. Instead of having them link to www.mysite.com, get them to link to www.mysite.com/products/index.html - that'll give Googlebot some food for thought.

[ncw164x]

I had the same on two new sites for nearly two months and then "bingo" started deep spidering, you do start to worry that something is wrong but I did not change anything and googlebot started to spider

One of the sites does not have a site map, I did this as a test to see if this makes any difference on having one or not

If you dont have a robots.txt file then you will see 404 errors in the server error log file so it's always best to have one even if it's empty

ncw164x