Forum Moderators: open

Message Too Old, No Replies

Googlebot Starange behaviour

Googlebot not crawling

         

saha_m

3:25 pm on Dec 23, 2003 (gmt 0)

10+ Year Member



I am new to this , I found a starange behaviour of googlebot. For last one month i can notice googlebot visit in my logs as follows.

GET /robots.txt HTTP/1.0
GET / HTTP/1.0

Googlebot is not crawling my site after that. What will be the reason.

Please help

Netizen

4:05 pm on Dec 23, 2003 (gmt 0)

10+ Year Member



Do you have a robots.txt file and has it changed recently?

It could be that Google is just checking to see if your home page has changed.

swampy webber

9:38 pm on Dec 23, 2003 (gmt 0)

10+ Year Member



It's pretty common for Googlebot to check the home page only.

Essex_boy

10:58 am on Dec 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



Its BECOMING the norm that Googlebot only goes to your home page, id suggest submiting all your pages via the google submit page.

MS_Excel

11:07 am on Dec 24, 2003 (gmt 0)



There is no need to submit all your pages to Google. Set up a basic HTML site map for Google to follow and have a link to it from your home page.

What do you have a robot file? They are only needed to stop Google

andy_boyd

11:54 am on Dec 24, 2003 (gmt 0)

10+ Year Member



Another helpful thing is getting other relevant sites to link to pages deep within your site. Instead of having them link to www.mysite.com, get them to link to www.mysite.com/products/index.html - that'll give Googlebot some food for thought.

ncw164x

12:29 pm on Dec 24, 2003 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I had the same on two new sites for nearly two months and then "bingo" started deep spidering, you do start to worry that something is wrong but I did not change anything and googlebot started to spider

One of the sites does not have a site map, I did this as a test to see if this makes any difference on having one or not

If you dont have a robots.txt file then you will see 404 errors in the server error log file so it's always best to have one even if it's empty

ncw164x