Welcome to WebmasterWorld Guest from 126.96.36.199
Forum Moderators: goodroi
I have a relatively new site (up about 8 months). MSN and Yahoo spiders consistently traverse the site with no problem. But Googlebot seems to be a bit stubborn. It comes about 2-3 times a month and hits the robots.txt page, the home page (/), and is redirected (302) to the location of the actual site (/html). But that is about it--it doesn't go any further. And of course the site doesn't get indexed with Google.
The site resides in a directory called /html (i.e. www.mysite.net/html). I have a temporary redirect (http 302) setup from / to /html.
Any help would be appreciated.
Some problems that could cause Googlebot problems are
1) HTML errors, try verifying the HTML of your page adheres to the standards.
Actually a 302 response is correct in this instance, I don't think that is the problem, as my site had that for years without an issue with Googlebot.
Dijkgraaf, Please explain now a 302 can be good. That goes against everything I've heard at this forum.
Googlebot only looks at the robots.txt file
Googlebot 2.1 (google.com/bot.html) look at the content
Googlebot 2.1 (googlebot.com/bot.html) looks at the homepage and one specific section.
It goes on with different ids on those names about 10 in total.
They seem to be looking at different things?
Maybe the 'wrong' spider is looking at your site?
Good luck, Adrian
Regardless, I finally got Google to spider my full site. I built a Sitemap and about 2 days after submitting it to Google its spider came along and traversed my whole site. Thanks for all the help.