Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
I have a relatively new site (up about 8 months). MSN and Yahoo spiders consistently traverse the site with no problem. But Googlebot seems to be a bit stubborn. It comes about 2-3 times a month and hits the robots.txt page, the home page (/), and is redirected (302) to the location of the actual site (/html). But that is about it--it doesn't go any further. And of course the site doesn't get indexed with Google.
The site resides in a directory called /html (i.e. www.mysite.net/html). I have a temporary redirect (http 302) setup from / to /html.
Any help would be appreciated.
Some problems that could cause Googlebot problems are
1) HTML errors, try verifying the HTML of your page adheres to the standards.
Googlebot only looks at the robots.txt file
Googlebot 2.1 (google.com/bot.html) look at the content
Googlebot 2.1 (googlebot.com/bot.html) looks at the homepage and one specific section.
It goes on with different ids on those names about 10 in total.
They seem to be looking at different things?
Maybe the 'wrong' spider is looking at your site?
Good luck, Adrian
Regardless, I finally got Google to spider my full site. I built a Sitemap and about 2 days after submitting it to Google its spider came along and traversed my whole site. Thanks for all the help.