homepage Welcome to WebmasterWorld Guest from 54.237.54.83
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Google only hits robots and index
Google only hits robots and index
rostom




msg:1528437
 4:14 pm on Jul 4, 2005 (gmt 0)

Hello Everyone,
Looking for some help out on this one. I did a search and didn't find postings about this exact problem so I thought I'd post and see if anyone has ideas.

I have a relatively new site (up about 8 months). MSN and Yahoo spiders consistently traverse the site with no problem. But Googlebot seems to be a bit stubborn. It comes about 2-3 times a month and hits the robots.txt page, the home page (/), and is redirected (302) to the location of the actual site (/html). But that is about it--it doesn't go any further. And of course the site doesn't get indexed with Google.

The site resides in a directory called /html (i.e. www.mysite.net/html). I have a temporary redirect (http 302) setup from / to /html.

Any help would be appreciated.
rostom

 

Clint




msg:1528438
 5:00 pm on Jul 4, 2005 (gmt 0)

Why are you using a 302 redirect instead of 301? That might be the problem if there is a problem. FWIW, sometimes the G-bot only hits one of my pages, never all at the same time.

Dijkgraaf




msg:1528439
 2:39 am on Jul 5, 2005 (gmt 0)

Actually a 302 response is correct in this instance, I don't think that is the problem, as my site had that for years without an issue with Googlebot.

Some problems that could cause Googlebot problems are
1) HTML errors, try verifying the HTML of your page adheres to the standards.
2) JavaScript or Flash Navigation. If there aren't any standard URL's for the bots to follow, then it won't find your other pages. You could overcome this by publishing a site map and submitting that to Google.

Reid




msg:1528440
 4:50 am on Jul 6, 2005 (gmt 0)

try a tool called "poodle predictor".

I don't know what affect that would have redirecting / to a deeper folder. googlebot will never stop requesting "/" and that IS the homepage. How you redirect from there (to the same domain?) is up to you.

Clint




msg:1528441
 8:00 am on Jul 10, 2005 (gmt 0)

Actually a 302 response is correct in this instance, I don't think that is the problem, as my site had that for years without an issue with Googlebot.

Dijkgraaf, Please explain now a 302 can be good. That goes against everything I've heard at this forum.

Dijkgraaf




msg:1528442
 11:49 pm on Jul 10, 2005 (gmt 0)

302 is only a problem if it is someone else trying to make Google think that your page is at their URL by them giving a 302 response from their URL.
However for a request for GET / at www.example.com giving a 302 response saying http://www.example.com/index.html should not cause a problem.
What you are in esence saying is, please index the contents of index.html under the URL http://www.example.com/ which is possible exactly what you want.

rostom




msg:1528443
 3:29 pm on Jul 11, 2005 (gmt 0)

In addition to Dijkgraaf comments, I have found that numerous large companies use a 302 for their main page. A few examples are www.dhl-usa.com, www.ibm.com, and www.lnt.com.

But I still can't figure out why Google doesn't spider my site. As I mentioned before, according to the website logs GoogleBot hits my root (/), gets redirected with a 302 to /html, receives a 200 and then just stops. I don't have any javascript on the site and all the links to the rest of the site are available right on the main page.

rostom

Dijkgraaf




msg:1528444
 8:26 pm on Jul 11, 2005 (gmt 0)

Google sometimes just takes a while to warm up to a site, and all you can do is wait paitiently.
The best thing to do is publish a site map and submit it to Google.
https://www.google.com/webmasters/sitemaps/docs/en/about.htm
That might encourage it to do a deeper spidering of your site.

holidayrentals




msg:1528445
 3:30 pm on Jul 15, 2005 (gmt 0)

Rostom,
I was actually looking at the spiders info today. Which Google spider is looking at your site? On my logs there are a few spiders from G. And they 'seem' to be conducting different tasks.

Googlebot only looks at the robots.txt file
Googlebot 2.1 (google.com/bot.html) look at the content
Googlebot 2.1 (googlebot.com/bot.html) looks at the homepage and one specific section.
It goes on with different ids on those names about 10 in total.

They seem to be looking at different things?
Maybe the 'wrong' spider is looking at your site?
Good luck, Adrian

rostom




msg:1528446
 3:51 pm on Jul 15, 2005 (gmt 0)

I don't remember which GoogleBot it was that was hitting my site. And the logs have been rolled over by now, so I can't check.

Regardless, I finally got Google to spider my full site. I built a Sitemap and about 2 days after submitting it to Google its spider came along and traversed my whole site. Thanks for all the help.

rostom

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved