Forum Moderators: Robert Charlton & goodroi

Message Too Old, No Replies

Webmaster Tools says 403 errors - other engines can index the site

         

coolme

8:16 pm on Mar 25, 2008 (gmt 0)

10+ Year Member



I submitted my site to Google about three weeks ago to have it indexed, but when I accessed my Webmaster Tools account yesterday I saw a 403 (Forbidden) error code. When I submitted my site to MSN and Yahoo and they were both able to index it very quickly. MSN took less than 24 hours to have it up and running in its listing, and Yahoo took 48 hours. I added my URL to Google and created a sitemap through the Webmaster Tools account. After waiting for three weeks I get 10 “Web Crawl” warnings with the following error codes.

---------------------------------------------------------------------
Indexing ¦ Top search queries »
Index status: No pages from your site are currently included in Google's index. Indexing can take time. You may find it helpful to review our information for webmasters and webmaster guidelines. Loading...

Web crawl errors
HTTP errors 10 Details »
Not found 0 --
URLs not followed 0 --
URLs restricted by robots.txt 0 --
URLs timed out 0 --
Unreachable URLs 0 --
Total: 10

DetailLast Calculated
4xx error

Mar 22, 2008
403 (Forbidden)
---------------------------------------------------------------------

After doing some research I found that these warnings could be caused by my host service provider. I contacted GoDaddy and asked them if their servers were blocking the Google bot from indexing my site, and this is what they had to say: (NOTHING! Just a boilerplate reply.)

Could someone explain me why Google is not able to index my site, but Yahoo and MSN are? I don't have any Flash or dynamic content on my site. I only have 50 pages of text, nothing out of this world. Why am I getting a 403 (Forbidden) error code? What does this really mean? Can I do anything in my Webmaster Tools account that would get this resolved? I resubmitted my sitemap, but I still have the 403 warning message displayed in my account. Has anyone encounter this problem before?

[edited by: tedster at 8:29 pm (utc) on Mar. 25, 2008]

jd01

8:35 pm on Mar 25, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



I would start with the FireFox User-Agent Switcher, create a GoogleBot user-agent, and attempt to access the site.

The correct UA (AFAIK) sting is:
Googlebot/2.1 (+http://www.googlebot.com/bot.html)

If you receive a 403, then I would check the .htaccess or bot-trap script, or anywhere else you might be able to create a forbidden error.

If you do not receive an error, I would wait and see if the issue corrects itself. I received a forbidden error on one of my sites for about an hour the other night, spent 30 minutes dissecting my .htaccess, and then realized it was not me, it was a temporary issue with my host.

It's entirely possible you experienced the same thing, and GoogleBot happened to stop by while there was some sort of temporary issue.

My guess is it's a GoDaddy issue, but a temporary one, not permanent, because they would have serious issues with webmasters if they were banning Google from their servers.

Justin

jd01

9:07 pm on Mar 25, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member



Interesting...

The host I was referring to, and had the issue with was not GoDaddy, but I think I received the forbidden error on the night of the 22nd. (It was the either the 22nd or 23rd). Makes me wonder if there was some type of wide spread issue for a while. My host is on the West Coast, and I think GoDaddy's main server farm is in Arizona, and it seems odd there were 'forbidden errors' going around on hosts which are normally stable.

Justin

coolme

4:56 pm on Mar 26, 2008 (gmt 0)

10+ Year Member



I was also wondering. Could this be a permissions issue within my account? I never change any directory or files permissions to my site, since I opened the account with GoDaddy. Should I have changed any directory permissions to allow Google to crawl my site?