Welcome to WebmasterWorld Guest from 54.162.141.212

Message Too Old, No Replies

Can't get Google Webmaster Tools robots.txt file to work.

     
1:55 pm on Feb 23, 2011 (gmt 0)

5+ Year Member



Can't get Google Webmaster Tools robots.txt file to work.

I searched but all the threads were 5-6 years old.

Why am I getting the following error under "Google Webmaster Tools> Site Configuration> Crawler Access".

robots.txt file 403 (Forbidden)

The box that shows the text of the robots.txt file is empty.

I have multiple sites/folders under root:/htdocs.

I do NOT have any .htaccess files.

I only have one robots.txt file under root:/htdocs/website_name_folder

The following is what I have in the robots.txt file.

User-agent: *
Disallow: /WordList.html
Disallow: /links.html
Disallow: /ContactUs.html

Would anyone have any idea why I am getting the 403 (Forbidden) message?

Thanks,
yeto
8:33 pm on Feb 23, 2011 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Hard to say. Have you tried the "Fetch as googlebot" utility?
8:51 pm on Feb 23, 2011 (gmt 0)

5+ Year Member



Are you able to access robots.txt through browser?

If not, check file permissions of robots.txt file.
12:30 am on Feb 24, 2011 (gmt 0)

WebmasterWorld Senior Member sgt_kickaxe is a WebmasterWorld Top Contributor of All Time 5+ Year Member



You say you have multiple sites/folders under root. The robots.txt needs to be in the folder with the site in it, not in the account root, if you have many sites on one server.

[edited by: Sgt_Kickaxe at 12:36 am (utc) on Feb 24, 2011]

12:33 am on Feb 24, 2011 (gmt 0)

WebmasterWorld Senior Member g1smd is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



The only robots.txt file that will be accessed and read for any site "example.com", is the one at the example.com/robots.txt URL.

Verify with a browser that you can read the file at that URL. Each site will have its own robots.txt file, and it will need to be in the right server folder to ensure that it too is accessible at example.com/robots.txt as a URL.
3:56 am on Feb 24, 2011 (gmt 0)

5+ Year Member



My host said to change file permission from 640 to 644. It worked. Now I have to learn what that actually did. Thanks for all your help.

yeto
4:02 am on Feb 24, 2011 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here's the specifics about those permission settings:

640
Owner has Read and Write
Group has Read only
Other has no rights

644
Owner has Read and Write
Group has Read only
Other has Read only
4:19 am on Feb 24, 2011 (gmt 0)

5+ Year Member



Should my .htaccess file be 644 as well?

Thanks to all for the kind and helpful replies.

Thanks in advance for all the help,
yeto
5:08 am on Feb 24, 2011 (gmt 0)

WebmasterWorld Senior Member tedster is a WebmasterWorld Top Contributor of All Time 10+ Year Member



Here's what I found

chmod your .htaccess and .htpasswd file
chmod your .htpasswd files 640, .htaccess files 644 chmod php files 600,

chmod files that you really dont want people to see as 400 (wp-config.php)
and NEVER chmod 777, if something requires write access use 766 or 775

[askapache.com...]
I'd say take any further technical questions about the Apache server to our Apache Forum [webmasterworld.com]. jdMorgan moderates that forum and knows more about Apache than anyone else I know. Since this is th Google SEO forum, we have people on all kinds of servers and going too deep into Apache is kind of off-topic.
3:47 pm on Feb 24, 2011 (gmt 0)

5+ Year Member



Oh okay. Thanks to everyone for the kind help.

yeto
 

Featured Threads

Hot Threads This Week

Hot Threads This Month