homepage Welcome to WebmasterWorld Guest from 54.204.215.209
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Subscribe to WebmasterWorld

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Can't get Google Webmaster Tools robots.txt file to work.
yetoyeto




msg:4271019
 1:55 pm on Feb 23, 2011 (gmt 0)

Can't get Google Webmaster Tools robots.txt file to work.

I searched but all the threads were 5-6 years old.

Why am I getting the following error under "Google Webmaster Tools> Site Configuration> Crawler Access".

robots.txt file 403 (Forbidden)

The box that shows the text of the robots.txt file is empty.

I have multiple sites/folders under root:/htdocs.

I do NOT have any .htaccess files.

I only have one robots.txt file under root:/htdocs/website_name_folder

The following is what I have in the robots.txt file.

User-agent: *
Disallow: /WordList.html
Disallow: /links.html
Disallow: /ContactUs.html

Would anyone have any idea why I am getting the 403 (Forbidden) message?

Thanks,
yeto

 

tedster




msg:4271205
 8:33 pm on Feb 23, 2011 (gmt 0)

Hard to say. Have you tried the "Fetch as googlebot" utility?

lets_learn




msg:4271211
 8:51 pm on Feb 23, 2011 (gmt 0)

Are you able to access robots.txt through browser?

If not, check file permissions of robots.txt file.

Sgt_Kickaxe




msg:4271267
 12:30 am on Feb 24, 2011 (gmt 0)

You say you have multiple sites/folders under root. The robots.txt needs to be in the folder with the site in it, not in the account root, if you have many sites on one server.

[edited by: Sgt_Kickaxe at 12:36 am (utc) on Feb 24, 2011]

g1smd




msg:4271268
 12:33 am on Feb 24, 2011 (gmt 0)

The only robots.txt file that will be accessed and read for any site "example.com", is the one at the example.com/robots.txt URL.

Verify with a browser that you can read the file at that URL. Each site will have its own robots.txt file, and it will need to be in the right server folder to ensure that it too is accessible at example.com/robots.txt as a URL.

yetoyeto




msg:4271326
 3:56 am on Feb 24, 2011 (gmt 0)

My host said to change file permission from 640 to 644. It worked. Now I have to learn what that actually did. Thanks for all your help.

yeto

tedster




msg:4271329
 4:02 am on Feb 24, 2011 (gmt 0)

Here's the specifics about those permission settings:

640
Owner has Read and Write
Group has Read only
Other has no rights

644
Owner has Read and Write
Group has Read only
Other has Read only
yetoyeto




msg:4271335
 4:19 am on Feb 24, 2011 (gmt 0)

Should my .htaccess file be 644 as well?

Thanks to all for the kind and helpful replies.

Thanks in advance for all the help,
yeto

tedster




msg:4271343
 5:08 am on Feb 24, 2011 (gmt 0)

Here's what I found

chmod your .htaccess and .htpasswd file
chmod your .htpasswd files 640, .htaccess files 644 chmod php files 600,

chmod files that you really dont want people to see as 400 (wp-config.php)
and NEVER chmod 777, if something requires write access use 766 or 775

[askapache.com...]

I'd say take any further technical questions about the Apache server to our Apache Forum [webmasterworld.com]. jdMorgan moderates that forum and knows more about Apache than anyone else I know. Since this is th Google SEO forum, we have people on all kinds of servers and going too deep into Apache is kind of off-topic.

yetoyeto




msg:4271539
 3:47 pm on Feb 24, 2011 (gmt 0)

Oh okay. Thanks to everyone for the kind help.

yeto

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved