homepage Welcome to WebmasterWorld Guest from 50.19.172.0
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Google / Google SEO News and Discussion
Forum Library, Charter, Moderators: Robert Charlton & aakk9999 & brotherhood of lan & goodroi

Google SEO News and Discussion Forum

    
Can't get Google Webmaster Tools robots.txt file to work.
yetoyeto

5+ Year Member



 
Msg#: 4271017 posted 1:55 pm on Feb 23, 2011 (gmt 0)

Can't get Google Webmaster Tools robots.txt file to work.

I searched but all the threads were 5-6 years old.

Why am I getting the following error under "Google Webmaster Tools> Site Configuration> Crawler Access".

robots.txt file 403 (Forbidden)

The box that shows the text of the robots.txt file is empty.

I have multiple sites/folders under root:/htdocs.

I do NOT have any .htaccess files.

I only have one robots.txt file under root:/htdocs/website_name_folder

The following is what I have in the robots.txt file.

User-agent: *
Disallow: /WordList.html
Disallow: /links.html
Disallow: /ContactUs.html

Would anyone have any idea why I am getting the 403 (Forbidden) message?

Thanks,
yeto

 

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4271017 posted 8:33 pm on Feb 23, 2011 (gmt 0)

Hard to say. Have you tried the "Fetch as googlebot" utility?

lets_learn



 
Msg#: 4271017 posted 8:51 pm on Feb 23, 2011 (gmt 0)

Are you able to access robots.txt through browser?

If not, check file permissions of robots.txt file.

Sgt_Kickaxe

WebmasterWorld Senior Member sgt_kickaxe us a WebmasterWorld Top Contributor of All Time



 
Msg#: 4271017 posted 12:30 am on Feb 24, 2011 (gmt 0)

You say you have multiple sites/folders under root. The robots.txt needs to be in the folder with the site in it, not in the account root, if you have many sites on one server.

[edited by: Sgt_Kickaxe at 12:36 am (utc) on Feb 24, 2011]

g1smd

WebmasterWorld Senior Member g1smd us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4271017 posted 12:33 am on Feb 24, 2011 (gmt 0)

The only robots.txt file that will be accessed and read for any site "example.com", is the one at the example.com/robots.txt URL.

Verify with a browser that you can read the file at that URL. Each site will have its own robots.txt file, and it will need to be in the right server folder to ensure that it too is accessible at example.com/robots.txt as a URL.

yetoyeto

5+ Year Member



 
Msg#: 4271017 posted 3:56 am on Feb 24, 2011 (gmt 0)

My host said to change file permission from 640 to 644. It worked. Now I have to learn what that actually did. Thanks for all your help.

yeto

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4271017 posted 4:02 am on Feb 24, 2011 (gmt 0)

Here's the specifics about those permission settings:

640
Owner has Read and Write
Group has Read only
Other has no rights

644
Owner has Read and Write
Group has Read only
Other has Read only
yetoyeto

5+ Year Member



 
Msg#: 4271017 posted 4:19 am on Feb 24, 2011 (gmt 0)

Should my .htaccess file be 644 as well?

Thanks to all for the kind and helpful replies.

Thanks in advance for all the help,
yeto

tedster

WebmasterWorld Senior Member tedster us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 4271017 posted 5:08 am on Feb 24, 2011 (gmt 0)

Here's what I found

chmod your .htaccess and .htpasswd file
chmod your .htpasswd files 640, .htaccess files 644 chmod php files 600,

chmod files that you really dont want people to see as 400 (wp-config.php)
and NEVER chmod 777, if something requires write access use 766 or 775

[askapache.com...]

I'd say take any further technical questions about the Apache server to our Apache Forum [webmasterworld.com]. jdMorgan moderates that forum and knows more about Apache than anyone else I know. Since this is th Google SEO forum, we have people on all kinds of servers and going too deep into Apache is kind of off-topic.

yetoyeto

5+ Year Member



 
Msg#: 4271017 posted 3:47 pm on Feb 24, 2011 (gmt 0)

Oh okay. Thanks to everyone for the kind help.

yeto

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Google / Google SEO News and Discussion
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved