homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

URLs restricted by robots.txt
How to remove that restricton or reinclude URL

5+ Year Member

Msg#: 3718446 posted 12:11 pm on Aug 8, 2008 (gmt 0)

In my webmaster tool I have this message

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error"

How to re-include my site/URLs as my site:example.com web site is decreasing the total number of indexed pages day by day I think that one URL removing request does that.

[edited by: goodroi at 1:28 pm (utc) on Aug. 8, 2008]
[edit reason] Please no site specifics [/edit]



WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

Msg#: 3718446 posted 2:28 pm on Aug 9, 2008 (gmt 0)

In your Webmaster Tools accout you have a diagnostics section. Go there and run tests on your robots.txt file, it will show you what is blocked and needs fixing. The tool lets you try changes right there so you can see if it is fixed.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved