Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

URLs restricted by robots.txt

How to remove that restricton or reinclude URL



12:11 pm on Aug 8, 2008 (gmt 0)

5+ Year Member

In my webmaster tool I have this message

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error"

How to re-include my site/URLs as my site:example.com web site is decreasing the total number of indexed pages day by day I think that one URL removing request does that.

[edited by: goodroi at 1:28 pm (utc) on Aug. 8, 2008]
[edit reason] Please no site specifics [/edit]


2:28 pm on Aug 9, 2008 (gmt 0)

WebmasterWorld Administrator 5+ Year Member Top Contributors Of The Month

In your Webmaster Tools accout you have a diagnostics section. Go there and run tests on your robots.txt file, it will show you what is blocked and needs fixing. The tool lets you try changes right there so you can see if it is fixed.

Featured Threads

Hot Threads This Week

Hot Threads This Month