homepage Welcome to WebmasterWorld Guest from 54.227.141.230
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
URLs restricted by robots.txt
How to remove that restricton or reinclude URL
Raheel




msg:3718448
 12:11 pm on Aug 8, 2008 (gmt 0)

In my webmaster tool I have this message

"Google was unable to crawl the URL due to a robots.txt restriction: your robots.txt file might prohibit the Googlebot entirely; it might prohibit access to the directory in which this URL is located; or it might prohibit access to the URL specifically. Often, this is not an error"

How to re-include my site/URLs as my site:example.com web site is decreasing the total number of indexed pages day by day I think that one URL removing request does that.

[edited by: goodroi at 1:28 pm (utc) on Aug. 8, 2008]
[edit reason] Please no site specifics [/edit]

 

not2easy




msg:3719222
 2:28 pm on Aug 9, 2008 (gmt 0)

In your Webmaster Tools accout you have a diagnostics section. Go there and run tests on your robots.txt file, it will show you what is blocked and needs fixing. The tool lets you try changes right there so you can see if it is fixed.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About
© Webmaster World 1996-2014 all rights reserved