homepage Welcome to WebmasterWorld Guest from 23.22.173.58
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Robots.txt not working
samit



 
Msg#: 4069577 posted 5:04 am on Jan 28, 2010 (gmt 0)

Hello Friends,

My client is suffering from a odd problem. He blocked few pages of his site through robots.txt from getting crawled by search engines bots. But those pages are still coming in the google index. Client is not aware whether the pages are crawled before it get blocked through robots or not. But he don't want those pages to be in index.

What can be a possible solution of this problem? Also I would like to know, how a page can be indexed inspite of getting blocked through robots.txt.

Regards
Samit

 

AnkitMaheshwari

5+ Year Member



 
Msg#: 4069577 posted 5:07 am on Jan 28, 2010 (gmt 0)

The webpage might have got indexed before it was blocked by Robots.txt file.

To get the page out of the index, use the URL removal in the Google Webmaster tool under the Crawler Section.

samit



 
Msg#: 4069577 posted 5:38 am on Jan 28, 2010 (gmt 0)

Thanks for the quick responses.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved