Welcome to WebmasterWorld Guest from 54.144.107.83

Forum Moderators: goodroi

Message Too Old, No Replies

Robots.txt not working

     
5:04 am on Jan 28, 2010 (gmt 0)

New User

5+ Year Member

joined:Jan 12, 2010
posts:13
votes: 0


Hello Friends,

My client is suffering from a odd problem. He blocked few pages of his site through robots.txt from getting crawled by search engines bots. But those pages are still coming in the google index. Client is not aware whether the pages are crawled before it get blocked through robots or not. But he don't want those pages to be in index.

What can be a possible solution of this problem? Also I would like to know, how a page can be indexed inspite of getting blocked through robots.txt.

Regards
Samit

5:07 am on Jan 28, 2010 (gmt 0)

Preferred Member

5+ Year Member

joined:Mar 29, 2007
posts:592
votes: 0


The webpage might have got indexed before it was blocked by Robots.txt file.

To get the page out of the index, use the URL removal in the Google Webmaster tool under the Crawler Section.

5:38 am on Jan 28, 2010 (gmt 0)

New User

5+ Year Member

joined:Jan 12, 2010
posts:13
votes: 0


Thanks for the quick responses.