Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt not working


samit - 5:04 am on Jan 28, 2010 (gmt 0)


Hello Friends,

My client is suffering from a odd problem. He blocked few pages of his site through robots.txt from getting crawled by search engines bots. But those pages are still coming in the google index. Client is not aware whether the pages are crawled before it get blocked through robots or not. But he don't want those pages to be in index.

What can be a possible solution of this problem? Also I would like to know, how a page can be indexed inspite of getting blocked through robots.txt.

Regards
Samit


Thread source:: http://www.webmasterworld.com/robots_txt/4069577.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com