Page is a not externally linkable
- Search Engines
-- Sitemaps, Meta Data, and robots.txt
---- Robots.txt disallow: /index.php?


keeper - 2:26 am on Sep 21, 2003 (gmt 0)


Following on from your Google example,
A direct request to:
/mac
Shows this URL has both pagerank and cache info.
However:
/groups
(which is blocked in the robots.txt) has neither.

From this behaviour I would take the leap and say that:
/this.php will be crawled and indexed if your robots.txt has a directive to disallow: /this.php?

At least for Google, not sure how the other engines will take it. I checked the robots documentation as well and couldn't find any specific examples..


Thread source:: http://www.webmasterworld.com/robots_txt/54.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com