seonut - 7:52 pm on Jun 4, 2003 (gmt 0)

If I may, since you're on the subject of A robots.txt ask this question? In need of some sort of answers.

If a site has a robots.txt and certain pages has disallow to be indexed and for the pages that does not get index, does the spider just takes the url and index that without meta information?

thanks in advance.

seonut