dwilson - 6:42 pm on Dec 11, 2003 (gmt 0)

I just put up the site mentioned in my profile last month. The logs show that spiders from several search engines have arrived. But they've gotten only the first page and robots.txt.

Here is the robots.txt file:
#Robots.txt for www.MyDomain.com
#Email editor@MyDomain.com with any questions.

User-agent: *
Disallow: /images/
Disallow: Purchase.php

I am trying to disallow spiders from indexing the /images folder and the Purchase.php page. Am I doing more than I'm intending?

I do find that when I follow a link at www.MyDomain.com, I end up dropping the "www." and go to MyDomain.com/MyPage.php.

Would that mess up a spider? I'm using relative links ... /MyDirectory/MyPage.php ... throughout.

Thanks for the help!