Forum Moderators: goodroi

Message Too Old, No Replies

Is this done correctly

         

mikemcs

1:33 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?

robots.txt:
User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php

Log:
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508

closed

3:00 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.