| Is this done correctly
|
mikemcs
msg:1526798 | 1:33 pm on Mar 3, 2004 (gmt 0) |
Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages? robots.txt:
User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php Log:
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508
|
closed
msg:1526799 | 3:00 pm on Mar 3, 2004 (gmt 0) |
Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.
|
|
|
