Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Is this done correctly

1:33 pm on Mar 3, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 7, 2003
votes: 0

Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?

User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php

GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508

3:00 pm on Mar 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
votes: 0

Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.