Welcome to WebmasterWorld Guest from 54.234.38.8

Forum Moderators: goodroi

Message Too Old, No Replies

Is this done correctly

     
1:33 pm on Mar 3, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Apr 7, 2003
posts:105
votes: 0


Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?

robots.txt:
User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php

Log:
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508

3:00 pm on Mar 3, 2004 (gmt 0)

Full Member

10+ Year Member

joined:Aug 20, 2003
posts:255
votes: 0


Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.