Welcome to WebmasterWorld Guest from 54.159.214.27

Forum Moderators: goodroi

Is this done correctly

   
1:33 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?

robots.txt:
User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php

Log:
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508

3:00 pm on Mar 3, 2004 (gmt 0)

10+ Year Member



Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.
 

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month