Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Is this done correctly



1:33 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?

User-agent: googlebot
Disallow: /CCS/PFCatalog.php
Disallow: /CCS/Catalog/MyList.php
Disallow: /PCR/Catalog/PFCatalog.php
Disallow: /PCR/Catalog/MyList.php

GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=514
GET /<sitename>.com/CCS/Catalog/Mylist.php?mylistitem=278
GET /<sitetname>.com/CCS/Catalog/Mylist.php?mylistitem=508


3:00 pm on Mar 3, 2004 (gmt 0)

10+ Year Member

Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.

Featured Threads

Hot Threads This Week

Hot Threads This Month