| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Is this done correctly|
| 1:33 pm on Mar 3, 2004 (gmt 0)|
Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?
| 3:00 pm on Mar 3, 2004 (gmt 0)|
Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved