| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Become a Pro Member|
|Is this done correctly|
Below I have included my robots.txt file which is at least 3 weeks old and then some log entries from about 5 minuets ago. Why is google spidering these pages?
Maybe you didn't make your entries case sensitive? In your robots.txt, you have MyList, but the requests are for Mylist.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved