| Welcome to WebmasterWorld Guest from 18.104.22.168 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Accredited PayPal World Seller|
|Search Engines doesn't work with my robots.txt file|
| 9:50 pm on Jan 18, 2005 (gmt 0)|
I set up a new robots.txt file 2 days ago.
I put it under [mashoomashoo.com...]
The file looks like this:
The search engines still look at the disallowed directories. For example Googlebot was in:
What is wrong?
Thanks in advance for the help
| 10:38 pm on Jan 18, 2005 (gmt 0)|
This is all you need to disallow any spiders what obey the robots.txt file access to your modules directory and any other sub directories below modules
now that said googlebot has been prying in places it would not normally go, so it could be a glitch with googlebot at this present time.
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved