| Welcome to WebmasterWorld Guest from 188.8.131.52 |
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
| 2:15 pm on May 31, 2002 (gmt 0)|
I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:
User-agent: * # directed to all spiders, not just Scooter
Am I inadvertandly excluding all robots with that first line???? Yipes!
| 2:21 pm on May 31, 2002 (gmt 0)|
You can validate your robots.txt file here.
I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.
here is also a tutorial
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
© Webmaster World 1996-2013 all rights reserved