| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor 2014|
| 2:15 pm on May 31, 2002 (gmt 0)|
I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:
User-agent: * # directed to all spiders, not just Scooter
Am I inadvertandly excluding all robots with that first line???? Yipes!
| 2:21 pm on May 31, 2002 (gmt 0)|
You can validate your robots.txt file here.
I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.
here is also a tutorial
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved