Welcome to WebmasterWorld Guest from 54.167.174.11

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt question

   
2:15 pm on May 31, 2002 (gmt 0)

10+ Year Member



I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:

User-agent: whistlepig.integer.com
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
Disallow: /cgi-bin/
Disallow: /sds/
Disallow: /640480/
Disallow: /connectors/
Disallow: /java/
Disallow: /tmp/
Disallow: /ddin/

Am I inadvertandly excluding all robots with that first line???? Yipes!

2:21 pm on May 31, 2002 (gmt 0)

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You can validate your robots.txt file here.

[searchengineworld.com...]

I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.

here is also a tutorial

[searchengineworld.com...]