Welcome to WebmasterWorld Guest from 54.196.244.45

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt question

     
2:15 pm on May 31, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:May 19, 2002
posts:61
votes: 0


I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:

User-agent: whistlepig.integer.com
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
Disallow: /cgi-bin/
Disallow: /sds/
Disallow: /640480/
Disallow: /connectors/
Disallow: /java/
Disallow: /tmp/
Disallow: /ddin/

Am I inadvertandly excluding all robots with that first line???? Yipes!

2:21 pm on May 31, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member

joined:Mar 20, 2002
posts:3732
votes: 0


You can validate your robots.txt file here.

[searchengineworld.com...]

I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.

here is also a tutorial

[searchengineworld.com...]