Welcome to WebmasterWorld Guest from 54.226.62.251

Forum Moderators: goodroi

Message Too Old, No Replies

robots.txt question

     

coolshop

2:15 pm on May 31, 2002 (gmt 0)

10+ Year Member



I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:

User-agent: whistlepig.integer.com
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
Disallow: /cgi-bin/
Disallow: /sds/
Disallow: /640480/
Disallow: /connectors/
Disallow: /java/
Disallow: /tmp/
Disallow: /ddin/

Am I inadvertandly excluding all robots with that first line???? Yipes!

korkus2000

2:21 pm on May 31, 2002 (gmt 0)

WebmasterWorld Senior Member korkus2000 is a WebmasterWorld Top Contributor of All Time 10+ Year Member



You can validate your robots.txt file here.

[searchengineworld.com...]

I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.

here is also a tutorial

[searchengineworld.com...]

 

Featured Threads

Hot Threads This Week

Hot Threads This Month