homepage Welcome to WebmasterWorld Guest from 54.197.130.16
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Visit PubCon.com
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
robots.txt question
coolshop




msg:1528847
 2:15 pm on May 31, 2002 (gmt 0)

I haven't looked at my robots.txt in years, and it suddenly dawned on me that I should check it out:

User-agent: whistlepig.integer.com
Disallow: /
User-agent: * # directed to all spiders, not just Scooter
Disallow: /cgi-bin/
Disallow: /sds/
Disallow: /640480/
Disallow: /connectors/
Disallow: /java/
Disallow: /tmp/
Disallow: /ddin/

Am I inadvertandly excluding all robots with that first line???? Yipes!

 

korkus2000




msg:1528848
 2:21 pm on May 31, 2002 (gmt 0)

You can validate your robots.txt file here.

[searchengineworld.com...]

I think your txt file is ok. You are only disallowing whistlepig.integer.com from all your content.

here is also a tutorial

[searchengineworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved