homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

Robots.txt & How your site is spidered
Positives of Robots.txt file aside from disallowing bots

 3:38 pm on Jun 14, 2003 (gmt 0)

I have heard that most people use the Robots.txt to just disallow your specified spiders/bots. And all information relating to the robots.txt file always talk about disallow, ie:
User-agent: grub
Disallow: /

But I wasn't sure how to use it to help your site. Maybe to direct the spiders to specified pages, or even use it somehow as a site-map/link list of all your pages.

What are the different benefits of the robots.txt file aside from banning certain spiders & how would you set it up. Thanks alot for any help, just trying to clear things up a bit :)



 9:56 pm on Jun 14, 2003 (gmt 0)

Hi floridadesigns,

There's not much else you can do with robots.txt except disallow a bot or implicityly allow a bot by not disallowing it.

The following allows googlebot and disallows all others:

User-agent: googlebot

User-agent: *
Disallow: /

You can find all you need to know about robots.txt at robotstxt.org.



 11:15 pm on Jun 14, 2003 (gmt 0)

Ok good. thanks for clearing that up for me.

That is what I always thought, but alot of people & websites are making it seem like its benefits extend much farther. If there is anything else useful I should know.... please let me know :)


 11:31 pm on Jun 14, 2003 (gmt 0)

Robots.txt is useful to disallow specific robots from specific pages/scripts/images, etc. Like your meta-description (in some engines), it gives you additional control over the "presentation" of your site, allows you to prevent visiotrs from entering into your site on some random page, etc.

Disallowing pest 'bots is a secondary function - although it is a much-discussed subject.

A better example robots.txt might be:

User-aqent: Googlebot
User-agent: Slurp
User-agent: Scooter
Disallow: /shopping_cart/
Disallow: /cgi-bin/
Disallow: /mail/e-mailform.html

Ref: A Standard for Robots Exclusion [robotstxt.org]


Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved