Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
But I wasn't sure how to use it to help your site. Maybe to direct the spiders to specified pages, or even use it somehow as a site-map/link list of all your pages.
What are the different benefits of the robots.txt file aside from banning certain spiders & how would you set it up. Thanks alot for any help, just trying to clear things up a bit :)
There's not much else you can do with robots.txt except disallow a bot or implicityly allow a bot by not disallowing it.
The following allows googlebot and disallows all others:
You can find all you need to know about robots.txt at robotstxt.org.
Disallowing pest 'bots is a secondary function - although it is a much-discussed subject.
A better example robots.txt might be:
Ref: A Standard for Robots Exclusion [robotstxt.org]