i understand this is to direct the crawlers/spiders to only crawl certain pages in your site.
what is the advantage of this..? ium not sure i want to stop any of the pages being crawled.. presumably this depends on quite what content and pages i have on offer.. could someone explain a little.. many thanks
You can block certain spiders altogether that you may not want to crawl your site. (For example, foreign search engines or image spiders - such as Google's image). This can save bandwidth, particularly if you have a lot of pages or large images.