Welcome to WebmasterWorld Guest from 188.8.131.52
Forum Moderators: goodroi
i understand this is to direct the crawlers/spiders to only crawl certain pages in your site.
what is the advantage of this..?
ium not sure i want to stop any of the pages being crawled..
presumably this depends on quite what content and pages i have on offer..
could someone explain a little..