Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
So my other thought was to disallow every page of the site, but then allow individual instructional/info pages.
Something along the lines of:
Is this feasible? Will this confuse the crawlers? Any help would be greatly appreciated.
[edited by: SwipeTheMagnets at 5:57 pm (utc) on July 1, 2009]
Whenever you make changes to your robots.txt file it is wise to monitor crawler behavior for a few days to make sure everything is fine. One typo can potentially cause nightmares.
you should use the robots.txt checker [google.com] available in your Google Webmaster Tools dashboard.
if i recall correctly, your robots.txt file will exclude everything for all robots since it uses the first match in the order specified.
you might want to try something like this: