Forum Moderators: open
[google.com...]
[google.com...]
[google.com...]
[help.yahoo.com...]
[help.yahoo.com...]
You may go to most any website and add "/robots.txt" to the domain name and view their file.
You'll see a variety.
These are regular text files.
There is a small software which may be acquired for free which will create a robots.txt for you:
RoboGen
[rietta.com...]
Don