Welcome to WebmasterWorld Guest from 220.127.116.11
Forum Moderators: goodroi
I guess I'm just irrated that it leaves al the time..wondering if Ican use the robots.txt page as an invite using index/follow instructions
<meta name="robots" content="index,follow">
then create a link to my site map?
99% sure it shouldnt cause any problems.. it's just that 1% uncertainty that keeps nagging at me ..
I'll try it out on one of my other least important sites..
thanks for your help
It's easy. Just point your browser to your favorite website and put in the domain name follwed by /robots.txt like this:
and you'll get their robots.txt file. Edit is and upload it to your root directory next to the INDEX file. Here's a clip from Brett's - he had mentioned last weke he had to exclude unknown bots because they were hiting his site so hard and costing him bandwidth. Not a bad idea IMHO to start with this one...
# WebmasterWorld.com: robots.txt
# GNU Robots.txt Feel free to use with credit given to WebmasterWorld.
# Please, we do NOT allow nonauthorized robots any longer.
# Yes, feel free to copy and use the following.
User-agent: Hatena Antenna
-----truncated by paybacksa