Welcome to WebmasterWorld Guest from 184.108.40.206
Forum Moderators: goodroi
work in progress
duplicate pages made for different platforms and browsers
Anything that needs to be stored on site you dont want to publish for some reason.
This file can let you disalow specific files or folders to specific well behaving spiders.
If you just want to free your error log file from robots.txt 404, just paste this in a .txt file and leave it at root level.
If a spider cant find my page that is bad, all my visits come from search engines.
Must I have robot.txt now, because I have never used it before and im listed good under another domain name which site is down now.