Welcome to WebmasterWorld Guest from 18.104.22.168
Forum Moderators: goodroi
I am new to using the robot.txt with my site and though my questions may seem dumb to some of you, I'd greatly appreciate any assistance.
Do I have to list all of the different spiders in the robots.txt file?
Does anyone have a sample of what it should appear as?
What are the most effective approaches towards using robots.txt and spiders?
Thank you all for your assistance in this matter, I do appreciate it.
joined:June 27, 2000
Robots.txt tutorial [searchengineworld.com]
There are some excellent links at the bottom of that page as well.
The only spiders you need to list in your robots.txt file are those you *don't* want to come to your site: like email collectors.
An Error occurred whilst fetching the robots.txt file
This is what i get every day on my log files.
Can you help