| Welcome to WebmasterWorld Guest from 126.96.36.199 |
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
|Pubcon Gold Sponsor 2015! |
|I am new to all of this about Spiders...Robot.txt|
I am new to using the robot.txt with my site and though my questions may seem dumb to some of you, I'd greatly appreciate any assistance.
Do I have to list all of the different spiders in the robots.txt file?
Does anyone have a sample of what it should appear as?
What are the most effective approaches towards using robots.txt and spiders?
Thank you all for your assistance in this matter, I do appreciate it.
The only "dumb" question around here is the one that is not asked. Do not ever feel bad about asking a question.
Robots.txt tutorial [searchengineworld.com]
There are some excellent links at the bottom of that page as well.
The only spiders you need to list in your robots.txt file are those you *don't* want to come to your site: like email collectors.
This is my first posting, could you show me the text i would have to put in a robot.txt to stop e-mail spiders. thanks maddog
Thanks iv found it
Could you post it please
I have just had a look for my robot.txt on my site and gould not find it, gould some let me know where i would find this file, Thanks
It should be in the root like:
I have had a look at a Robots.txt Syntax Checking
An Error occurred whilst fetching the robots.txt file
This is what i get every day on my log files.
Can you help
I checked the robots.txt file on the website in your profile with this robots.txt validator:
and it said it was correct. Is that the site you are talking about?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved