| Welcome to WebmasterWorld Guest from 22.214.171.124 |
register, login, search, subscribe, help, library, PubCon, announcements, recent posts, open posts,
|Pubcon Platinum Sponsor|
|I am new to all of this about Spiders...Robot.txt|
| 2:39 am on Jul 11, 2001 (gmt 0)|
I am new to using the robot.txt with my site and though my questions may seem dumb to some of you, I'd greatly appreciate any assistance.
Do I have to list all of the different spiders in the robots.txt file?
Does anyone have a sample of what it should appear as?
What are the most effective approaches towards using robots.txt and spiders?
Thank you all for your assistance in this matter, I do appreciate it.
| 3:17 am on Jul 11, 2001 (gmt 0)|
The only "dumb" question around here is the one that is not asked. Do not ever feel bad about asking a question.
Robots.txt tutorial [searchengineworld.com]
There are some excellent links at the bottom of that page as well.
The only spiders you need to list in your robots.txt file are those you *don't* want to come to your site: like email collectors.
| 9:06 pm on Jul 22, 2001 (gmt 0)|
This is my first posting, could you show me the text i would have to put in a robot.txt to stop e-mail spiders. thanks maddog
Thanks iv found it
| 9:29 pm on Jul 22, 2001 (gmt 0)|
Could you post it please
| 9:43 pm on Jul 22, 2001 (gmt 0)|
I have just had a look for my robot.txt on my site and gould not find it, gould some let me know where i would find this file, Thanks
| 9:44 pm on Jul 22, 2001 (gmt 0)|
It should be in the root like:
| 9:55 pm on Jul 22, 2001 (gmt 0)|
I have had a look at a Robots.txt Syntax Checking
An Error occurred whilst fetching the robots.txt file
This is what i get every day on my log files.
Can you help
| 12:42 am on Jul 23, 2001 (gmt 0)|
I checked the robots.txt file on the website in your profile with this robots.txt validator:
and it said it was correct. Is that the site you are talking about?
All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
WebmasterWorld ® and PubCon ® are a Registered Trademarks of Pubcon Inc.
© Pubcon Inc. 1996-2012 all rights reserved