grnidone
msg:1526676 | 3:17 am on Jul 11, 2001 (gmt 0) |
The only "dumb" question around here is the one that is not asked. Do not ever feel bad about asking a question. Robots.txt tutorial [searchengineworld.com] There are some excellent links at the bottom of that page as well. The only spiders you need to list in your robots.txt file are those you *don't* want to come to your site: like email collectors. -G
|
Cronin
msg:1526677 | 9:06 pm on Jul 22, 2001 (gmt 0) |
Hi all.
This is my first posting, could you show me the text i would have to put in a robot.txt to stop e-mail spiders. thanks maddog
Thanks iv found it
|
highman
msg:1526678 | 9:29 pm on Jul 22, 2001 (gmt 0) |
Could you post it please
|
Cronin
msg:1526679 | 9:43 pm on Jul 22, 2001 (gmt 0) |
I have just had a look for my robot.txt on my site and gould not find it, gould some let me know where i would find this file, Thanks
|
highman
msg:1526680 | 9:44 pm on Jul 22, 2001 (gmt 0) |
It should be in the root like:
[yoursite.com...]
|
Cronin
msg:1526681 | 9:55 pm on Jul 22, 2001 (gmt 0) |
I have had a look at a Robots.txt Syntax Checking
on [tardis.ed.ac.uk...] An Error occurred whilst fetching the robots.txt file Not found This is what i get every day on my log files. Can you help
|
grnidone
msg:1526682 | 12:42 am on Jul 23, 2001 (gmt 0) |
Cronin, I checked the robots.txt file on the website in your profile with this robots.txt validator: [searchengineworld.com...] and it said it was correct. Is that the site you are talking about? -G
|
|
