BGumble
msg:1528823 | 3:52 pm on May 7, 2003 (gmt 0) |
Hi James, welcome to webmasterworld! All bots I've seen will still spider even without the file present.
|
Mike12345
msg:1528824 | 3:54 pm on May 7, 2003 (gmt 0) |
[webmasterworld.com...] build a robots.txt to keep the nasties out, and let the nice ones in. :)
|
creative craig
msg:1528825 | 3:54 pm on May 7, 2003 (gmt 0) |
I have put up a file on my site called robots.txt telling all spiders that they can spider, it keeps the 404's down. Craig
|
BGumble
msg:1528826 | 4:03 pm on May 7, 2003 (gmt 0) |
Do you exclude crawlers that produce few results like WiseNut or AltaVista? Or let them through hoping they might one day become relevant. On a large site, extraneous crawlers can chew lots of bandwidth.
|
Mike12345
msg:1528827 | 4:05 pm on May 7, 2003 (gmt 0) |
If its not worth its weight in sales then it doesnt get in. Thats my motto, which is probably about to get rubbished.
|
GaryK
msg:1528828 | 4:18 pm on May 7, 2003 (gmt 0) |
Naw, that's not rubbish. I just think you have to make sure to replace "weight in sales" with a more generalized term that can apply across the board. For me, there are no sales involved in any of my websites so I determine a crawlers worth by how many visitors I wind up getting from the crawler's website. For others it might also take into account how many of those visitors are converted into customers. :)
|
madeonmoon
msg:1528829 | 4:36 pm on May 7, 2003 (gmt 0) |
thanks to all! i think i might create one just in case it could be helpful in the long run. as a benefit, this will decrease 'page not founds'.
|
Mike12345
msg:1528830 | 4:37 pm on May 7, 2003 (gmt 0) |
Very valid point GaryK, formally ammended. Thanks :)
|
|
