Forum Moderators: open
Bignooz
The other one http*//4webhelp.com/spiders/spidersl.shtml is dead.
Pendanticist.
[robotstxt.org...]
This is, in fact, the same link as that labelled "search engine robots" in an earlier post in this thread.
John
I just told you the other day in usenet that I hadn't used you pages in a while :(
That explains my incorrect link to your gracious effort.
Keep up the good work.
Don
- djr
The Perfect .htaccess [webmasterworld.com] has some valuable information you might wish to peruse.
Pendanticist.
Thanks for the welcome, although I've been lurking around for a few months ;)
I finally implemented the (almost perfect) .htaccess from the thread you mentioned. However, does this mean all bots not mentioned in that list are good bots?
- jp
However, does this mean all bots not mentioned in that list are good bots?
No. It only means that historically, those bots (listed) are known to have performed rather badly by stealing content, bandwidth hogging and etc. New ones pop-up all the time.
Have you ever seen snippet posts where a poster mentions a 'new' bot having scarfed their content and a few other posters chimming in? You can bet that in the background, there is a flurry of other webmasters/admins scurrying to add that particular one to their ban list before it steals their content.
(That list was ammended and tweaked many times to provide a good base from which you can add to, as dictated by your log files.)
Lastly, the range of bots in that list is broad. Some are image thieves that are of particular importance to site owners who have image rich content, while other non-image intensive sites may not be too concerned.
So, the significance of that ban list varies from site-to-site.
>Thanks for the welcome
My pleasure. :)
Pendanticist.
