Forum Moderators: open
Well, there are good bots/spiders and there are some that are not so good.
The not so good ones are the ones that pilfer your content by downloading all your pages very quickly, thus causing severe drain on your server. Not to mention the potential theft of your work.
Also, there are bots/spiders like EmailSiphon that will peruse your site looking for the @ for future UCE/SPAMing tactics.
In another post last evening:
[webmasterworld.com...]
entitiled - How 'bout a Spider Trap Tutorial..., I asked for a Tutorial so we could all have a go at hindering these bad bots/spiders effectively and without hovering over our access_log files 24/7. As I understand Spider Traps do just that and in a variety of ways.
<added>Also, type .htaccess into the Site Search for a wealth of supportive information for help in identifying most of the bad bots.</added>
Pendanticist.
I'll check that post. I've been reading up on my robot.txt tutorials. Kinda in overload from all the diferent bots. My host only allows 30 gigs a month so, I'm going to attempt to write a robot.txt that excludes my gifs and flash. Hope I don't scare them all away.
Thanks again!
Get the robots.txt writen. The "good" bots- the ones you WANT (Google, Fast, etc) will obey that, and have no problem with you, or your site, and you will get just what you want.
It is those other 'bot's alluded to above that you need to worry about. Even if you exclude FLASH, they may grab it. Just for informational purposes, you might want to make a trap, if only to see if there are bots going to your site and ignoring your robots.txt!
Good Luck!
Dave
I forgot! Dopn't let ANY of this freak you out! There are plenty of people here who will help you out...
Just take it one day at a time, and you will get there!
dave
