How do I detect good and bad bots here?

check those names

[silverbytes]

bright.net caching robot
Inktomi Slurp
Scooter
Almaden.ibm.com/cs/crawler
Ask Jeeves
DTS Agent
Enterprise Search
Fetch API Request
GaisBot
GigaBot
GirafaBot
GoogleBot
ia_archiver
iaea
Indy
Java1.x.x
libwww-perl
LinkWalker
lwp-trivial
Microsoft URL Control
Mozilla
MSIECrawler
MSNBOT
MSNIA
NetResearchServer
obot
Openfind data gatherer
PHP
Pompos ([e-mail])
sitecheck.internetseer.com
Stupid email harvester
WiseNut bot

Iīm specially worried by LinkWalker, Fetch API Request not to mention the Stupid email harvester

[jatar_k]

Well you can look through this list

A Close to perfect .htaccess ban list [webmasterworld.com]

Though banning is a personal choice there are some that are, more often than not, up to no good.

It is important to read documentation about bots and then observe their behaviour. If they are doing harm to your site then ban them but it is important to make sure that it won't effect users trying to get to your site.

[silverbytes]

Thank you!
Related to bots, and google bot specifically:
I see in my monthly report that googlebot acceded
some resources that are not in my server anymore...

how could that be possible?

/mydocumentthatisnotintheserver.htm is listed as hit, but some other new documents really uploaded donīt...

Any clue?

[mack]

Google does this from time to time because it knows they where there the last time it visited. It will download robots.txt (if present) and if the files are not disallowed it will then attempt to re-index them. Google has no way of knowing the pages are no longer there until your server returns a 404.

Hope this is of some help.

Mack.

[silverbytes]

yes thanks, but why is not crawling my new html documents?... robots txt allows all bots all directories...