How to check if an IP is a valid robot or search engine bot?

[georgec]

Hi:
Is there a service or site that let me easily check if an IP belongs to a valid robot or search engine bot, at least the major ones? From time to time I see IPs hitting my server rather hard, though am never sure whether I should ban them out of fear they are legitimate.

Thanks,

[volatilegx]

There are a number of techniques used to identify search engine spiders.

1) Check the registration of the IP address. You can try looking it up at [dnsstuff.com...] . If the IP address is registered to a search engine company, then it may be a search engine spider.

2) Do a reverse DNS lookup. Check to see if any domain names are associated with the IP address. This can also be done on dnsstuff.com. If the domain name is that of a search engine, then it may be a search engine spider.

3) Use Google to do a WebmasterWorld site search for the User Agent. Chances are that we are already talking about it and may have identified it as a spider.

4) There are a number of websites that identify User Agents as search engine spiders. Try a Google search with the User Agent and the words "search engine".

5) Some User Agents will have a URL to a "bot page", which describes the bot. Others will have an email address to which you can send queries.

6) If you haven't found anything at this point, post here with the User Agent, IP address, and any other information you've dug up. Maybe someone here knows about it.

[wilderness]

[webmasterworld.com...]
[psychedelix.com...]
[iplists.com...]
[jafsoft.com...]
[joseluis.pellicer.org...]
[botspot.com...]
[robotstxt.org...]
[siteware.ch...]

Other related items
[spiderhunter.com...]

[georgec]

Thanks guys for the info. That was very helpful.