Welcome to WebmasterWorld Guest from

Forum Moderators: open

Message Too Old, No Replies

list of bots

10:27 am on Mar 9, 2006 (gmt 0)

Preferred Member from GB 

10+ Year Member

joined:Jan 26, 2004
votes: 0

where can i get a list of bot user_agents?
4:10 pm on Mar 9, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Mar 22, 2001
votes: 0

Here's a pretty good one: [jafsoft.com...]

However, if you are cloaking, I recommend doing it by spider IP addresses instead of user agent, or maybe do it as a combination.

6:32 pm on Mar 9, 2006 (gmt 0)

Full Member

10+ Year Member

joined:Jan 4, 2006
votes: 0

The Web Robots Database:-


10:25 pm on Mar 11, 2006 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Nov 5, 2005
votes: 1

I'm always amazed how many people ask for a list of robots / crawlers / spiders because a handy list simply does not exist, neither would one be current for more than 24 hours.

Every single day I see new robots in my logs (...and when I research them, I see new sites with scraped copies of others' lists of robots...boo-hiss). On my sites, the unbridled proliferation of bots and crawlers from every country imaginable is both intriguing, and irritating because it's costly in terms of bandwidth used::value returned. So I block all but a handful.

So anyway, here's a linked list of my most reliable 'robot research' sources. (Many of the URLs are the first of many pages of data, usually listed/linked alphabetically.) These are original sites compiling and offering their own site's data just as they've been doing for years. The webmasters (and their programs) are doing the obsessively hard work and terrifically good work, too. I tip my hat to every single one.

The Best:

List of User-Agents (Spiders, Robots, Crawler, Browser) [psychedelix.com]

The Best of the Rest:

SUMMARY.NET -- Known Robots [summary.net] (live site demo)

Stefan Helbing's Table of bad robots [helbing.nu]

KLOTH.NET -- List of Bad Bots [kloth.net] (alas, not as current as it once was)

John A Fotheringham's Search engine robots that visit your web site [jafsoft.com]

AWM-Webmaster.com -- Browser, Spider, Robots und Crawlers [awm-webmaster.com]

Note: You could go slightly goofy trying to combine the preceding pages' entries in .htaccess, etc., in order to block or otherwise handle the worst one by one. (I know. I tried:) There are simply too danged many of the evil spawn...