Open Invitation

making webbots/spiders id-ing easier

[transistor]

With previous permission from Littleman and Brett, I want to invite you to visit my Web Robots/Spiders Database [joseluis.pellicer.org].
It has over 450 different User Agents.
You can:
- Search by name and/or type
- Add comments to help identify and catalog User Agents
- Submit new User Agents
- Mark or flag User Agents for future reference
- Generate Analog files: Browinclude, Browexclude, Robotinclude
- Generate Apache files: mod_rewrite ban list and htaccess ban list.

I hope you find this resource useful. :)
Cheers

[jdMorgan]

transistor,

Wow! Looks like a lot of work...

Thanks,
Jim

[brotherhood of LAN]

Nice one transistor :)

I'm hoping to study my raw logs through SQL and a macro/PHP script here and there......I'll know what ones to filter out to find the "real" visitors now!

[idiotgirl]

Looks like Christmas came early. I guess I won't just read a book and go to sleep tonight after all.

Thanks for all your hard work.

[transistor]

Glad you all like it! :)
Please feel free to send me any suggestions to make it more useful: more reports, configuration files (maybe for other web servers), more types, anything.

[volatilegx]

Fabulous!

[frontpage]

Excellent resource. How do you intend to screen comments/posts to verify the posted information?

Suggestion: You need to have multiple verifications on IP addresses/useragents to prevent innocent IP's from being banned by users of your data.

Just a thought.