Welcome to WebmasterWorld Guest from 54.158.214.111

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
Where to put an Allow/Deny in httpd.conf
Putting in a list of banned IP's
6 carfac 5:48 pm Jan 24, 2003
top 10 list of useful bots to allow on your site?
instead of a trap how about only the good ones
5 amznVibe 4:27 pm Jan 24, 2003
pavuk/0.9pl28
3 wilderness 8:48 am Jan 24, 2003
Scooter Doing Double Duty?
Scooter hits my site from Spain?
3 carfac 5:07 am Jan 24, 2003
W3CRobot/5.4.0
Does anyone else think this is a fake?
4 Dreamquick 1:23 am Jan 24, 2003
techperf/2.0
didn't see this posted yet
1 hasbeen 5:20 pm Jan 23, 2003
potbot 1.0
Unknown robot from everyone's internet
9 jdMorgan 4:57 pm Jan 23, 2003
IAArchiver-1.0 & robots.txt
Okay how much do I need to pay to take a big stick to the archive.org bot?
4 Dreamquick 1:32 pm Jan 23, 2003
65.102.17.89 just tripped a site trapdoor
this IP appears in a removed message?
15 amznVibe 4:44 am Jan 23, 2003
Lacnic
IP ranges
2 wilderness 11:48 pm Jan 22, 2003
Inktomi Search
158.140.2.102
6 Adam_C 5:12 am Jan 22, 2003
Trapdoors, Spiders, ..I'm Freekin
6 TomJones 5:42 am Jan 21, 2003
Is this Bot new?
potbot 1.0
4 Rugles 11:12 pm Jan 20, 2003
Potbot
2 billdaly 8:56 pm Jan 20, 2003
OpenSRS spider?
OpenSRS bot masked as IE 6.0 browser
1 aleksl 3:45 pm Jan 20, 2003
Spinway
a bot?
2 volatilegx 3:17 pm Jan 20, 2003
road runner: imagescape Robot
2 ZeroCool 7:39 am Jan 19, 2003
Caching Robot
Any reference?
2 frontpage 8:50 pm Jan 18, 2003
PayPal spider, let it in or not?
PayPal "spiderman" does not obey robots.txt
5 amznVibe 2:39 pm Jan 18, 2003
DeepIndex
New European search engine
4 volatilegx 5:32 am Jan 18, 2003
Pompos
Pompos spider hitting hard
1 saoi_jp 2:23 pm Jan 16, 2003
Grabbed 165 pages in 1-1/2 minutes....
3 pendanticist 11:38 am Jan 16, 2003
anyone who or what this is
4 paul_london 12:38 am Jan 16, 2003
CDI Corporation
64.94.199.9
3 wilderness 12:26 am Jan 16, 2003
Fastnet
209.92.205.7
3 wilderness 12:24 am Jan 16, 2003

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members