Welcome to WebmasterWorld Guest from 54.227.14.23

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
LinkSweeper/1.1
One of them Warez thangs.
5 pendanticist 9:26 pm Jan 26, 2003
NPBot-1/2.0
crawler1.crawler918.com
3 pendanticist 11:49 am Jan 26, 2003
Newbie Spidering Question
How does that work?
8 lanesharon 5:20 am Jan 26, 2003
BaiDuSpider
29 Steph 3:39 am Jan 26, 2003
UdmSearch 3.0.3
2 delpino 5:37 am Jan 25, 2003
spiders behavior on 301 redirect
6 pardo 5:02 am Jan 25, 2003
Watchfire WebXM 1.0
Anyone Recognize?
6 pendanticist 1:50 am Jan 25, 2003
Newbie QUESTION(s) about bots
What is the typical % of bot traffic for any given site?
2 sys700 7:41 pm Jan 24, 2003
Where to put an Allow/Deny in httpd.conf
Putting in a list of banned IP's
6 carfac 5:48 pm Jan 24, 2003
top 10 list of useful bots to allow on your site?
instead of a trap how about only the good ones
5 amznVibe 4:27 pm Jan 24, 2003
pavuk/0.9pl28
3 wilderness 8:48 am Jan 24, 2003
Scooter Doing Double Duty?
Scooter hits my site from Spain?
3 carfac 5:07 am Jan 24, 2003
W3CRobot/5.4.0
Does anyone else think this is a fake?
4 Dreamquick 1:23 am Jan 24, 2003
techperf/2.0
didn't see this posted yet
1 hasbeen 5:20 pm Jan 23, 2003
potbot 1.0
Unknown robot from everyone's internet
9 jdMorgan 4:57 pm Jan 23, 2003
IAArchiver-1.0 & robots.txt
Okay how much do I need to pay to take a big stick to the archive.org bot?
4 Dreamquick 1:32 pm Jan 23, 2003
65.102.17.89 just tripped a site trapdoor
this IP appears in a removed message?
15 amznVibe 4:44 am Jan 23, 2003
Lacnic
IP ranges
2 wilderness 11:48 pm Jan 22, 2003
Inktomi Search
158.140.2.102
6 Adam_C 5:12 am Jan 22, 2003
Trapdoors, Spiders, ..I'm Freekin
6 TomJones 5:42 am Jan 21, 2003
Is this Bot new?
potbot 1.0
4 Rugles 11:12 pm Jan 20, 2003
Potbot
2 billdaly 8:56 pm Jan 20, 2003
OpenSRS spider?
OpenSRS bot masked as IE 6.0 browser
1 aleksl 3:45 pm Jan 20, 2003
Spinway
a bot?
2 volatilegx 3:17 pm Jan 20, 2003
road runner: imagescape Robot
2 ZeroCool 7:39 am Jan 19, 2003