Welcome to WebmasterWorld Guest from 54.159.89.7

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Microsoft spiders!
From the bCentral Network
1 Will 3:23 pm Oct 16, 2001
User agent "-" (null)
Are there any legitimate agents that use "-" (null) only?
5 seifried 8:28 pm Oct 15, 2001
anyone seen this one
217.27.131.25
4 dolphin 1:39 pm Oct 15, 2001
Incy Wincy?
Looking for some information on this one...
10 Eric_Lander 9:47 am Oct 15, 2001
Lycos Spider?
Traces to Korea
4 Will 3:39 pm Oct 11, 2001
Indy Library
6 Marcia 3:27 pm Oct 11, 2001
ScoreBot/5
6 Son_House 11:03 pm Oct 10, 2001
New NorthernLight bot
2 littleman 1:44 am Oct 10, 2001
Slurp/2.0-Kiteweekly?
Kiteweekly?
6 grnidone 6:19 pm Oct 9, 2001
Mozilla/3.Mozilla/2.01 (Win95; I)
from nucleus.com
6 idiotgirl 9:18 am Oct 9, 2001
ArchitextSpider
1 littleman 1:23 am Oct 9, 2001
Mozilla/3.0 NAVIO (dtv0)
Verio - Best.com - and Ravenous
2 bobriggs 6:16 am Oct 6, 2001
Sources of SE IP numbers
Anyone know of a frequently updated list of SE IP numbers?
2 Wallace 9:25 pm Oct 5, 2001
compatible;+Indy+Library
211.101.236.93
3 dolphin 11:33 pm Oct 4, 2001
knowledge/0.5
Anyone heard of bot called "knowledge/0.5"?
4 LeBain 9:10 pm Oct 3, 2001
InfoNaviRobot(F107)
InfoNaviRobot(F107)
4 bot_watcher 5:05 pm Oct 3, 2001
larbin2.1.1 - 137.203.128.34
137.203.128.34
15 dolphin 5:26 pm Oct 2, 2001
tracerlock
spidering 2 pages every 15 minutes
4 bobriggs 12:27 am Oct 1, 2001
Spider Language
Do spiders have a specific language associated to them?
4 sweeties 4:13 pm Sep 28, 2001
Spider Language
Do spiders have a specific language associated to them?
1 sweeties 2:56 pm Sep 28, 2001
Does anyone know of where I can get a list of known spiders?
7 IanTurner 6:14 pm Sep 26, 2001
http://www.almaden.ibm.com/cs/crawler/
4 littleman 12:27 pm Sep 26, 2001
More from AltaVista's Scooter-1.3FH
Seems they have more spiders
1 engine 5:35 pm Sep 25, 2001
GulperBot
Open GRiD Project
6 bobriggs 4:47 am Sep 25, 2001
About/0.1libwww-perl/5.47
4 Marcia 4:08 pm Sep 24, 2001