Welcome to WebmasterWorld Guest from 54.163.23.73

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
lwp-trivial?
4 Brett_Tabke 11:36 pm Jun 26, 2001
Hits from bigip1-snat.sv.av.com ( 209.73.164.50)
Altavista new bot ?
4 bfleisch 11:38 pm Jun 25, 2001
Heads up (as toolman would say)
3 Froggyman 6:32 pm Jun 25, 2001
Is this an Alexa spider?
3 toolman 4:08 pm Jun 25, 2001
Disney.com Spidering
lwp-trivial/1.28
2 Froggyman 1:09 pm Jun 25, 2001
Kjjwro
3 jimmykav 12:48 pm Jun 25, 2001
tivra spider
2 roryfrost 12:29 am Jun 25, 2001
Mercator
v1-2jg
3 startup 5:17 am Jun 23, 2001
Spiders and counter stats
3 Outtatheblue 5:05 pm Jun 22, 2001
Atom Indexer v1.1a
217.11.167.34
6 skirril 3:43 pm Jun 19, 2001
This is new...
Ink UK?
2 toolman 9:54 am Jun 19, 2001
Anyone know who this is?
209.67.206.133
3 toolman 8:16 am Jun 17, 2001
MSN what is the useragent
2 zipup 9:02 pm Jun 15, 2001
List your banned IP's / domains and why
7 icehousedesigns 1:06 pm Jun 14, 2001
Teoma Spider IP list
where can I find one?
5 Travoli 1:04 pm Jun 14, 2001
Crescent Internet ToolPak HTTP OLE Control v.1.0
Should I ban it in my robots.txt file?
2 merlin78 5:26 am Jun 14, 2001
Inktomi hitting pages over and over
what the heck?
5 Travoli 1:04 pm Jun 13, 2001
AOL UAbut different IP
4 toolman 8:08 pm Jun 12, 2001
Site friendly spiders : the list
4 Brett_Tabke 7:56 pm Jun 12, 2001
A new UA to me
Fast
2 startup 7:54 pm Jun 11, 2001
robot names - please help
5 Ade 6:37 pm Jun 11, 2001
Spider Spammer
Spammin' the UA
3 awoyo 12:47 pm Jun 11, 2001
Anyone know what this is?
larbin_2.1.0 larbin2.1.0@somewhere.com
3 icehousedesigns 6:06 pm Jun 7, 2001
New 404 test from DIIbot
2 jimmykav 6:04 pm Jun 7, 2001
MSIEcrawler
For those of you that didn't know...
3 icehousedesigns 3:35 pm Jun 7, 2001