Welcome to WebmasterWorld Guest from 54.234.63.187

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
This is new...
Ink UK?
2 toolman 9:54 am Jun 19, 2001
Anyone know who this is?
209.67.206.133
3 toolman 8:16 am Jun 17, 2001
MSN what is the useragent
2 zipup 9:02 pm Jun 15, 2001
List your banned IP's / domains and why
7 icehousedesigns 1:06 pm Jun 14, 2001
Teoma Spider IP list
where can I find one?
5 Travoli 1:04 pm Jun 14, 2001
Crescent Internet ToolPak HTTP OLE Control v.1.0
Should I ban it in my robots.txt file?
2 merlin78 5:26 am Jun 14, 2001
Inktomi hitting pages over and over
what the heck?
5 Travoli 1:04 pm Jun 13, 2001
AOL UAbut different IP
4 toolman 8:08 pm Jun 12, 2001
Site friendly spiders : the list
4 Brett_Tabke 7:56 pm Jun 12, 2001
A new UA to me
Fast
2 startup 7:54 pm Jun 11, 2001
robot names - please help
5 Ade 6:37 pm Jun 11, 2001
Spider Spammer
Spammin' the UA
3 awoyo 12:47 pm Jun 11, 2001
Anyone know what this is?
larbin_2.1.0 larbin2.1.0@somewhere.com
3 icehousedesigns 6:06 pm Jun 7, 2001
New 404 test from DIIbot
2 jimmykav 6:04 pm Jun 7, 2001
MSIEcrawler
For those of you that didn't know...
3 icehousedesigns 3:35 pm Jun 7, 2001

2 Froggyman 10:24 am Jun 7, 2001
Another one for the experts.
widevine.seattle.winstar.net
3 icehousedesigns 7:40 pm Jun 6, 2001
AOL spider?
ACB998C0.ipt.aol.com
2 icehousedesigns 1:52 pm Jun 6, 2001
Heads Up!
9 toolman 3:08 pm Jun 5, 2001
inktomi2-lut.server.ntl.com
.. Found this one grabbing only one page.
2 Rumbas 2:28 pm Jun 5, 2001
psbot/0.1
picsearch is currently indexing pictures
8 Son_House 12:32 pm Jun 5, 2001
216.35.112.143 is Fast ?
10 berno 3:19 am Jun 5, 2001
Why run a 404 test?
If they're searching for stolen content?
6 mivox 12:33 am Jun 5, 2001
MIIxpc?
10 mark_roach 12:03 am Jun 5, 2001
New Altavista Behaviour...
libwww-perl/5.46 for av?
5 skirril 11:51 pm Jun 4, 2001

Featured Threads

Hot Threads This Week

Hot Threads This Month