Welcome to WebmasterWorld Guest from 54.167.155.147

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Teoma Spider IP list
where can I find one?
5 Travoli 1:04 pm Jun 14, 2001
Crescent Internet ToolPak HTTP OLE Control v.1.0
Should I ban it in my robots.txt file?
2 merlin78 5:26 am Jun 14, 2001
Inktomi hitting pages over and over
what the heck?
5 Travoli 1:04 pm Jun 13, 2001
AOL UAbut different IP
4 toolman 8:08 pm Jun 12, 2001
Site friendly spiders : the list
4 Brett_Tabke 7:56 pm Jun 12, 2001
A new UA to me
Fast
2 startup 7:54 pm Jun 11, 2001
robot names - please help
5 Ade 6:37 pm Jun 11, 2001
Spider Spammer
Spammin' the UA
3 awoyo 12:47 pm Jun 11, 2001
Anyone know what this is?
larbin_2.1.0 larbin2.1.0@somewhere.com
3 icehousedesigns 6:06 pm Jun 7, 2001
New 404 test from DIIbot
2 jimmykav 6:04 pm Jun 7, 2001
MSIEcrawler
For those of you that didn't know...
3 icehousedesigns 3:35 pm Jun 7, 2001

2 Froggyman 10:24 am Jun 7, 2001
Another one for the experts.
widevine.seattle.winstar.net
3 icehousedesigns 7:40 pm Jun 6, 2001
AOL spider?
ACB998C0.ipt.aol.com
2 icehousedesigns 1:52 pm Jun 6, 2001
Heads Up!
9 toolman 3:08 pm Jun 5, 2001
inktomi2-lut.server.ntl.com
.. Found this one grabbing only one page.
2 Rumbas 2:28 pm Jun 5, 2001
psbot/0.1
picsearch is currently indexing pictures
8 Son_House 12:32 pm Jun 5, 2001
216.35.112.143 is Fast ?
10 berno 3:19 am Jun 5, 2001
Why run a 404 test?
If they're searching for stolen content?
6 mivox 12:33 am Jun 5, 2001
MIIxpc?
10 mark_roach 12:03 am Jun 5, 2001
New Altavista Behaviour...
libwww-perl/5.46 for av?
5 skirril 11:51 pm Jun 4, 2001
Alexa IP's
2 toolman 7:05 pm Jun 4, 2001
Difference in slurps
1 bobriggs 3:49 pm Jun 1, 2001
question
a page with a frame
1 mimmo 8:19 am May 31, 2001
excite
spider?
3 Ove 8:40 pm May 30, 2001