Welcome to WebmasterWorld Guest from 54.166.117.130

Forum Moderated by: Ocean10000 & incrediBILL

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Browsers
Can you identify these?
9 designfreak 12:56 pm Apr 20, 2001
fed.looksmart.com
Has anyone seen this one?
5 awoyo 1:07 pm Apr 19, 2001
Help
ip address
6 Gerry 5:11 am Apr 19, 2001
Surfairy
another one for you DNS junkies
6 vis 12:21 pm Apr 18, 2001
file not found
4 stanleytk 2:58 am Apr 18, 2001
First-search.com
4 stn24 9:27 am Apr 17, 2001
Cyveillance notorious bandwidth wasters
just got some more ips, new to me at least
3 jeremy_goodrich 9:41 pm Apr 16, 2001
Exclusion of bots from dynamic pages
A flexible alternative to robots.txt
2 Everyman 5:15 pm Apr 16, 2001
Internet Archive and Thunderstone? Who are they
2 Justin 5:09 pm Apr 16, 2001
Any Body?
What is this dpcatalog1.go2net.com
10 drbill 6:11 pm Apr 13, 2001
I have not seen this one before.
6 miles 5:13 pm Apr 13, 2001
Ip: 199.245.188.102
4 double0seven 4:52 am Apr 13, 2001
RPT-HTTPClient/0.3-2
what is this spider doing
5 stanleytk 7:29 pm Apr 12, 2001
Yahoo who? just found this, todays log -- what's up?
2 Marcia 4:23 am Apr 12, 2001
Indy Library
211.101.236.29 Mozilla/3.0 (compatible; Indy Library)
9 Son_House 5:03 pm Apr 11, 2001
007/v7 spider
3 stanleytk 2:03 pm Apr 11, 2001
This? I've never seen this before..
4 BoneHeadicus 6:29 pm Apr 9, 2001
is this a spider or google employee
3 nube 6:27 pm Apr 9, 2001
New Excite Bot?
storm.excite.com
3 Son_House 5:02 pm Apr 9, 2001
Bot or not?
h-206-132-131-163.globalcenter.net
4 Son_House 3:50 am Apr 7, 2001
User Agent: Pizilla++ ver 2.45
IP 216.218.197.2
3 volatilegx 9:29 pm Apr 6, 2001
Spiders
how can I make one / find one (cgi or perl)
5 luckynh 5:34 pm Apr 5, 2001
How can I easily view all incomers?
5 Eric_Lander 1:55 pm Apr 5, 2001
What
2 drbill 5:31 am Apr 5, 2001
Anybody know this one?
User-Agent: Mozilla/5.0 URL-Spider
2 volatilegx 8:17 pm Apr 4, 2001

Featured Threads

My Threads

Hot Threads This Week

Hot Threads This Month