Welcome to WebmasterWorld Guest from 174.129.187.112

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Not a Spider...
Just want to know what the request is!
5 carfac 4:10 pm Oct 15, 2002
NutchCrawler
7 volatilegx 1:39 pm Oct 15, 2002
WorQmada/1.0
5 Scooter24 8:45 am Oct 15, 2002
PingALing
ANOTHER "is your website up" bot!
4 carfac 5:26 pm Oct 14, 2002
NG/1.0 update
Spider from France reads, ignores robots.txt
13 jdMorgan 5:08 pm Oct 14, 2002
webcrawler@nospam
4 Ove 3:44 pm Oct 14, 2002
GirafaBot
Who? What? Where?
8 frontpage 10:39 am Oct 14, 2002
URL Spider Pro
7 misosoph 10:58 pm Oct 13, 2002
Interesting visit by AskJeeves
Making a dead page cleanup pass?
5 jdMorgan 7:17 pm Oct 13, 2002
Why is important to know which spiders
are visiting?
7 dauction 3:42 am Oct 13, 2002
Mozilla/2.0
2 Scooter24 12:23 pm Oct 11, 2002
Lexibot
Allow or ban
4 mack 11:38 pm Oct 10, 2002
rabaz
Seems well behaved
2 carfac 5:19 pm Oct 10, 2002
ua: ht
3 carfac 4:50 pm Oct 10, 2002
What is contype UA?
5 transistor 3:55 pm Oct 10, 2002
What is Alexa?
3 palmpal 5:45 am Oct 10, 2002
BlogBot
Another one
1 carfac 7:59 pm Oct 7, 2002
YComp 5.x.x.x in so many user agents
what the heck is it?
10 GaryK 11:25 pm Oct 6, 2002
Ywo new- and different- ones spotted today
6 carfac 7:46 pm Oct 6, 2002
Microsoft Internet Explorer/4.40.426
Obvious - yet very strange - fake browser UA
5 Dreamquick 6:10 pm Oct 4, 2002
SlySearch?
22 hanuman 3:21 pm Oct 4, 2002
$botname
Someone loose a variable?
3 carfac 2:18 pm Oct 4, 2002
MARTINI - sysadmin@looksmart.net
Is this a new L$ robot or just someone who likes their liquor?
10 Dreamquick 7:58 am Oct 4, 2002
HTTrack 3.0x
what is this?
5 allybongo 8:48 pm Oct 3, 2002
Whois "-" "-"?
How do I block "-" "-" from my Robots.txt file?
11 jrota 5:35 pm Oct 3, 2002