Welcome to WebmasterWorld Guest from 54.147.158.215

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
Lexibot
Allow or ban
4 mack 11:38 pm Oct 10, 2002
rabaz
Seems well behaved
2 carfac 5:19 pm Oct 10, 2002
ua: ht
3 carfac 4:50 pm Oct 10, 2002
What is contype UA?
5 transistor 3:55 pm Oct 10, 2002
What is Alexa?
3 palmpal 5:45 am Oct 10, 2002
BlogBot
Another one
1 carfac 7:59 pm Oct 7, 2002
YComp 5.x.x.x in so many user agents
what the heck is it?
10 GaryK 11:25 pm Oct 6, 2002
Ywo new- and different- ones spotted today
6 carfac 7:46 pm Oct 6, 2002
Microsoft Internet Explorer/4.40.426
Obvious - yet very strange - fake browser UA
5 Dreamquick 6:10 pm Oct 4, 2002
SlySearch?
22 hanuman 3:21 pm Oct 4, 2002
$botname
Someone loose a variable?
3 carfac 2:18 pm Oct 4, 2002
MARTINI - sysadmin@looksmart.net
Is this a new L$ robot or just someone who likes their liquor?
10 Dreamquick 7:58 am Oct 4, 2002
HTTrack 3.0x
what is this?
5 allybongo 8:48 pm Oct 3, 2002
Whois "-" "-"?
How do I block "-" "-" from my Robots.txt file?
11 jrota 5:35 pm Oct 3, 2002
Need a Bit of a Tutorial...
Trying to figure out what it is...
3 carfac 8:04 pm Oct 2, 2002
Web Download Tool
Something else to ban
12 carfac 6:39 pm Oct 2, 2002
We weirdest UA Ever!
5 carfac 7:23 pm Oct 1, 2002
New Fast IP
3 volatilegx 5:12 pm Oct 1, 2002
Another new one!
Lickity_Split
6 carfac 3:23 am Sep 30, 2002
What is w.x.y.z.nombres.ttd.es
What engine is this a spider from?
3 bnhall 1:21 pm Sep 28, 2002
Scooter/3.2.SF0
2 Scooter24 8:13 pm Sep 27, 2002
www.go2-yellowpages.com
Deep crawling for awhile now, and incorrectly
4 savvy1 8:11 pm Sep 27, 2002
Spiders/Bots and Javascript
Spiders without javascript indexing our "javascript required" page
6 Advantex 8:46 pm Sep 25, 2002
Caught another "Trademark Search" robot
3 carfac 6:44 pm Sep 25, 2002
antibot-V1.1.5/i586-linux-2.2
belongs to AntiSearch
8 volatilegx 5:32 pm Sep 25, 2002

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members