Welcome to WebmasterWorld Guest from 54.196.117.47

Forum Moderated by: Ocean10000 & incrediBILL & keyplyr

Search Engine Spider and User Agent Identification


PreModerated forum to id search engine spiders

 
Thread SubjectMessagesStarted byLast Message
202.130.245.216 what's it? a new spider?
4 Chris_1977 10:08 pm Mar 19, 2003
Freshbot and Robots.txt
7 DavidT 6:58 pm Mar 19, 2003
Eversion Avenger/37.17 (Chorus/MiX 3.2; 4-bit)
No luck with Google {SIGH} -- HELP :)
8 xxX_RIPP_Xxx 7:10 am Mar 19, 2003
Some quaint and unique (?) UAs *lol*
17 thermoman 5:13 am Mar 19, 2003
Mozilla/3.0 (compatible)
NOT Mozilla/3.0 (compatible;)
3 pendanticist 11:39 pm Mar 18, 2003
My Spider report showing bots getting error and leaving
Unknown error is keeping bots out of my site.
12 juniperwasting 9:18 pm Mar 18, 2003
could "netscape 5" be a spider?
4 BillM 7:02 am Mar 18, 2003
How to Ban Bots which come without http_user_agent info?
It's real problem, I don't know how to set it up with rewrite mod
11 Chris_1977 2:52 am Mar 18, 2003
aol.com crawler = googlebot.com?
9 damiengazier 1:42 am Mar 18, 2003
Lite Bot 0616B
Ver numbers decreasing?
3 MarieC 6:54 pm Mar 17, 2003
216.191.20.26
3 Brett_Tabke 6:23 pm Mar 17, 2003
lachesis spider from Japan (Nara province)
lachesis spider from Japan (Nara province)
5 misslulu 2:10 pm Mar 17, 2003
PlantyNet_WebRobot_V1.9
Has anyone seen this?
2 MarieC 12:26 pm Mar 16, 2003
obot spidered by 5 day old site
3 coffeelover 9:11 am Mar 16, 2003
T-h-u-n-d-e-r-s-t-o-n-e
Anyone know about this one?
11 pendanticist 5:14 am Mar 16, 2003

4 StanTheMan 7:28 pm Mar 15, 2003
Anyone know about UA: Datum/0.1
coming from IPs with a web server
2 acronym 6:40 pm Mar 15, 2003
k2spider
new UA coming out of Glaxo?
2 acronym 7:15 am Mar 15, 2003
psbot/0.1 Anyone else getting hit by it?
picsearch bot
6 Alphawolf 2:17 am Mar 15, 2003
FAST-WebCrawler/3.7/FirstPage
What does it do?
5 oLeon 12:20 am Mar 15, 2003
Spider From China : Dynegy-communication
SPIDER FROM CHINA : Zeus 2.6
2 misslulu 6:38 pm Mar 14, 2003
file vs directory
better optimization
2 tthach2 5:41 am Mar 14, 2003
Turnititbot
google spam reporting.....Nah
5 eljefe3 2:59 am Mar 14, 2003
lachesis spider form Japan (Nara province)
lachesis spider form Japan (Nara province)
4 misslulu 12:58 am Mar 14, 2003
Php/4.0.6
Double Calls for pages, with brief pause in between requests.
3 pendanticist 6:20 am Mar 13, 2003

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members