Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
snprtz
8 wilderness 6:07 am Jan 29, 2006
Hotbot-Lycos or Slurp?
Smarter Stats getting confused?
7 roodle 11:12 am Jan 27, 2006
How Differentiate Users from Bots?
Offering different access
7 grahamberends 11:50 pm Jan 26, 2006
Inktomi - 68.142.211.201
2 classifieds 4:57 pm Jan 24, 2006
Anupa
Browser or spider?
2 GaryK 4:54 pm Jan 23, 2006
enoola crawler
2 keyplyr 10:17 pm Jan 21, 2006
List of Spiders to Block
blocking spiders for web analytics reporting
8 chicagoSEO 7:49 pm Jan 20, 2006
Vision Research Lab image spider
no robots
2 bull 4:31 pm Jan 20, 2006
Which Google Bot UA is correct?
I'm seeing two variations on the 2.1 bot
4 GaryK 3:49 pm Jan 20, 2006
Exabot/2.0
5 bumpaw 10:08 am Jan 20, 2006
How do you make sure a new, genuine, spider is not blocked?
3 inbound 9:53 am Jan 12, 2006
TeragramCrawler
4 bobothecat 9:49 am Jan 11, 2006
plinki/0.1
you got plinked! (thats a good thing..)
3 DanA 8:34 pm Jan 10, 2006
Banning APNIC
What would be the reprocutions
5 twist 7:50 am Jan 7, 2006
Vortex/2.2
4 bobothecat 8:10 pm Jan 5, 2006
LapozzBot?
4 keyplyr 8:44 am Dec 28, 2005
New empty site started being visited by spiders
2 sren 3:21 pm Dec 27, 2005
OpenIntelligenceData/1.1
2 bobothecat 3:19 pm Dec 27, 2005
Getting slammed with phony robots
Is it my site or niche? Are others seeing this?
22 larryhatch 7:41 pm Dec 23, 2005
Wavefire
2 keyplyr 5:35 pm Dec 21, 2005
What's wrong with this simple .htaccess file?
Works for one phony spider, not another ..
17 larryhatch 12:57 am Dec 19, 2005
AOL users running bots.
any tips for halting them?
3 davelms 7:34 am Dec 13, 2005
Y!j-bsc/1.0
Yahoo JP indexer?
3 keyplyr 2:38 pm Dec 11, 2005
zerxspid.?
3 mcneely 11:18 pm Dec 8, 2005
Microsoft?
sandcrawler - compatibility testing
4 fiestagirl 7:20 am Dec 5, 2005