Forum Moderated by: open

Crawler, Spider, and User Agent ID


Forum to identify search engine spiders and user agents

 
Thread SubjectMessagesStarted byLast Message
nicebot? opinions, please :S
New bot? calls itself 'nicebot'...
2 blueninja 3:04 am Jun 27, 2005
tpiol.com
we were hit this week by spider
2 TomWaits 4:39 am Jun 26, 2005
Anyone familiar with brandimensions?
Visitors from their site
7 aleksl 4:16 am Jun 26, 2005
www.archive.org
6 OZmike 9:00 pm Jun 24, 2005
Msn
4 wilderness 2:02 am Jun 24, 2005
Convera Crawler is Posting links.
10 Jabba 4:46 am Jun 18, 2005
Msnptc/1.0
What is this bot?
4 delphian 12:37 pm Jun 16, 2005
Web Trends - Bots file update?
2 jaime1985 9:14 pm Jun 15, 2005
NationalDirectory-WebSpider
6 wilderness 3:33 am Jun 14, 2005
Slurp is a great name...
It's gone mad and is slurping my robots.txt
2 badass101 4:03 pm Jun 8, 2005
search egine IP
Keyword ranking from ip address
3 Bigdream 3:15 pm Jun 6, 2005
Mediapartners-Google/2.1
Now crawling from Japan?
5 bose 2:47 pm Jun 6, 2005
Is it a google robot?
What is it?
2 zielak 12:02 am Jun 6, 2005
abot/0.1
New name for npbot.
29 fiestagirl 4:25 am Jun 2, 2005
OmniExplorer_Bot/1.09
just seen this one yesterday
2 macrost 9:37 pm Jun 1, 2005
Twiceler
no website
5 bose 5:07 pm Jun 1, 2005
Do scrapers forge HTTP Referer?
Wondering what the consensus is...
4 dataguy 6:53 pm May 31, 2005
BecomeBot - nasty little critter, very aggressive
Crawling too quickly.
5 andye 8:42 am May 25, 2005
googlebot and log files problem
what do you think about these logs?
9 macdar 11:45 pm May 24, 2005
MyCrawler
What is it? ban it?
4 guitaristinus 10:41 pm May 24, 2005
Missigua Locator 1.9 is Baaack
Spidered my entire site.
26 larryhatch 11:46 am May 24, 2005
de
Multiple UA's
5 wilderness 10:27 am May 24, 2005
HelpSupportServices
Lots of references to this UA but nothing definitive
3 GaryK 6:12 pm May 23, 2005
EmeraldShield WebBot
Is this real?
2 jim_w 1:20 pm May 23, 2005
New Yahoo spider?
2 Strange 8:52 pm May 20, 2005