Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Spider Hunting or Site Building? Goes in waves, do you trust your bot blocker? |
8 | incrediBILL | 9:26 pm June 28, 2009 | ||||
| smart.apnoti.com Robot/v1.34 |
3 | keyplyr | 5:53 pm June 28, 2009 | ||||
| DoubleVerify Crawler |
4 | keyplyr | 9:01 am June 26, 2009 | ||||
| Pinboard (pinboard.cantbedone.org) |
4 | Pfui | 3:34 am June 26, 2009 | ||||
| HolmesBot (http://holmes.ge) |
6 | Pfui | 4:20 pm June 23, 2009 | ||||
| Zanran ZanranCrawler/0.2 (helpdesk@zanran.com) |
3 | Frank_Rizzo | 8:22 am June 22, 2009 | ||||
| WordPress support page bot? |
5 | GaryK | 4:19 am June 22, 2009 | ||||
| msnbot using IE UA should i suspect screenshots? |
3 | GaryK | 11:21 pm June 21, 2009 | ||||
| acont.de's ACONTBOT |
15 | Pfui | 11:04 pm June 21, 2009 | ||||
| Baidu Behaving Badly: Goes undercover w/ cloaked UA; omits robots.txt "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" |
8 | Pfui | 9:47 pm June 21, 2009 | ||||
| Mahalobot/1.0 |
3 | keyplyr | 5:51 pm June 20, 2009 | ||||
| googlebot head request anyone seen googlebot performing head requests? |
16 | enigma1 | 2:18 pm June 18, 2009 | ||||
| MSIndianWebcrawl |
8 | Pfui | 7:11 am June 16, 2009 | ||||
| SpiderMan/1.0 |
5 | Pfui | 4:07 am June 16, 2009 | ||||
| Something new from Yandex? VisualParser |
8 | GaryK | 4:57 am June 14, 2009 | ||||
| AppEngine-Google |
4 | dstiles | 10:24 pm June 13, 2009 | ||||
| Servage Robot |
2 | caribguy | 3:11 pm June 8, 2009 | ||||
| renlifangbot MSN China |
9 | keyplyr | 12:33 am June 5, 2009 | ||||
| FollowSite dot com What is this thing? |
15 | tangor | 12:26 am June 5, 2009 | ||||
| AppEngine-Google: New Google UA to watch for |
14 | GaryK | 8:29 am June 4, 2009 | ||||
| Bing user agent? Does Bing use Msnbot or not? |
5 | ning | 7:40 am June 3, 2009 | ||||
| msnbot/2.0b New crawler from MS in the works?[2] ( 1 2 ) |
41 | GaryK | 6:20 am June 2, 2009 | ||||
| Hunting Google Safebrowsing Diagnostic Spidering Anyone seen anything? |
6 | incrediBILL | 2:56 am May 31, 2009 | ||||
| livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm MSN strikes again |
12 | Pfui | 12:27 am May 31, 2009 | ||||
| Topsy.com / labs.topsy.com / butterfly.topsy.com Butterfly/1.0 *and* libwww-perl |
2 | Pfui | 7:37 pm May 27, 2009 |
