Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Baidu Behaving Badly: Goes undercover w/ cloaked UA; omits robots.txt "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)" |
8 | Pfui | 9:47 pm June 21, 2009 | ||||
| Mahalobot/1.0 |
3 | keyplyr | 5:51 pm June 20, 2009 | ||||
| googlebot head request anyone seen googlebot performing head requests? |
16 | enigma1 | 2:18 pm June 18, 2009 | ||||
| MSIndianWebcrawl |
8 | Pfui | 7:11 am June 16, 2009 | ||||
| SpiderMan/1.0 |
5 | Pfui | 4:07 am June 16, 2009 | ||||
| Something new from Yandex? VisualParser |
8 | GaryK | 4:57 am June 14, 2009 | ||||
| AppEngine-Google |
4 | dstiles | 10:24 pm June 13, 2009 | ||||
| Servage Robot |
2 | caribguy | 3:11 pm June 8, 2009 | ||||
| renlifangbot MSN China |
9 | keyplyr | 12:33 am June 5, 2009 | ||||
| FollowSite dot com What is this thing? |
15 | tangor | 12:26 am June 5, 2009 | ||||
| AppEngine-Google: New Google UA to watch for |
14 | GaryK | 8:29 am June 4, 2009 | ||||
| Bing user agent? Does Bing use Msnbot or not? |
5 | ning | 7:40 am June 3, 2009 | ||||
| msnbot/2.0b New crawler from MS in the works?[2] ( 1 2 ) |
41 | GaryK | 6:20 am June 2, 2009 | ||||
| Hunting Google Safebrowsing Diagnostic Spidering Anyone seen anything? |
6 | incrediBILL | 2:56 am May 31, 2009 | ||||
| livebot-searchsense/0.1; +http://search.msn.com/msnbot.htm MSN strikes again |
12 | Pfui | 12:27 am May 31, 2009 | ||||
| Topsy.com / labs.topsy.com / butterfly.topsy.com Butterfly/1.0 *and* libwww-perl |
2 | Pfui | 7:37 pm May 27, 2009 | ||||
| How often does googlebot hit a large new website? How many reqs/min? |
12 | DiscoStu | 9:56 pm May 25, 2009 | ||||
| is there a data centre ip list |
7 | topr8 | 4:46 am May 25, 2009 | ||||
| Total Domain Data Corp Just got a total eclipse of the heart... |
7 | caribguy | 4:31 am May 25, 2009 | ||||
| searchers.co.uk crawling from 213.122.174.x |
6 | Hobbs | 4:56 am May 23, 2009 | ||||
| Google IP Information Request |
4 | hugh | 8:46 pm May 21, 2009 | ||||
| flatlandbot/allspark |
13 | wilderness | 8:23 am May 20, 2009 | ||||
| Profane user agents Do you get any/many? |
16 | GaryK | 9:50 am May 17, 2009 | ||||
| New MSN Mobile Crawler UA Launched without fanfare |
12 | Samizdata | 8:54 pm May 12, 2009 | ||||
| ms research robot MSN strikes again. Again. |
3 | Pfui | 8:28 pm May 10, 2009 |
