Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| mp3Spider cn-search-devel at yahoo-inc dot com Yahoo China bot didn't read robots.txt |
3 | GaryK | 4:39 pm June 6, 2006 | ||||
| Kyluka |
10 | fusion5 | 10:16 pm June 5, 2006 | ||||
| Human or Spider? |
2 | besnette | 8:07 pm June 4, 2006 | ||||
| 404s show weird compound URLs This must be a bot but what are they looking for? |
9 | starhugger | 3:07 am May 31, 2006 | ||||
| Yeti |
3 | keyplyr | 5:21 pm May 30, 2006 | ||||
| FatBot+2.0;+www.FatLens.com Who is this guy? |
4 | hvacdirect | 5:07 am May 24, 2006 | ||||
| Favcollector/2.1 Qe? |
3 | Woz | 8:03 pm May 23, 2006 | ||||
| MSNPTC/1.0 on advertising.msn.com MSNPTC/1.0 spider of Microsoft AdCenter |
3 | woland71 | 2:35 pm May 23, 2006 | ||||
| Google bots are visiting like crazy? |
3 | myowndistrict | 6:08 pm May 22, 2006 | ||||
| Google Wireless Transcoder (again) from Google, but does not obey robots.txt |
11 | simplicity | 2:51 pm May 22, 2006 | ||||
| Bots Coming Into My Site How do I know if they are crawling? |
2 | F_Rose | 8:38 pm May 17, 2006 | ||||
| msnbot-media/1.0 .microsoft.com and (MS's) .phx.gbl |
3 | Pfui | 4:07 pm May 17, 2006 | ||||
| hundreds of hits per day on robots.txt from FireFox UAs Appear to be through "planetlab" |
6 | abates | 10:48 pm May 16, 2006 | ||||
| Shopping.com sdcresearchlabs-testbot |
2 | fiestagirl | 5:36 am May 11, 2006 | ||||
| Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 6.0) YES, this is -not- a human visitor |
4 | JAB_Creations | 8:40 pm May 8, 2006 | ||||
| bot/1.0 |
3 | bobothecat | 6:48 pm May 8, 2006 | ||||
| spider or scraper? based on IP, how can you get the REAL STORY? |
12 | arnarn | 4:58 pm May 8, 2006 | ||||
| Mediabot coming through Yahoo! Proxy |
9 | volatilegx | 11:59 pm May 7, 2006 | ||||
| Microsoft-WebDAV - What was this trying to do? |
7 | JAB_Creations | 7:33 am May 7, 2006 | ||||
| Gt::www/1.026 a PHP/Perl class? |
2 | hakre | 4:33 pm May 5, 2006 | ||||
| KakleBot |
2 | bobothecat | 7:24 pm May 4, 2006 | ||||
| Explore XY |
11 | wilderness | 4:16 pm May 4, 2006 | ||||
| OutfoxBot /0.1? Anyone know who what where? |
2 | fusion5 | 3:41 pm May 4, 2006 | ||||
| Tarantula/0.1 |
3 | bumpaw | 8:33 pm May 1, 2006 | ||||
| TAIS Pinger (A SQL thing? An MSN thing?) Looked for (unlinked) referring site's files on my site. Loose w/ privacy. |
3 | Pfui | 6:23 pm Apr 26, 2006 |
