Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| MSIE 7/8 on Windows 98 New browsers on old machines. |
3 | dstiles | 10:45 pm Jan 24, 2010 | ||||
| Cuil quietly fixes Twiceler crawler To handle multi-user-agent policies in robots.txt |
3 | jdMorgan | 8:38 pm Jan 23, 2010 | ||||
| .watson.ibm.com -- Crawler now cloaked? |
3 | Pfui | 4:16 am Jan 23, 2010 | ||||
| Seznam Czech bot |
4 | dstiles | 6:27 pm Jan 21, 2010 | ||||
| gamma.ucsd.edu tests multiple bad bots Nmap Scripting Engine, favicon Internet survey research |
4 | Pfui | 5:53 am Jan 21, 2010 | ||||
| What is KKman (bad bot?) Should it be banned? |
3 | Merganser | 2:39 am Jan 21, 2010 | ||||
| If You're Gonna Spoof . At Least Get It Right A Bit of Humor From the Logs |
2 | TheMadScientist | 10:09 pm Jan 19, 2010 | ||||
| Dow Jones Searchbot |
9 | dstiles | 10:08 pm Jan 18, 2010 | ||||
| Spying spiders - do you know (m)any of them |
5 | smallcompany | 7:21 pm Jan 17, 2010 | ||||
| php-openid/2.1.1 (php/hphp) curl/7.19.4 Something new from Facebook comes... |
2 | Pfui | 5:55 pm Jan 17, 2010 | ||||
| Yahoo's cloaked crawler(s) crawler*.dls.srch.kr3.yahoo.com |
3 | Pfui | 11:30 pm Jan 16, 2010 | ||||
| Yahoo blank UA Is it really mobiles? |
6 | dstiles | 3:42 am Jan 16, 2010 | ||||
| Is "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" legit? |
11 | KenB | 2:39 am Jan 12, 2010 | ||||
| Bad bot? It Visits only index 5-12 times per day 122.108.33.nnn |
5 | TrailBum | 12:00 am Jan 12, 2010 | ||||
| Vortex (formerly Scout.Vortex?) |
2 | Pfui | 7:25 pm Jan 11, 2010 | ||||
| CligooRobot/1.0 |
5 | keyplyr | 6:57 pm Jan 11, 2010 | ||||
| ParchedAnalysisAgent (distributed HTTP access/referer spammer) |
5 | Pfui | 10:14 pm Jan 7, 2010 | ||||
| cipherkey.net linked to a wide variety of bad bots DFAgent/DivineFlowersExplorer, FinancialAnalysisAgent, LegalX Deep Analyzer |
4 | Pfui | 10:13 pm Jan 7, 2010 | ||||
| Diigo caching pages in violation of meta tags & cloaking bot |
4 | KenB | 6:18 pm Jan 7, 2010 | ||||
| MJ12bot v1.3.0 Implements Ground Breaking Validation Capability [4] ( 1 2 3 4 ) |
99 | System | 11:49 pm Jan 3, 2010 | ||||
| Orgbybot/OrgbyBot v1.2 |
7 | GaryK | 6:15 am Jan 1, 2010 | ||||
| Linguee Bot |
19 | Pfui | 6:58 pm Dec 31, 2009 | ||||
| aiHitBot |
4 | Pfui | 3:47 am Dec 30, 2009 | ||||
| Updating bot ban list & cleaning out obsolete entries |
11 | KenB | 8:15 pm Dec 29, 2009 | ||||
| Twiceler/cuil.com craziness (FWIW) |
7 | Pfui | 7:05 am Dec 29, 2009 |
