Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Blocking Monitoring Services looking for list of ips.[2] ( 1 2 ) |
38 | Brett_Tabke | 4:54 pm Apr 18, 2003 | ||||
| List of search engine robots? |
2 | Jesse_Smith | 10:36 pm Apr 17, 2003 | ||||
| 20 | nativenewyorker | 10:55 am Apr 17, 2003 | |||||
| NetNose A new engine? |
6 | volatilegx | 9:57 pm Apr 16, 2003 | ||||
| IUSA Browser? Doesn't get robots.txt |
5 | kewlbeezer | 7:04 pm Apr 16, 2003 | ||||
| Honeypot concept for IIS, already done? Has anyone written yet, global.asa to block ignoring robots.txt spiders? |
5 | shadowspawn | 7:04 pm Apr 16, 2003 | ||||
| Guess What I want? 65.240.143.205 |
7 | wilderness | 7:55 am Apr 16, 2003 | ||||
| Mediapartners-Google/2.1 anyone seen this one? getting hammered |
16 | acronym | 8:23 am Apr 15, 2003 | ||||
| new spiders to me anyone know why two new spiders but why theyare crawling |
2 | chompy | 7:44 am Apr 15, 2003 | ||||
| is 64.171.96.89 a bot? ip spam on server |
4 | indiandomain | 6:24 am Apr 15, 2003 | ||||
| HTTPLib/1.0 whats this guy? |
2 | Andrue | 5:28 am Apr 15, 2003 | ||||
| no UA but then a: Mozilla/5.0 (Macintosh UA Mozilla/5.0 (Macintosh |
4 | Andrue | 4:53 am Apr 15, 2003 | ||||
| PF Free Web Search Tool Another one to ban...<sigh> |
5 | pendanticist | 4:14 am Apr 15, 2003 | ||||
| Way to "block" userAgents with ASP.NET Just a little script I have written.. |
6 | kewlbeezer | 6:37 pm Apr 14, 2003 | ||||
| spiders agent names |
11 | bignooz | 10:37 am Apr 14, 2003 | ||||
| Do you guys ban Cyveillance? They don't behave very well sometimes |
9 | kewlbeezer | 10:21 am Apr 14, 2003 | ||||
| Thai spider, or something else? wierd hits from asianet.co.th |
7 | Ankheg | 2:47 am Apr 14, 2003 | ||||
| Mac Finder 1.0.39 How about this crawler? |
7 | kewlbeezer | 11:58 pm Apr 13, 2003 | ||||
| Googlebots being spoofed? How can you tell? |
14 | pendanticist | 9:41 pm Apr 10, 2003 | ||||
| List of 'BAD' bots? Which one are they? |
3 | le_gber | 8:57 pm Apr 10, 2003 | ||||
| Mozilla/3.0 (compatible) Again Pulling strange urls |
5 | jimbeetle | 8:17 pm Apr 10, 2003 | ||||
| 65.92.240.232 - Is this a Googlebot? Googlebot/1.0 |
8 | Jesse_Smith | 4:04 pm Apr 10, 2003 | ||||
| URL_Spider_Pro/3.0 Update to URL Spider Pro/1.92 |
2 | pendanticist | 5:42 am Apr 10, 2003 | ||||
| proxy.google.com |
2 | coffeelover | 12:49 am Apr 10, 2003 | ||||
| timboBot/0.9 weblog indexer |
2 | frontpage | 10:33 pm Apr 9, 2003 |
