Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| 64.40.105.197 Look.com |
4 | wilderness | 6:03 pm Sept 7, 2003 | ||||
| PlantyNet_WebRobot_V1.9 a bot and its two humans |
2 | coyote | 2:17 am Sept 6, 2003 | ||||
| Exalead -- it comes for your .mov files Why should I let this sucker on my system? |
4 | berli | 12:36 am Sept 6, 2003 | ||||
| 68.166.235.57 changed UA |
5 | coyote | 3:51 am Sept 5, 2003 | ||||
| Illinois State Tech Labs from a french IP |
3 | coyote | 2:28 am Sept 5, 2003 | ||||
| Feedster Crawler |
2 | frontpage | 10:52 pm Sept 4, 2003 | ||||
| Is slurp friendly? doesn't follow 301's and ignores robots.txt |
3 | wingslevel | 7:09 pm Sept 4, 2003 | ||||
| aaatravelermags.com (12.10.127.130) new spider? |
5 | gibEll | 12:23 am Sept 4, 2003 | ||||
| new bot: http://www.almaden.ibm.com/cs/crawler [c01] IBM Web Fountain |
9 | claus | 11:46 pm Sept 2, 2003 | ||||
| eseek-crawler-larbin-2.63 crawler@exactseek.com exactseek.com - 2 UA's |
4 | claus | 11:53 pm Sept 1, 2003 | ||||
| managednetworks.com Grabbed / from a bunch of sites |
6 | seindal | 10:29 pm Sept 1, 2003 | ||||
| What isthe purpose of "HEAD / HTTP/1.1" or "GET / HTTP/1.1"? I see these entries often in my web logs |
7 | Wizcrafts | 8:26 pm Sept 1, 2003 | ||||
| suchtop-bot-1.14 Can't find anything about it |
4 | Elmar | 8:41 am Sept 1, 2003 | ||||
| 216.33.229.163 Can I block this IP? |
2 | salson | 1:04 pm Aug 30, 2003 | ||||
| Is 63.225.216.82 a bot? It's slamming pages that don't exist |
4 | malexandrou | 12:25 am Aug 30, 2003 | ||||
| 216.217.36.130 stupid user or bot? |
5 | coyote | 10:49 pm Aug 29, 2003 | ||||
| From China with Spam heads up on a few IPs |
2 | coyote | 7:12 am Aug 28, 2003 | ||||
| Tandem scooter? (Scooter/3.3 & Scooter/3.3_SF) two UA's now |
4 | claus | 6:46 pm Aug 27, 2003 | ||||
| New UA signature for FAST |
2 | rbs10025 | 6:41 pm Aug 27, 2003 | ||||
| Mozilla/4.0 (compatible; GoogleToolbar 1.1.60-deleon; Windows 98 SE 4. wot? |
2 | claus | 6:07 pm Aug 27, 2003 | ||||
| Will this work Trying to get my robots.txt file right |
3 | le_gber | 2:28 pm Aug 27, 2003 | ||||
| Update on boitho.com-robot/1.1 Possible UA fake |
7 | cyberkat | 1:11 pm Aug 27, 2003 | ||||
| .net Clr 1.1.4322 Is this ok or not? |
4 | JadedJane | 4:57 am Aug 27, 2003 | ||||
| 64.37.103.2 (bundy.infomaxinc.com) |
5 | WarmGlow | 7:27 pm Aug 26, 2003 | ||||
| Mister Pix II 2.10 User-Agent Mister Pix II 2.10 |
3 | tnlnyc | 2:16 pm Aug 26, 2003 |
