Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| WebCapture Is this a bot? |
2 | copongcopong | 2:04 am Mar 28, 2003 | ||||
| "Generic" 128.255.244.82 |
2 | DavidT | 9:16 pm Mar 27, 2003 | ||||
| colocation |
3 | wilderness | 6:03 pm Mar 27, 2003 | ||||
| What's This? AOLserver-Tcl/3.3.1+ad13 |
3 | Oaf357 | 12:40 am Mar 27, 2003 | ||||
| When should I disable sessions? |
5 | Tonearm | 6:07 pm Mar 26, 2003 | ||||
| semanticdiscovery/0.1 A new information harvester |
18 | jdMorgan | 3:47 pm Mar 26, 2003 | ||||
| Another Japanese Bot: Aruyo/0.01 Just grabbed robots.txt |
2 | carfac | 7:27 am Mar 25, 2003 | ||||
| Unknown user agents that appear to be bots - valid spiders or proxies? Unknown user agents that appear to be bots - valid spiders or proxies? |
2 | FoodPlaces | 5:55 am Mar 25, 2003 | ||||
| Who's looking at my pages Is this a search engine? |
2 | Roscoe | 9:46 pm Mar 23, 2003 | ||||
| New Heavy Use Spider? 1900 pages + |
2 | Blade | 10:29 pm Mar 22, 2003 | ||||
| I need a new program to parse out my logs. Not satisfied with the reports I am getting |
7 | juniperwasting | 11:18 pm Mar 21, 2003 | ||||
| NetRatings / 64.124.34.46 Generating a lot of 404s |
4 | pallo | 11:13 pm Mar 21, 2003 | ||||
| Intelliseek brand protection robot |
7 | ScottMcG | 5:54 pm Mar 21, 2003 | ||||
| Black Widow Web Crawler Something to watch out for? |
5 | MarieC | 2:09 pm Mar 21, 2003 | ||||
| AmfibiBOT what is it? |
7 | khillman | 6:08 pm Mar 20, 2003 | ||||
| Current verified SPAM-bots User-agents and IP's |
4 | Orange_XL | 9:23 am Mar 20, 2003 | ||||
| 202.130.245.216 what's it? a new spider? |
4 | Chris_1977 | 10:08 pm Mar 19, 2003 | ||||
| Freshbot and Robots.txt |
7 | DavidT | 6:58 pm Mar 19, 2003 | ||||
| Eversion Avenger/37.17 (Chorus/MiX 3.2; 4-bit) No luck with Google {SIGH} -- HELP :) |
8 | xxX_RIPP_Xxx | 7:10 am Mar 19, 2003 | ||||
| Some quaint and unique (?) UAs *lol* |
17 | thermoman | 5:13 am Mar 19, 2003 | ||||
| Mozilla/3.0 (compatible) NOT Mozilla/3.0 (compatible;) |
3 | pendanticist | 11:39 pm Mar 18, 2003 | ||||
| My Spider report showing bots getting error and leaving Unknown error is keeping bots out of my site. |
12 | juniperwasting | 9:18 pm Mar 18, 2003 | ||||
| could "netscape 5" be a spider? |
4 | BillM | 7:02 am Mar 18, 2003 | ||||
| How to Ban Bots which come without http_user_agent info? It's real problem, I don't know how to set it up with rewrite mod |
11 | Chris_1977 | 2:52 am Mar 18, 2003 | ||||
| aol.com crawler = googlebot.com? |
9 | damiengazier | 1:42 am Mar 18, 2003 |
