Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| 'index.html' and 'index.htm' Will spiders see both if they are both in root directory? |
2 | TWhalen | 9:51 pm Apr 9, 2003 | ||||
| bad bot or bad self made spider trap? |
6 | matthias | 6:30 pm Apr 9, 2003 | ||||
| snorkel-x.markwatch.com Is this a spider? |
5 | scone | 6:04 pm Apr 9, 2003 | ||||
| AlkalineBOT behaving a bit aggressive... |
7 | RonPK | 9:17 am Apr 9, 2003 | ||||
| what's this (PHP/4.2.3) |
5 | fandangood | 7:58 am Apr 9, 2003 | ||||
| What to do with names and I.P. addresses of spiders? why spend a ton of time gathering info? |
11 | Brad1ey | 1:06 am Apr 9, 2003 | ||||
| Windows NT 5.2? I wasn't aware of this type of browser... |
15 | kewlbeezer | 5:11 pm Apr 8, 2003 | ||||
| Weak bot 80.8.120.54 |
3 | wilderness | 12:45 am Apr 8, 2003 | ||||
| Larbin-experimental Rude bot |
5 | herb | 6:43 pm Apr 7, 2003 | ||||
| Google Visit |
11 | DavidT | 5:31 pm Apr 7, 2003 | ||||
| net-sweeper Ignored Robots.txt |
10 | frontpage | 5:15 pm Apr 7, 2003 | ||||
| Microsoft Spider poking around? seems to be behaving itself |
7 | WitchLars | 9:11 am Apr 7, 2003 | ||||
| fresh vs deep ips |
22 | Brett_Tabke | 4:43 am Apr 7, 2003 | ||||
| WebStripper I know, you probably already have it banned... |
4 | carfac | 7:11 pm Apr 6, 2003 | ||||
| WebFilter Robot 1.0 Good/Bad or just plain Ugly? |
8 | pendanticist | 4:59 pm Apr 5, 2003 | ||||
| Four visited in one day Is it unusual? |
3 | jaybee | 4:33 pm Apr 5, 2003 | ||||
| NCSA_Beta_1 The Supercomputer is after me! |
3 | bcolflesh | 4:48 am Apr 5, 2003 | ||||
| Cloaked Spider!? ...something strange went through my site! |
3 | frogg | 12:41 am Apr 4, 2003 | ||||
| WebCopier v2.8a |
10 | madmatt69 | 8:27 pm Apr 3, 2003 | ||||
| Google Confused? Is my site messing with google? |
3 | Skylo | 7:01 pm Apr 2, 2003 | ||||
| New Sighting: NexaBot/1.0 213.173.191.108 |
3 | skirril | 6:59 pm Apr 2, 2003 | ||||
| Msie 5.5 Msie 5.5 Spam |
13 | Andrue | 11:23 am Apr 2, 2003 | ||||
| openfind outdated threads |
2 | wilderness | 3:03 pm Apr 1, 2003 | ||||
| Badly-written spider Possibly using open .CA proxies |
2 | jdMorgan | 6:45 pm Mar 31, 2003 | ||||
| Filtering bots from traffic reports Agent name and/or IP address? |
6 | aspdaddy | 6:45 pm Mar 31, 2003 |
