Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Pioneer User Agent |
3 | frontpage | 7:28 pm July 10, 2002 | ||||
| Internet Explore 5.x - **mumble mumble** hyperactive chinese bot **mumble mumble** |
6 | Dreamquick | 7:50 am July 10, 2002 | ||||
| AltaVista NOIMAGEINDEX name="robots" content="noimageindex" Is this new? |
16 | misosoph | 11:24 pm July 9, 2002 | ||||
| Spider Trap needs GOOD Spider list Having setup a good spider trap I need to exclude the good ones... |
15 | scottspence | 4:44 am July 9, 2002 | ||||
| Heartbeat page requests... .. could be a little off topic... |
2 | scottspence | 10:04 pm July 8, 2002 | ||||
| Should a spider take more than a few pages at a time? whole site not been indexed |
4 | nickc001 | 8:10 am July 8, 2002 | ||||
| DTS Agent Anyone know what this one is? |
6 | jdMorgan | 12:55 am July 8, 2002 | ||||
| Steeler from 157.82.157.91 One of those research bots you guys hate. |
7 | mbauser2 | 10:09 am July 7, 2002 | ||||
| wysiwyg possible to ban? |
12 | keyplyr | 8:57 am July 6, 2002 | ||||
| mylinea? |
2 | Sinner_G | 9:28 pm July 5, 2002 | ||||
| SansarnRobot and S@... Originating from 203.185.132.67 |
2 | hanuman | 8:12 am July 5, 2002 | ||||
| Fast (alltheweb) - "FirstPage" - noticed this for the first time today |
4 | engine | 11:24 pm July 4, 2002 | ||||
| I need a list of Google IPs |
2 | tomasz | 12:17 pm July 4, 2002 | ||||
| Dual Proxy what is it? |
7 | keyplyr | 5:58 pm July 3, 2002 | ||||
| Spider Bait - anyone heard of it? apparently it hides the ? in database driven pages |
3 | charon | 10:43 am July 3, 2002 | ||||
| the NEW Hotbot/Lycos alliance Which spiders now? No more inktomi |
5 | Friday | 6:57 am July 3, 2002 | ||||
| Lachesis anyone heard of it? |
5 | incywincy | 10:45 pm July 2, 2002 | ||||
| Inktomi bot? |
3 | Weblamer | 8:56 pm July 2, 2002 | ||||
| Pompos/1.0 pompos@iliad.fr Anybody seen this? |
5 | volatilegx | 9:55 pm July 1, 2002 | ||||
| Who's Libby and why does she want to be my friend? |
3 | idiotgirl | 8:09 pm July 1, 2002 | ||||
| Is this valid robots.txt? valid robots.txt * asterisk disallow |
3 | pwc21 | 3:36 pm July 1, 2002 | ||||
| display error pages to banned users .htaccess too aggressive |
6 | idiotgirl | 7:04 pm June 30, 2002 | ||||
| Spider? choking on javascript Huge increase in 404's |
2 | Robert_Scott | 10:20 am June 29, 2002 | ||||
| Possible Spambot wih user agent "MFC Library" |
2 | Bluestreak | 10:39 pm June 27, 2002 | ||||
| Annoying and unusual bot activity from other countries |
3 | Bluestreak | 2:43 pm June 27, 2002 |
