Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Another new one! Lickity_Split |
6 | carfac | 3:23 am Sept 30, 2002 | ||||
| What is w.x.y.z.nombres.ttd.es What engine is this a spider from? |
3 | bnhall | 1:21 pm Sept 28, 2002 | ||||
| Scooter/3.2.SF0 |
2 | Scooter24 | 8:13 pm Sept 27, 2002 | ||||
| www.go2-yellowpages.com Deep crawling for awhile now, and incorrectly |
4 | savvy1 | 8:11 pm Sept 27, 2002 | ||||
| Spiders/Bots and Javascript Spiders without javascript indexing our "javascript required" page |
6 | Advantex | 8:46 pm Sept 25, 2002 | ||||
| Caught another "Trademark Search" robot |
3 | carfac | 6:44 pm Sept 25, 2002 | ||||
| antibot-V1.1.5/i586-linux-2.2 belongs to AntiSearch |
8 | volatilegx | 5:32 pm Sept 25, 2002 | ||||
| NetMechanic V3.0 Seems to be spidering? |
2 | Nick_W | 8:56 am Sept 24, 2002 | ||||
| I am getting hit really hard... ia_archiver |
8 | mack | 4:27 pm Sept 23, 2002 | ||||
| sherlock/1.3 httpget/1.3 |
2 | frontpage | 6:57 pm Sept 22, 2002 | ||||
| Scrubby 2.2 Is this a new spider? |
2 | Nick_W | 4:37 pm Sept 22, 2002 | ||||
| V5 HTTP/S Component Who is that? |
3 | frontpage | 3:57 pm Sept 22, 2002 | ||||
| Spider from multiple IP's all at once |
7 | carfac | 2:07 am Sept 22, 2002 | ||||
| 63.215.240.27 (unknown.level3.net) Scooter/2.0 |
6 | Key_Master | 3:45 pm Sept 20, 2002 | ||||
| av |
4 | PeteU | 7:21 pm Sept 19, 2002 | ||||
| 128.242.197.101 I am paranoid but . . . |
4 | rogerRabbit | 10:54 am Sept 19, 2002 | ||||
| MSIE 5.01 requesting robots.txt? anyone who knows this? |
3 | bull | 3:37 pm Sept 18, 2002 | ||||
| searching for formmail another candidate to ban? |
2 | bull | 9:43 am Sept 18, 2002 | ||||
| The number of googles spiders The number of googles spiders what does it stands for |
2 | Ove | 7:14 am Sept 18, 2002 | ||||
| minibot any ideas? |
2 | Finder | 11:10 pm Sept 17, 2002 | ||||
| Mysterious Spider |
5 | tesla | 7:44 pm Sept 17, 2002 | ||||
| Ask Jeeves Ask Jeeves disregarding robots.txt |
16 | Scooter24 | 5:00 pm Sept 17, 2002 | ||||
| Ill-behaved libwww-perl UA Reads and ignores robots.txt |
3 | jdMorgan | 4:15 pm Sept 17, 2002 | ||||
| Scooter ignoring one page entry in robots how can i get it to not index this page? |
8 | semick | 7:48 pm Sept 16, 2002 | ||||
| Is this a user... or a bunch of people? |
2 | carfac | 6:57 pm Sept 14, 2002 |
