Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| top 10 list of useful bots to allow on your site? instead of a trap how about only the good ones |
5 | amznVibe | 4:27 pm Jan 24, 2003 | ||||
| pavuk/0.9pl28 |
3 | wilderness | 8:48 am Jan 24, 2003 | ||||
| Scooter Doing Double Duty? Scooter hits my site from Spain? |
3 | carfac | 5:07 am Jan 24, 2003 | ||||
| W3CRobot/5.4.0 Does anyone else think this is a fake? |
4 | Dreamquick | 1:23 am Jan 24, 2003 | ||||
| potbot 1.0 Unknown robot from everyone's internet |
9 | jdMorgan | 4:57 pm Jan 23, 2003 | ||||
| IAArchiver-1.0 & robots.txt Okay how much do I need to pay to take a big stick to the archive.org bot? |
4 | Dreamquick | 1:32 pm Jan 23, 2003 | ||||
| 65.102.17.89 just tripped a site trapdoor this IP appears in a removed message? |
15 | amznVibe | 4:44 am Jan 23, 2003 | ||||
| Lacnic IP ranges |
2 | wilderness | 11:48 pm Jan 22, 2003 | ||||
| Inktomi Search 158.140.2.102 |
6 | Adam_C | 5:12 am Jan 22, 2003 | ||||
| Trapdoors, Spiders, ..I'm Freekin |
6 | TomJones | 5:42 am Jan 21, 2003 | ||||
| Is this Bot new? potbot 1.0 |
4 | Rugles | 11:12 pm Jan 20, 2003 | ||||
| Potbot |
2 | billdaly | 8:56 pm Jan 20, 2003 | ||||
| Spinway a bot? |
2 | volatilegx | 3:17 pm Jan 20, 2003 | ||||
| road runner: imagescape Robot |
2 | ZeroCool | 7:39 am Jan 19, 2003 | ||||
| Caching Robot Any reference? |
2 | frontpage | 8:50 pm Jan 18, 2003 | ||||
| PayPal spider, let it in or not? PayPal "spiderman" does not obey robots.txt |
5 | amznVibe | 2:39 pm Jan 18, 2003 | ||||
| DeepIndex New European search engine |
4 | volatilegx | 5:32 am Jan 18, 2003 | ||||
| Grabbed 165 pages in 1-1/2 minutes.... |
3 | pendanticist | 11:38 am Jan 16, 2003 | ||||
| anyone who or what this is |
4 | paul_london | 12:38 am Jan 16, 2003 | ||||
| CDI Corporation 64.94.199.9 |
3 | wilderness | 12:26 am Jan 16, 2003 | ||||
| Fastnet 209.92.205.7 |
3 | wilderness | 12:24 am Jan 16, 2003 | ||||
| WAP masquerding as G deepcrawler |
3 | ga_ga | 12:09 pm Jan 15, 2003 | ||||
| Internetseer, changing tactics. |
2 | skirril | 10:14 am Jan 15, 2003 | ||||
| Datedate do spiders look at that? |
3 | fashezee | 8:56 pm Jan 14, 2003 | ||||
| mil.ten nsa.p.group Any thoughts? |
5 | newsparrot | 7:38 am Jan 14, 2003 |
