Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| New Spider? Gigabot/1.0 |
2 | dcheney | 9:15 am May 25, 2002 | ||||
| Win32; WinHttp.WinHttpRequest.5 67.113.225.66 |
4 | wilderness | 12:39 am May 25, 2002 | ||||
| Block and Complain success Sometimes they do clean up their act |
5 | jdMorgan | 5:31 am May 24, 2002 | ||||
| Interesting spiders from AV & Mercator |
17 | Air | 3:47 pm May 23, 2002 | ||||
| Cruizer IV Does anyone know it |
3 | olwen | 8:38 am May 22, 2002 | ||||
| What is this user agent? lwp-request/2.01 |
3 | korkus2000 | 9:15 pm May 21, 2002 | ||||
| Zealbot LookSmart Spider |
2 | Key_Master | 6:43 am May 21, 2002 | ||||
| How do I disallow spiders? I want to know what code to put on a page to not allow spiders to crawl. |
2 | Mitch | 5:03 pm May 20, 2002 | ||||
| another one to deny |
7 | littleman | 5:31 am May 20, 2002 | ||||
| Tcl http client package 1.0 What is this? |
4 | misosoph | 11:22 am May 19, 2002 | ||||
| Anyone Seen This INK bot |
3 | dstanovic | 11:01 am May 19, 2002 | ||||
| Ask Jeeves spider? |
2 | korkus2000 | 1:30 pm May 18, 2002 | ||||
| Anyone seen this Ask Jeeves spider? Ask Jeeves spider - why? |
6 | IanTurner | 1:55 am May 18, 2002 | ||||
| Coding a spider How to give yourself an agent name |
6 | chris_f | 6:53 pm May 17, 2002 | ||||
| Inktomi New C class or did I forget to add this one again? |
2 | msgraph | 4:47 pm May 17, 2002 | ||||
| webcollage variation collage.cgi/1.82 |
4 | misosoph | 2:09 pm May 17, 2002 | ||||
| Unknown Crawler 193.95.83.130 |
9 | chris_f | 8:07 am May 15, 2002 | ||||
| spider-loh-tc063.proxy.aol.com What Is This |
5 | Funfone | 8:44 pm May 14, 2002 | ||||
| AltaVista Spider? trek17.sv.av.com |
3 | HyperGeek | 4:27 pm May 14, 2002 | ||||
| Unknown - Lots of activity 66.28.250.176 |
3 | HyperGeek | 4:02 pm May 14, 2002 | ||||
| Two mystery spiders Heavy crawlers |
15 | Everyman | 5:09 am May 14, 2002 | ||||
| Need help ID ing a spider |
2 | RobertS | 12:35 pm May 13, 2002 | ||||
| What is Dsurf? |
6 | Brett_Tabke | 9:41 am May 12, 2002 | ||||
| Gozilla/4.0 Just looking for formmail.pl |
7 | bobriggs | 9:30 am May 12, 2002 | ||||
| UniverserseBot/1.0 from 209.17.183.249 NSlookup says thats gw.activestate.com |
2 | mbauser2 | 11:21 pm May 11, 2002 |
