Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| MSIE 5.01 requesting robots.txt? anyone who knows this? |
3 | bull | 3:37 pm Sep 18, 2002 | ||||
| searching for formmail another candidate to ban? |
2 | bull | 9:43 am Sep 18, 2002 | ||||
| The number of googles spiders The number of googles spiders what does it stands for |
2 | Ove | 7:14 am Sep 18, 2002 | ||||
| minibot any ideas? |
2 | Finder | 11:10 pm Sep 17, 2002 | ||||
| Mysterious Spider |
5 | tesla | 7:44 pm Sep 17, 2002 | ||||
| Ask Jeeves Ask Jeeves disregarding robots.txt |
16 | Scooter24 | 5:00 pm Sep 17, 2002 | ||||
| Ill-behaved libwww-perl UA Reads and ignores robots.txt |
3 | jdMorgan | 4:15 pm Sep 17, 2002 | ||||
| Scooter ignoring one page entry in robots how can i get it to not index this page? |
8 | semick | 7:48 pm Sep 16, 2002 | ||||
| Is this a user... or a bunch of people? |
2 | carfac | 6:57 pm Sep 14, 2002 | ||||
| MFC Foundation Class Library 4.0 |
3 | Scooter24 | 2:21 pm Sep 14, 2002 | ||||
| Polybot Old News? |
3 | brotherhood_of_LAN | 11:14 pm Sep 12, 2002 | ||||
| Where are these coming from? VERY bad spiders attacking my site... |
9 | irock | 11:11 pm Sep 12, 2002 | ||||
| New Marketing Campaign Bot won't go away... |
2 | fiestagirl | 6:17 am Sep 12, 2002 | ||||
| Inktomi poking around 209.131.51.166 |
4 | fiestagirl | 11:42 pm Sep 10, 2002 | ||||
| ec009.inktomi.com |
7 | littleman | 7:56 pm Sep 10, 2002 | ||||
| Is there a list of user-agents from robots |
9 | martin | 8:20 am Sep 10, 2002 | ||||
| dloader(NaverRobot)/1.0 Anyone know what this is? |
4 | Finder | 10:23 pm Sep 9, 2002 | ||||
| NaverRobot/1.0 Seems Polite but Weird |
3 | carfac | 10:20 pm Sep 9, 2002 | ||||
| Advice about Visitors with No UA If they don't tell you who they are... |
12 | carfac | 9:20 pm Sep 9, 2002 | ||||
| Identifying AaronCarter/14.7 Or what to do when you have way to much time... |
7 | Tapolyai | 1:18 pm Sep 8, 2002 | ||||
| How can you identify spiders without log access |
3 | lazyz | 8:02 pm Sep 7, 2002 | ||||
| Hits by Mozzilla/3.0- Not too polite |
2 | carfac | 6:06 pm Sep 7, 2002 | ||||
| DTS Agent Any new info on this UA? |
3 | SmallTime | 9:38 pm Sep 6, 2002 | ||||
| What spider is this? I found it in the kitchen.... |
4 | Reeper | 3:35 am Sep 6, 2002 | ||||
| Python-urllib/1.15 more RR |
3 | wilderness | 11:34 am Sep 5, 2002 |
