Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Ask Jeeves Ask Jeeves disregarding robots.txt |
16 | Scooter24 | 5:00 pm Sept 17, 2002 | ||||
| Ill-behaved libwww-perl UA Reads and ignores robots.txt |
3 | jdMorgan | 4:15 pm Sept 17, 2002 | ||||
| Scooter ignoring one page entry in robots how can i get it to not index this page? |
8 | semick | 7:48 pm Sept 16, 2002 | ||||
| Is this a user... or a bunch of people? |
2 | carfac | 6:57 pm Sept 14, 2002 | ||||
| MFC Foundation Class Library 4.0 |
3 | Scooter24 | 2:21 pm Sept 14, 2002 | ||||
| Polybot Old News? |
3 | brotherhood_of_LAN | 11:14 pm Sept 12, 2002 | ||||
| Where are these coming from? VERY bad spiders attacking my site... |
9 | irock | 11:11 pm Sept 12, 2002 | ||||
| New Marketing Campaign Bot won't go away... |
2 | fiestagirl | 6:17 am Sept 12, 2002 | ||||
| Inktomi poking around 209.131.51.166 |
4 | fiestagirl | 11:42 pm Sept 10, 2002 | ||||
| ec009.inktomi.com |
7 | littleman | 7:56 pm Sept 10, 2002 | ||||
| Is there a list of user-agents from robots |
9 | martin | 8:20 am Sept 10, 2002 | ||||
| dloader(NaverRobot)/1.0 Anyone know what this is? |
4 | Finder | 10:23 pm Sept 9, 2002 | ||||
| NaverRobot/1.0 Seems Polite but Weird |
3 | carfac | 10:20 pm Sept 9, 2002 | ||||
| Advice about Visitors with No UA If they don't tell you who they are... |
12 | carfac | 9:20 pm Sept 9, 2002 | ||||
| Identifying AaronCarter/14.7 Or what to do when you have way to much time... |
7 | Tapolyai | 1:18 pm Sept 8, 2002 | ||||
| How can you identify spiders without log access |
3 | lazyz | 8:02 pm Sept 7, 2002 | ||||
| Hits by Mozzilla/3.0- Not too polite |
2 | carfac | 6:06 pm Sept 7, 2002 | ||||
| DTS Agent Any new info on this UA? |
3 | SmallTime | 9:38 pm Sept 6, 2002 | ||||
| What spider is this? I found it in the kitchen.... |
4 | Reeper | 3:35 am Sept 6, 2002 | ||||
| Python-urllib/1.15 more RR |
3 | wilderness | 11:34 am Sept 5, 2002 | ||||
| New Spider FS-WebC |
3 | skirril | 10:16 am Sept 5, 2002 | ||||
| proxy.google.com, UA: UP.Browser/3.1.04-SY03 UP.Link/5.0.2.3c (Google |
11 | martin | 7:20 am Sept 3, 2002 | ||||
| I caught you. Ok, you got me. So now what? |
3 | bobriggs | 9:53 am Sept 2, 2002 | ||||
| empyreum.cz empyreum spider search engine czech |
4 | weesnich | 3:34 am Sept 1, 2002 | ||||
| Cyveillance behaving badly guess I'll write them a nice letter |
10 | idiotgirl | 10:44 pm Aug 30, 2002 |
