Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| How do I track the spiders? Novice here... |
2 | Phil_AM | 7:39 pm Sept 24, 2004 | ||||
| Seekon new spider |
2 | volatilegx | 5:14 pm Sept 24, 2004 | ||||
| findlinks/0.87 |
3 | wilderness | 6:22 pm Sept 22, 2004 | ||||
| Rogue Yahoo-MMAudVid crawler Yahoo crawler hammers website, ignores robots.txt |
22 | tangent | 9:07 am Sept 21, 2004 | ||||
| Microsoft Data Access Internet Publishing Provider DAV 1.1 |
2 | wilderness | 3:22 pm Sept 18, 2004 | ||||
| JetEye Spider? |
2 | joeychgo | 4:25 am Sept 18, 2004 | ||||
| Googlebot IP 66.249.64.n |
6 | Just_Guessing | 9:35 pm Sept 16, 2004 | ||||
| Is it Google or not? I would expect any Ip used by Google would DNS resolve?! |
4 | MaxGrenk | 11:32 pm Sept 15, 2004 | ||||
| Fast? |
3 | wilderness | 1:01 am Sept 15, 2004 | ||||
| Unexplained visits no referrer - endless visits..no javascript |
2 | websitegal | 8:39 pm Sept 13, 2004 | ||||
| Cerberian Drtrs Version-3.1-Build-16 Does not fetch robots.txt |
13 | jdMorgan | 1:43 pm Sept 13, 2004 | ||||
| automatic logfile spam generator how to stop automated robot queries with false headers |
6 | cayleyv | 2:00 am Sept 12, 2004 | ||||
| Alexa bot intent on harvesting email addresses Why does Alexa want email addresses? |
6 | surfin2u | 12:45 am Sept 12, 2004 | ||||
| bob102.inktomisearch.com requesting /?marcoz_bs=my+favorite+keyword |
2 | bnhall | 5:50 pm Sept 8, 2004 | ||||
| Googlebot is Mozilla 5.0 compatable now seem new robots still under deploy |
2 | chedong | 4:10 am Sept 7, 2004 | ||||
| Spambots in PDFs Can spambots harvest e-mail addresses from PDF files? |
3 | cdarling | 3:28 am Sept 7, 2004 | ||||
| MSIE/6.0 libwww-perl/5.76 Looking for advice on this robot. |
3 | Tony_L | 4:00 am Sept 6, 2004 | ||||
| ShowLinks/1.0+libwww/5.4.0 anyone seen this one |
5 | aswaine | 10:49 pm Sept 2, 2004 | ||||
| BruinBot |
5 | volatilegx | 8:05 pm Sept 2, 2004 | ||||
| Has anyone had this one? Being spidered for 2 days |
9 | rainman359 | 6:31 pm Sept 2, 2004 | ||||
| Exava (exabot@exava.com) New shopping search engine in Mountain View, CA |
20 | jamesa | 1:46 pm Sept 2, 2004 | ||||
| New Unfriendly U-A and Request URI Is this an attempted server exploit? |
8 | Wizcrafts | 4:06 am Sept 2, 2004 | ||||
| ArtfaceBot? |
3 | keyplyr | 6:14 pm Aug 30, 2004 | ||||
| Several new spiders New to me at least |
19 | volatilegx | 1:41 pm Aug 29, 2004 | ||||
| Netscape 3 |
3 | wilderness | 4:13 pm Aug 28, 2004 |
