Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Do scrapers forge HTTP Referer? Wondering what the consensus is... |
4 | dataguy | 6:53 pm May 31, 2005 | ||||
| BecomeBot - nasty little critter, very aggressive Crawling too quickly. |
5 | andye | 8:42 am May 25, 2005 | ||||
| googlebot and log files problem what do you think about these logs? |
9 | macdar | 11:45 pm May 24, 2005 | ||||
| MyCrawler What is it? ban it? |
4 | guitaristinus | 10:41 pm May 24, 2005 | ||||
| Missigua Locator 1.9 is Baaack Spidered my entire site. |
26 | larryhatch | 11:46 am May 24, 2005 | ||||
| de Multiple UA's |
5 | wilderness | 10:27 am May 24, 2005 | ||||
| HelpSupportServices Lots of references to this UA but nothing definitive |
3 | GaryK | 6:12 pm May 23, 2005 | ||||
| EmeraldShield WebBot Is this real? |
2 | jim_w | 1:20 pm May 23, 2005 | ||||
| New Yahoo spider? |
2 | Strange | 8:52 pm May 20, 2005 | ||||
| Casiknow |
3 | wilderness | 5:54 pm May 20, 2005 | ||||
| Deepy any info |
9 | volatilegx | 3:30 pm May 19, 2005 | ||||
| Googlebot question? |
2 | PPCguy | 1:25 am May 19, 2005 | ||||
| googlebot alexa toolbar? |
7 | arubicus | 2:03 am May 17, 2005 | ||||
| DataCha0s/2.0 Anyone Else seen this? |
5 | Ocean10000 | 7:33 am May 13, 2005 | ||||
| Webrank Explorer Did not check robots.txt |
2 | jazzguy | 10:02 pm May 12, 2005 | ||||
| Unidentified bot from dxi.net Generic UA string. Doesn't fetch robots.txt. |
4 | jazzguy | 6:47 pm May 11, 2005 | ||||
| Anyone know the purpose of this bot? |
3 | voltrader | 5:11 pm May 11, 2005 | ||||
| rotating user agent crawler harvester using a rotating user agent |
2 | classifieds | 5:09 pm May 11, 2005 | ||||
| converas retrievalware internet spider couldn't find anything in the archives on this one |
3 | jd01 | 5:42 am May 11, 2005 | ||||
| what spider is that anybody can help me identify this spider? |
7 | MkSlim | 1:45 am May 10, 2005 | ||||
| BlackBerry |
2 | wilderness | 5:34 am May 8, 2005 | ||||
| securecomputing.com content filtering service No bot info in user-agent string |
3 | jazzguy | 7:36 pm May 6, 2005 | ||||
| Is this really a Yahoo spider? Found a strange one in my log |
2 | Kelowna | 1:38 pm May 5, 2005 | ||||
| BigCliqueBOT aka "bigclicbot," and "BigCliqueBOT (Daisy)" |
3 | bose | 10:35 am May 5, 2005 | ||||
| ScSpider |
3 | wilderness | 6:42 pm May 4, 2005 |
