Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Hurricane Electric 570 MB in two hour period[2] ( 1 2 ) |
51 | OptiRex | 1:08 pm Aug 4, 2005 | ||||
| Gigabot/2.0 Is this legit? |
23 | dataguy | 2:16 am Aug 4, 2005 | ||||
| Who is "64.210.196.***"? I'm seeing this in my logs on very different domains. |
5 | fischermx | 3:24 am Aug 3, 2005 | ||||
| Ultimate short list of banned bots Wanna contribute? |
17 | silverbytes | 8:34 am Aug 2, 2005 | ||||
| Sohu search robot misbehaving Surprising considering their reach |
10 | jdMorgan | 5:18 pm July 31, 2005 | ||||
| annotate_google A private project by a Google programmer? |
3 | volatilegx | 4:14 pm July 31, 2005 | ||||
| Custom Bot/Robot #20 how to ban How to ban this bot |
3 | silverbytes | 12:07 am July 26, 2005 | ||||
| Spider ID? Thunder stone |
4 | wilderness | 3:43 pm July 25, 2005 | ||||
| TurnitinBot/2.0 is this bot harmfull is this bot can help me any way... or it is useless |
10 | osujit | 3:27 pm July 25, 2005 | ||||
| Microsoft URL Control - 6.00.8862 huge number of hits (for me anyway) in 2 hr. time |
2 | pjgarrit | 10:13 am July 24, 2005 | ||||
| Ask/Teoma or UUNET Spoofer? How to find what NETRANGES a Co. Owns? |
10 | Friday | 9:50 pm July 21, 2005 | ||||
| Ocp Hrs 1.0 I suspect it's an image bot. |
3 | GaryK | 4:58 pm July 18, 2005 | ||||
| Anyone heard of Compaq-NSIS-COP/1.0? |
4 | tkroll | 4:54 pm July 18, 2005 | ||||
| WebIndexer/1-dev Seems to be trying to grab DMOZ listed sites.... |
4 | bobothecat | 1:03 pm July 18, 2005 | ||||
| Possible Yahoo bot. No user agent or HTTP_REFERER header |
4 | volatilegx | 4:47 am July 18, 2005 | ||||
| MS Search 4.0 Robot |
4 | bobothecat | 2:58 pm July 15, 2005 | ||||
| Yahoo Seeker Going Wild - Is it worth it? 3 GB a day of bandwidth for this bot? |
2 | otc_cmnn | 10:22 pm July 13, 2005 | ||||
| Missigua Locator |
2 | mcneely | 9:26 pm July 13, 2005 | ||||
| JobSpider No jobs here! |
3 | bobothecat | 3:35 am July 13, 2005 | ||||
| Spoofed Googlebot They didn't quite get the UA and other things right |
2 | Dijkgraaf | 3:08 am July 13, 2005 | ||||
| Yahoo!'s Hotjobs bot |
2 | volatilegx | 5:46 pm July 11, 2005 | ||||
| ccubee |
7 | wilderness | 1:45 pm July 11, 2005 | ||||
| Problems with Voila Can't stop it, hundreds of thousands of hits per day. |
4 | motorhaven | 2:15 pm July 8, 2005 | ||||
| Yadows Crawler here's a new one |
4 | volatilegx | 8:20 am July 7, 2005 | ||||
| Microsoft-WebDAV-MiniRedir/5.1.2600 help please |
13 | thehittmann | 10:14 am July 3, 2005 |
