Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Nutch in disguise? BobCrawl and Yowedo from same IP |
3 | GaryK | 3:30 am Mar 2, 2009 | ||||
| verify-Compliance Notice An unusual referer? |
9 | dstiles | 11:35 pm Mar 1, 2009 | ||||
| /msgimport / requests what is this? |
17 | smallcompany | 7:05 pm Mar 1, 2009 | ||||
| Web-non-sense aka websense, surfcontrol, mailcontrol and blackspider |
5 | caribguy | 6:45 pm Mar 1, 2009 | ||||
| DailyPerfect another scraper? |
3 | dstiles | 8:48 pm Feb 28, 2009 | ||||
| Google AdWords API |
2 | caribguy | 5:32 pm Feb 27, 2009 | ||||
| WinWebBot by balaena.com |
3 | dstiles | 4:49 pm Feb 26, 2009 | ||||
| Google IP grabbing JS and CSS only - bot? |
6 | Receptional_Andy | 2:30 pm Feb 26, 2009 | ||||
| WebDAV-MiniRedir |
2 | wilderness | 1:26 am Feb 24, 2009 | ||||
| GrubNG Is this a Vista crawler? |
2 | tangor | 5:06 pm Feb 23, 2009 | ||||
| A pair of WordPress crawlers No idea what purpose they serve |
5 | GaryK | 2:37 am Feb 23, 2009 | ||||
| spider appending url? |
2 | smokeybarnable | 11:34 pm Feb 21, 2009 | ||||
| Do you see a lot of "google.co.jp" referers, looking more like a bot? I'm seeing a lot of "random" traffic from 122.208.209.nnn |
3 | kzim | 1:38 am Feb 20, 2009 | ||||
| WWW-Mechanize/0.72 Sneaky |
6 | tangor | 8:06 am Feb 19, 2009 | ||||
| Kintiskton Banned after I stopped laughing |
21 | caribguy | 4:56 am Feb 19, 2009 | ||||
| happyfunsearch.com HappyFunBot |
2 | jmccormac | 5:24 pm Feb 16, 2009 | ||||
| Yanga WorldSearch Bot |
18 | keyplyr | 2:28 pm Feb 16, 2009 | ||||
| Opera.com Bot Opera/9.10 (Windows NT 5.1; U; en) |
4 | Pfui | 10:12 pm Feb 15, 2009 | ||||
| Spider or ripper? |
5 | tangor | 10:10 pm Feb 15, 2009 | ||||
| Safari favicon bot? This is really weird! |
5 | tangor | 8:50 pm Feb 15, 2009 | ||||
| LucidMedia ClickSense |
7 | wilderness | 8:10 pm Feb 13, 2009 | ||||
| Same IP switching UAs, over 7,000 404 errors |
4 | smallcompany | 11:57 pm Feb 12, 2009 | ||||
| Revisiting "VB Project": Kick or Keep? |
3 | Pfui | 11:25 pm Feb 12, 2009 | ||||
| Google Wireless Transcoder |
6 | dstiles | 3:15 am Feb 11, 2009 | ||||
| Does Google have a mobile crawler? If so do I have the right UA?[2] ( 1 2 ) |
38 | GaryK | 12:24 am Feb 8, 2009 |
