Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| DomainCrawler What purpose does it serve? |
11 | GaryK | 9:30 pm Nov 26, 2008 | ||||
| UA's starting with "User-Agent:" Redundant UA declaration from MSIE look-alikes |
4 | caribguy | 11:17 am Nov 25, 2008 | ||||
| SkyGrid |
2 | keyplyr | 2:12 am Nov 25, 2008 | ||||
| Voracious |
2 | keyplyr | 11:13 am Nov 24, 2008 | ||||
| BTWebClient |
5 | koan | 5:00 pm Nov 23, 2008 | ||||
| Personifi heads up HEAD request with Wget/1.10.2 and then GET with Mozilla/5.0 X11 |
8 | caribguy | 8:52 pm Nov 21, 2008 | ||||
| More netsweeper. I feel the urge to 86 216.171.96.nnn - 216.171.111.nnn |
3 | caribguy | 8:25 pm Nov 19, 2008 | ||||
| Intersting UA |
13 | wilderness | 5:13 pm Nov 18, 2008 | ||||
| Strange new "no-referrer" traffic |
16 | Scarecrow | 4:05 pm Nov 18, 2008 | ||||
| Mozilla/5.0 (compatible; OpenX Spider; http://www.openx.org) Posed as this then crawled as Nutch |
9 | GaryK | 5:43 pm Nov 17, 2008 | ||||
| WebTV Ballmer's rival for YouTube? |
6 | Samizdata | 3:51 am Nov 17, 2008 | ||||
| Bots Attack Using Randomized User Agents Simply Blocking Libwww-PERL Won't Work |
12 | incrediBILL | 1:06 am Nov 17, 2008 | ||||
| 1813 how you're doing with it today? |
4 | smallcompany | 10:38 pm Nov 16, 2008 | ||||
| BeB-cart Why is a shopping cart crawling my site? |
3 | GaryK | 5:33 pm Nov 15, 2008 | ||||
| lnbot FAST Enterprise Crawler 6 used by LexisNexis |
4 | caribguy | 2:19 pm Nov 12, 2008 | ||||
| Japanese AV bot? |
3 | Megaclinium | 9:02 pm Nov 10, 2008 | ||||
| copyright sheriff Another rights enforcer? |
3 | GaryK | 10:59 pm Nov 9, 2008 | ||||
| is this Googlebot legit? "Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)" |
6 | Nkona | 7:31 pm Nov 9, 2008 | ||||
| Mozilla/5.0 (compatible; Google Keyword Tool; +https://adwords.google. Anyone see this bot crawling lately |
14 | trinorthlighting | 3:15 pm Nov 9, 2008 | ||||
| Fake Googlebot from SoftLayer creepy creepy crawly crawly |
26 | Samizdata | 11:36 pm Nov 8, 2008 | ||||
| Google Home Page as Referrer and Random User Agents Is this a scraper? |
9 | dataguy | 9:00 pm Nov 8, 2008 | ||||
| WebDataCentreBot Mozilla/5.0 (compatible; WebDataCentreBot/1.0; +http://WebDataCentre.com/) |
9 | Receptional_Andy | 10:07 pm Nov 5, 2008 | ||||
| GoogleBot Crawl Frequency What's a likely lag time for Google to crawl new title tags? |
6 | SamNiccolls | 7:50 pm Nov 3, 2008 | ||||
| What is mvk-it.com crawler? |
2 | rivergirl | 5:23 pm Nov 2, 2008 | ||||
| Top reasons to ban bots |
6 | Clark | 11:16 pm Oct 29, 2008 |
