Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| List of bad/spam bots out there? |
16 | Grinler | 8:36 pm Aug 16, 2006 | ||||
| Yahoo improving? . maybe |
4 | Mokita | 12:33 am Aug 15, 2006 | ||||
| QFKBot pagefetch didn't ask for robots.txt |
2 | Mokita | 12:08 am Aug 15, 2006 | ||||
| Looks like a browser but acts like a crawler AppleWebKit/521.25 |
2 | GaryK | 9:09 pm Aug 13, 2006 | ||||
| 192.comAgent violates robots.txt |
5 | Mokita | 8:40 pm Aug 12, 2006 | ||||
| Googlebot-Image/1.0 Off the deep-end |
3 | wilderness | 10:40 pm Aug 11, 2006 | ||||
| Layered Technologies |
2 | wilderness | 9:19 pm Aug 11, 2006 | ||||
| lynx/2.8.5dev.16 libwww-fm/2.14 ssl-mm/1.4.1 openssl/0.9.7a |
6 | jake66 | 3:52 pm Aug 11, 2006 | ||||
| MVAClient |
13 | wilderness | 4:25 am Aug 10, 2006 | ||||
| Strange htm pages in my log |
2 | admad1 | 3:45 am Aug 9, 2006 | ||||
| My Browser More abusive behavior from Yahoo? |
4 | GaryK | 8:07 pm Aug 8, 2006 | ||||
| MSN bot Crawlers Renamed |
20 | jake66 | 5:58 am Aug 8, 2006 | ||||
| Another GoogleBot wannabe They must think we are idiots! |
10 | GaryK | 10:06 pm Aug 7, 2006 | ||||
| User Agent: "13" coming from an Everyones Internet IP |
7 | Mokita | 1:08 pm Aug 7, 2006 | ||||
| Cocoal.icio.us/1.0 What is it? |
2 | GaryK | 2:43 am Aug 7, 2006 | ||||
| KRetrieve/1.1/dbsearchexpert.com |
2 | Pfui | 9:05 pm Aug 6, 2006 | ||||
| Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.myfamilyinc.com) Took disallowed files |
7 | GaryK | 3:18 am Aug 6, 2006 | ||||
| Goforitbot |
8 | Mokita | 6:11 pm Aug 5, 2006 | ||||
| Abusive IRLbot Abusive behaviour is intentional |
20 | andye | 3:26 pm Aug 5, 2006 | ||||
| google-sitemaps/1.0 |
5 | jake66 | 7:12 pm Aug 4, 2006 | ||||
| How do I identify spiders? |
6 | Geoffrey_james | 2:14 pm Aug 4, 2006 | ||||
| Not a real HEAD request? |
3 | wilderness | 10:35 am Aug 4, 2006 | ||||
| Szukacz claims to honour robots.txt but doesn't |
5 | Mokita | 4:28 pm Aug 3, 2006 | ||||
| Adwords Bot |
2 | volatilegx | 4:13 pm Aug 3, 2006 | ||||
| ebay indexing? |
6 | jake66 | 3:24 am Aug 3, 2006 |
