Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Mozilla/4.0 (compatible; MyFamilyBot/1.0; http://www.myfamilyinc.com) Took disallowed files |
7 | GaryK | 3:18 am Aug 6, 2006 | ||||
| Goforitbot |
8 | Mokita | 6:11 pm Aug 5, 2006 | ||||
| Abusive IRLbot Abusive behaviour is intentional |
20 | andye | 3:26 pm Aug 5, 2006 | ||||
| google-sitemaps/1.0 |
5 | jake66 | 7:12 pm Aug 4, 2006 | ||||
| How do I identify spiders? |
6 | Geoffrey_james | 2:14 pm Aug 4, 2006 | ||||
| Not a real HEAD request? |
3 | wilderness | 10:35 am Aug 4, 2006 | ||||
| Szukacz claims to honour robots.txt but doesn't |
5 | Mokita | 4:28 pm Aug 3, 2006 | ||||
| Adwords Bot |
2 | volatilegx | 4:13 pm Aug 3, 2006 | ||||
| ebay indexing? |
6 | jake66 | 3:24 am Aug 3, 2006 | ||||
| PigBlock No robots.txt |
4 | GaryK | 7:14 pm Aug 1, 2006 | ||||
| Purpose of this Crawler/Bot? |
7 | DXL | 7:04 pm Aug 1, 2006 | ||||
| The new Y! slurp |
3 | BlackTulip | 5:36 pm Aug 1, 2006 | ||||
| PediaSearch.com Crawler |
9 | keyplyr | 7:19 am July 30, 2006 | ||||
| SevenTwentyFour/LinkWalker - New Owner, Mission Watch out! Brand name surveillance via LinkWalker. |
4 | Wizcrafts | 5:00 am July 30, 2006 | ||||
| lwp::simple/5.803 pretending to be yahoo? |
6 | jake66 | 7:53 pm July 27, 2006 | ||||
| "NutchCVS" (again) but from penguin26.parc.xerox.com No robots.txt |
3 | Pfui | 9:02 am July 26, 2006 | ||||
| "GT::WWW/1.026" from .reverse.layeredtech.com No robots.txt |
7 | Pfui | 4:16 am July 26, 2006 | ||||
| MetagerBot/0.8-dev (MetagerBot; http://metager.de; ) Note space before close paren |
8 | Pfui | 7:19 pm July 25, 2006 | ||||
| Downloads from a blank UA? |
23 | keyplyr | 6:26 am July 25, 2006 | ||||
| HA! No User Agent for You! These just flat-out annoy me! |
10 | GaryK | 2:16 am July 25, 2006 | ||||
| Crawler/1.0 http://elibron.com No robots.txt |
5 | GaryK | 7:24 pm July 24, 2006 | ||||
| Mozilla/5.0 (compatible;MAINSEEK BOT) No robots.txt |
3 | GaryK | 3:06 pm July 24, 2006 | ||||
| Mozilla/5.0 (compatible; robtexbot/1.0; http://www.robtex.com/ ) Note space before close paren. Also: no robots.txt; uses site URL in ref |
10 | Pfui | 1:50 am July 24, 2006 | ||||
| "teoma agent1" from directhit.com -- no robots.txt |
2 | Pfui | 11:17 pm July 22, 2006 | ||||
| "research-spider" from .cs.brown.edu |
2 | Pfui | 11:16 pm July 22, 2006 |
