Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| NetIQ - what are they looking for? |
16 | Mokita | 6:50 pm Oct 22, 2006 | ||||
| `ائ Yahoo! or Open Proxy? |
2 | GaryK | 7:08 am Oct 22, 2006 | ||||
| Hitwise Spider How do you block |
13 | RichTC | 4:26 am Oct 21, 2006 | ||||
| Simple syntax errors Can be difficult to see |
2 | wilderness | 1:14 pm Oct 19, 2006 | ||||
| Cks Google UA? |
3 | keyplyr | 1:04 pm Oct 19, 2006 | ||||
| Requests threw a 406 error User agent: "Mozilla/5.0" |
4 | Mokita | 10:47 pm Oct 18, 2006 | ||||
| Mozilla/4.0; (Compitable; MSIE 6.0; Windows NT 5.1;) MS is doing sneaky stuff |
8 | GaryK | 8:32 pm Oct 16, 2006 | ||||
| Fotochecker Except it didn't check any photos! |
2 | GaryK | 4:27 pm Oct 16, 2006 | ||||
| BlockNote.Net HTML editor. Ripped an entire site. |
3 | GaryK | 3:35 pm Oct 16, 2006 | ||||
| Spider coming out of 208.17.184.* Mystery spider coming out of Verisign |
8 | moocow | 3:25 am Oct 16, 2006 | ||||
| Edacious & Intelligent Web Crawler |
12 | volatilegx | 7:38 pm Oct 15, 2006 | ||||
| MSRBOT With Abovenet IP address Is this legit |
18 | dataguy | 3:32 am Oct 14, 2006 | ||||
| Googlebot spoofing |
6 | GaryK | 6:17 pm Oct 12, 2006 | ||||
| Yahoo to be added to list of bad bots? Ignoring Crawl Delay - up to 19 pages/sec |
5 | inbound | 1:08 am Oct 10, 2006 | ||||
| Cloaked robot from Ask.com No robots.txt, requests and malformed URLs |
5 | jdMorgan | 6:25 am Oct 9, 2006 | ||||
| Yahoo YSM bot being bad - Please help! |
3 | inbound | 8:16 am Oct 7, 2006 | ||||
| MindMapCrawler What's this? |
7 | keyplyr | 5:24 pm Oct 6, 2006 | ||||
| User-Agent: Nelian Pty Ltd - Spider v2.1 Owner claims he occasionally crawls as Google |
8 | GaryK | 12:39 pm Oct 6, 2006 | ||||
| RSS READER (mctwist@mail.dr-k.info).? |
2 | koka | 7:40 am Oct 6, 2006 | ||||
| InfoPath.1 Revisited |
2 | smells_so_good | 5:03 am Oct 6, 2006 | ||||
| Heads up for a voracious scraper |
5 | Mokita | 2:28 pm Oct 1, 2006 | ||||
| iearthworm yet another rogue crawler from Yahoo! |
5 | GaryK | 1:30 am Oct 1, 2006 | ||||
| TMCrawler reads robots.txt, and that's it... |
2 | bobothecat | 4:42 am Sept 30, 2006 | ||||
| Yahoo! spidering from new Class Cs |
4 | volatilegx | 2:42 am Sept 30, 2006 | ||||
| VerticalMatch this is a new one |
2 | volatilegx | 3:39 pm Sept 28, 2006 |
