Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| NASA Search 1.0? |
6 | keyplyr | 6:00 pm July 12, 2006 | ||||
| Feed43 Proxy/1.0 (www.feed43.com) |
2 | zCat | 2:35 pm July 10, 2006 | ||||
| A Python-urllib hit 8000+ times yesterday |
2 | bigreat | 8:16 pm July 9, 2006 | ||||
| SBIder/0.8-dev Anyone getting this? |
15 | youfoundjake | 3:21 pm July 8, 2006 | ||||
| Hadrinka Tumaj Al-Kahal WebWasher 3.4 |
9 | Pfui | 6:18 am July 7, 2006 | ||||
| Yahoo Mindset No exclamation mark! |
3 | GaryK | 5:35 am July 5, 2006 | ||||
| page_verifier Scanning for malware |
5 | GaryK | 10:19 pm July 3, 2006 | ||||
| Redirect Test/0.1 from Amazon.com |
5 | GaryK | 10:10 pm July 3, 2006 | ||||
| lanshanbot/1.0 ( http://search.msn.com/msnbot.htm) This one looks familiar |
3 | GaryK | 9:52 pm July 3, 2006 | ||||
| New Google IPs GSA Crawler |
4 | fiestagirl | 3:29 pm July 3, 2006 | ||||
| Same-second hits from nec-labs using Java and -- Ken If at first you don't succeed, do a switcheroo. |
6 | Pfui | 3:20 am July 2, 2006 | ||||
| gsa-crawler Whois = Google? |
6 | coconutz | 11:21 pm July 1, 2006 | ||||
| /a1b2c3d4e5f6g7h8i9/nonexistentfile.php |
11 | Umbra | 3:32 am July 1, 2006 | ||||
| Snapbot Anyone know what it is?[2] ( 1 2 ) |
33 | Mokita | 3:18 am July 1, 2006 | ||||
| Psycheclone Anyone heard of this one? |
20 | malachite | 8:51 pm June 30, 2006 | ||||
| Excluding all UA's with spider, bot, and crawl in them Is this too broad a brush to paint with? |
5 | cfx211 | 8:16 pm June 29, 2006 | ||||
| Mozilla/4.0 (compatible; Google Desktop) New user-agent? |
3 | referer | 6:32 pm June 29, 2006 | ||||
| OutfoxBot disobeys robots.txt |
2 | keyplyr | 5:25 pm June 27, 2006 | ||||
| KBeeBot |
2 | keyplyr | 5:17 pm June 27, 2006 | ||||
| Charlotte the spider anyone seen this one yet? |
22 | innocbystr | 8:08 pm June 26, 2006 | ||||
| WebarooBot was RufusBot |
2 | keyplyr | 6:04 pm June 25, 2006 | ||||
| bot/1.0 claims to be Microsoft |
5 | 4string | 4:42 pm June 25, 2006 | ||||
| BeijingCrawler from -- Massachusetts (64.95.76.33); ignores robots.txt IP = Teragram Corp. Ran TeragramCrawler last month. |
4 | Pfui | 4:09 pm June 25, 2006 | ||||
| West Wind Internet Protocols 4.xx Never seen this before |
20 | zCat | 3:01 am June 25, 2006 | ||||
| Exalead Thumbnail bot |
6 | fiestagirl | 4:21 am June 24, 2006 |
