Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| I spotted Gigablast crawling - Haven't seen it for a while |
4 | engine | 12:00 pm Dec 12, 2010 | ||||
| Google Web Preview [3] ( 1 2 3 ) |
66 | Mokita | 11:55 pm Dec 11, 2010 | ||||
| Flight Deck Bot (experimental) Experiment fail |
12 | Dijkgraaf | 1:05 am Dec 11, 2010 | ||||
| YahooCacheSystem |
2 | JAB_Creations | 6:13 pm Dec 10, 2010 | ||||
| HuaweiSymantecSpider Privacy, ethics |
7 | Frank_Rizzo | 9:04 pm Dec 8, 2010 | ||||
| Discobot crawler How to stop this horrible bot? |
8 | grandma_genie | 2:08 am Dec 8, 2010 | ||||
| Subdomain indexed with robots.txt disallowing?! |
4 | zelv | 6:52 pm Dec 5, 2010 | ||||
| How to block referrers from entire TLD? Referrer spam is really a headache! |
3 | TechSense | 6:47 pm Nov 30, 2010 | ||||
| Lipperhey Link Explorer www.lipperhey.com |
4 | grandma_genie | 7:39 am Nov 28, 2010 | ||||
| 80legs 80legs abuse |
4 | MxAngel | 3:23 pm Nov 27, 2010 | ||||
| UA blocking of 80legs bot? using .htaccess to block 80legs bot |
8 | classifieds | 7:13 am Nov 27, 2010 | ||||
| PostRank |
3 | JAB_Creations | 7:03 pm Nov 26, 2010 | ||||
| Multiple User Agents after one page at the same time. Twitterbot, Topsy, Tweetmemebot |
5 | grandma_genie | 9:22 am Nov 21, 2010 | ||||
| Is this a legitimate User Agent? Mozilla/4.0 |
8 | grandma_genie | 12:15 pm Nov 19, 2010 | ||||
| Ask Jeeves-Teoma Cutbacks to cut crawling? |
9 | Pfui | 7:32 pm Nov 18, 2010 | ||||
| WebGo IS - 5724 |
13 | keyplyr | 9:37 pm Nov 11, 2010 | ||||
| Now seeing Bingbot Testing the waters a few days early[2] ( 1 2 ) |
31 | jdMorgan | 5:59 pm Nov 11, 2010 | ||||
| crawler4j |
4 | incrediBILL | 5:46 am Nov 9, 2010 | ||||
| pirst pirst; MSIE 8.0; |
2 | Dijkgraaf | 9:31 am Nov 6, 2010 | ||||
| SockrollBot |
3 | Pfui | 10:00 pm Nov 5, 2010 | ||||
| FyberSpider FyberSpider/1.3 (http://www.fybersearch.com/fyberspider.php) |
2 | Dijkgraaf | 1:52 am Nov 4, 2010 | ||||
| Kindle Qs |
2 | Pfui | 10:20 pm Nov 1, 2010 | ||||
| dedibox.fr Badly behaved bot returns only slightly better behaved. |
4 | Dijkgraaf | 12:22 am Nov 1, 2010 | ||||
| Requests for logfiles Whois DE |
4 | caribguy | 12:19 am Nov 1, 2010 | ||||
| RankFlex.com Webspider |
2 | incrediBILL | 12:12 am Nov 1, 2010 |
