Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Interesting site ripper/bot like critter Try blocking this one (other than IP...) |
3 | tangor | 6:47 am July 18, 2010 | ||||
| Yahoo Pipes 1.0 as UA Seems like RSS Bot? |
2 | blend27 | 8:44 pm July 16, 2010 | ||||
| KiwiStatus/0.3 New Zealand Search |
2 | Dijkgraaf | 8:25 pm July 16, 2010 | ||||
| Gentoo UA - to block or not to block |
2 | zerillos | 7:22 pm July 15, 2010 | ||||
| How do you decide on an IP ban? |
13 | Status_203 | 1:47 pm July 11, 2010 | ||||
| imrd |
7 | Pfui | 9:52 pm July 10, 2010 | ||||
| User-Agent: in User Agent string Strange undeclared bot |
5 | Dijkgraaf | 7:58 pm July 9, 2010 | ||||
| SlugBug Spider a new bot on the block |
2 | Dijkgraaf | 2:30 am July 9, 2010 | ||||
| SearchBlox |
2 | keyplyr | 1:02 am July 6, 2010 | ||||
| Facebook gets value of robots.txt Not a Spider Report, just info |
7 | tangor | 8:39 am July 2, 2010 | ||||
| bloggsi (incl. link to post-Tweet bot recap) |
3 | Pfui | 11:22 pm July 1, 2010 | ||||
| WordPress/2.9.2 What kind of UA is this? |
3 | tangor | 4:50 pm June 30, 2010 | ||||
| How to ban a crawler that I can't see I know I'm being crawled, but I can't find the bot |
15 | arieng | 1:54 am June 30, 2010 | ||||
| MSNbot changing to Bingbot on Oct.1, 2010 |
3 | incrediBILL | 10:24 pm June 29, 2010 | ||||
| Twenga-Bot |
2 | tangor | 5:36 am June 29, 2010 | ||||
| Box-Agent |
3 | Pfui | 10:02 pm June 28, 2010 | ||||
| binlar (larbin) From an old pest: 207-67-117-17* |
2 | Pfui | 10:04 pm June 26, 2010 | ||||
| CogentCo |
9 | keyplyr | 10:42 pm June 17, 2010 | ||||
| How to enable every spider from Google, Yahoo and Bing ? How to bann every other spider/bot/robot ! |
12 | Future | 12:26 pm June 16, 2010 | ||||
| Awfully quiet in Spiders lately. |
4 | incrediBILL | 12:20 am June 16, 2010 | ||||
| GoogleToolbar in User Agent Does this imply it's not readng the page? |
19 | dstiles | 9:11 pm June 12, 2010 | ||||
| eSobi - opinions? desktop-based clipping app sends invalid HTTP_IF_MODIFIED_SINCE headers |
11 | caribguy | 1:48 am June 10, 2010 | ||||
| msramlbot (from Microsoft; no robots.txt; no thanks) |
2 | Pfui | 5:40 pm June 7, 2010 | ||||
| Facebook spidering? |
2 | Bewenched | 7:55 am June 5, 2010 | ||||
| SetEnvIf User-Agent |
7 | wilderness | 5:33 am June 5, 2010 |
