Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| This thing shows up every 30-40 minutes on two of my sites |
16 | aristotle | 2:09 am Mar 3, 2015 | ||||
| Image Bot Names? |
4 | keyplyr | 10:46 pm Feb 27, 2015 | ||||
| Apple search engine bot? |
24 | johnhh | 8:49 pm Feb 25, 2015 | ||||
| IANA Private CIDRs Respectable Spooks only? |
4 | Angonasec | 7:14 pm Feb 25, 2015 | ||||
| OrgProbe Bot Warning |
12 | Angonasec | 2:33 pm Feb 25, 2015 | ||||
| 104.154.16.xxx Strange access from Google IPs |
8 | doc_z | 8:07 am Feb 24, 2015 | ||||
| Strange CFNetwork User-Agents com.apple.WebKit.WebContent |
4 | dstiles | 9:42 pm Feb 21, 2015 | ||||
| Onavo Proxy Mobile data compressor |
16 | dstiles | 10:44 pm Feb 17, 2015 | ||||
| Server Farms - August 2014 Tracking and Reporting Data Center IP Ranges [10] ( 1 2 3 4 5 6 7 8 9 10 ) |
285 | incrediBILL | 5:53 am Feb 17, 2015 | ||||
| WorldBrewBot |
3 | keyplyr | 6:06 pm Feb 14, 2015 | ||||
| Scansafe ranges |
5 | dstiles | 8:07 pm Feb 12, 2015 | ||||
| At Home with the Robots: 2015 edition |
12 | lucy24 | 8:27 am Feb 12, 2015 | ||||
| How can bots know the internal structure of a password protected site? |
7 | lamati | 10:35 am Feb 7, 2015 | ||||
| antiquated robots |
3 | lucy24 | 10:24 pm Feb 5, 2015 | ||||
| acapbot/0.1 |
4 | keyplyr | 8:11 pm Feb 4, 2015 | ||||
| Block everything from 54. except certain bots |
18 | physics | 8:51 pm Feb 1, 2015 | ||||
| Automating the server farm identification an alternative approach[3] ( 1 2 3 ) |
63 | trintragula | 12:55 am Feb 1, 2015 | ||||
| Android, Frames and Google Can android handle frames? |
4 | dstiles | 7:33 pm Jan 31, 2015 | ||||
| Looks like spiders from China? Receiving responsecode 500 errors from China visitors |
6 | Anders | 6:49 pm Jan 29, 2015 | ||||
| AdvBot |
9 | lucy24 | 9:32 pm Jan 24, 2015 | ||||
| Odd User Agent |
13 | wilderness | 8:32 pm Jan 22, 2015 | ||||
| "Mozilla/5.0 ()" - Is this used by any valid browsers? |
7 | physics | 12:15 am Jan 19, 2015 | ||||
| Named distributed crawler list |
6 | trintragula | 11:03 am Jan 9, 2015 | ||||
| HTTP header fields ACCEPT and ACCEPT-CHARSET |
3 | dstiles | 5:01 pm Jan 8, 2015 | ||||
| Opera Mini IP ranges |
5 | Xpat | 5:22 pm Jan 7, 2015 |
