Forum Moderated by: open
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Riddler honors robots.txt |
2 | keyplyr | 6:19 am May 16, 2014 | ||||
| Elisabot 1.0 user agent and IP information |
5 | keyplyr | 8:28 am May 15, 2014 | ||||
| Inktomi vulnerability probe WordPress probe |
9 | keyplyr | 1:08 am May 15, 2014 | ||||
| Facebook scraper ? I can't figure this out. |
6 | kahuna | 11:18 pm Apr 30, 2014 | ||||
| Tiscali IstellaBot |
3 | carson63000 | 9:17 pm Apr 30, 2014 | ||||
| MSIE 11 odd UA string like gecko |
2 | dstiles | 2:23 pm Apr 28, 2014 | ||||
| Level 3 - Block 4./8 and 8./8 or not? How to handle? |
15 | keyplyr | 1:01 pm Apr 28, 2014 | ||||
| Crawling by Google How google crawling algo works |
9 | hayden | 7:03 am Apr 18, 2014 | ||||
| How To Block Possible Click Bombing |
20 | EmptyRoom | 5:03 am Apr 15, 2014 | ||||
| Can the Googlebot read the blocking rules inside .htaccess? |
5 | Andy500 | 8:55 pm Apr 14, 2014 | ||||
| Can Cloudfare stop content scrapers? |
4 | Andy500 | 5:12 pm Apr 14, 2014 | ||||
| Should I block msnbot-media |
5 | shamrock | 8:26 pm Apr 9, 2014 | ||||
| Previous Server Farms Threads |
4 | incrediBILL | 4:46 pm Apr 9, 2014 | ||||
| Should blank user agents be blocked or not? |
5 | aristotle | 7:47 pm Apr 7, 2014 | ||||
| Server Farms - Feb. 2014 Ongoing Hosting Data Center Discussion[4] ( 1 2 3 4 ) |
109 | incrediBILL | 5:43 pm Apr 7, 2014 | ||||
| Is this googlebot or not? Seems this in GBot's Range but doesn't Validate |
14 | webcentric | 2:36 pm Apr 7, 2014 | ||||
| How to block googleusercontent.com blocking google and googleusercontent in htaccess |
8 | rogerroger | 5:33 am Apr 7, 2014 | ||||
| Bots Crawling Pages Blocked by Robots.txt |
7 | incrediBILL | 6:52 am Apr 5, 2014 | ||||
| Is there a list of user-agents to block as of today? I'm launching a new website soon and want to avoid scrapers |
6 | shamrock | 6:06 am Apr 5, 2014 | ||||
| Hack attempt from a MS range |
21 | keyplyr | 2:32 am Mar 25, 2014 | ||||
| What is Project 5 Research, Inc. |
12 | HollyWMarshall | 2:29 am Mar 25, 2014 | ||||
| Strange crawl |
8 | wilderness | 11:22 am Mar 23, 2014 | ||||
| Semalt Referrer spamming gone mad. |
23 | blend27 | 2:10 am Mar 22, 2014 | ||||
| seegnifybot user agent and IP data |
4 | keyplyr | 11:03 pm Mar 20, 2014 | ||||
| Best Practices for blocking in Windows Server, IIS, .NET environment? |
8 | webcentric | 3:11 am Mar 20, 2014 |
