Welcome to WebmasterWorld Guest from 54.80.8.44
Forum Moderated by: Ocean10000 & incrediBILL & keyplyr
Search Engine Spider and User Agent Identification
Forum to identify search engine spiders and user agents
| Thread Subject | Messages | Started by | Last Message | ||||
|---|---|---|---|---|---|---|---|
| Server Farms - Jan 2018 Continuing discussion of hosting company IP ranges[2] ( 1 2 ) |
53 | keyplyr | 10:31 pm Apr 22, 2018 | ||||
| At Home with the Robots: 2017 edition Critique of the Year's Active User Agents |
15 | lucy24 | 3:52 pm May 18, 2017 | ||||
| Blocking Methods How to block access to your server |
1 | keyplyr | 6:29 pm May 10, 2017 | ||||
| Default User Agents of Programming Libraries and Command Line Tools Resource page for common user agents |
15 | incrediBILL | 8:54 pm Apr 29, 2008 | ||||
| Leikibot |
5 | keyplyr | 7:53 pm Apr 24, 2018 | ||||
| 0 | 12:00 am Jan 1, 1970 | ||||||
| Prudent to block empty user agent visitors? |
25 | born2run | 6:26 pm Apr 24, 2018 | ||||
| Most of Your Traffic is Not Human [3] ( 1 2 3 ) |
69 | keyplyr | 1:20 pm Apr 24, 2018 | ||||
| VenusCrawler |
4 | keyplyr | 9:16 pm Apr 23, 2018 | ||||
| SurveyBot the new improved UA |
2 | lucy24 | 10:30 pm Apr 22, 2018 | ||||
| BacklinkCrawler |
2 | lucy24 | 9:28 pm Apr 22, 2018 | ||||
| another typo bites the dust |
2 | lucy24 | 8:57 pm Apr 22, 2018 | ||||
| The Knowledge AI |
5 | keyplyr | 1:16 am Apr 21, 2018 | ||||
| Yatke |
1 | keyplyr | 8:02 pm Apr 20, 2018 | ||||
| Buck |
6 | keyplyr | 11:03 pm Apr 19, 2018 | ||||
| Kudelski Security Innovation Port Scanner |
5 | blend27 | 2:19 am Apr 18, 2018 | ||||
| MauiBot |
13 | keyplyr | 10:49 pm Apr 17, 2018 | ||||
| CORS bot |
1 | keyplyr | 10:59 pm Apr 16, 2018 | ||||
| EU Rejects Robots 'Electronic Persons' Status |
9 | keyplyr | 11:22 pm Apr 14, 2018 | ||||
| IndeedBot |
3 | TorontoBoy | 6:54 am Apr 13, 2018 | ||||
| ltx71 |
2 | keyplyr | 4:00 am Apr 12, 2018 | ||||
| Whither Semalt and the like? reducing list of active crawlers like semalt in htaccess |
9 | Martin_Potter | 7:54 pm Apr 10, 2018 | ||||
| Cliqzbot |
12 | Bewenched | 1:16 am Apr 7, 2018 | ||||
| OutclicksBot |
6 | keyplyr | 1:39 am Apr 4, 2018 | ||||
| SitemapParser-VIPnytt |
3 | keyplyr | 8:37 pm Apr 2, 2018 |
Join The Conversation
- Register For Free! - Become a Pro Member!
- See forum categories - Enter the Forum
Moderators and Top Contributors
- Moderator List | Top Contributors:This Week, This Month, Mar, Feb, Archive, Top 100 All Time, Top Voted Members
Hot Threads This Week
- Google Updates and SERP Changes - April 2018
- April 2018 AdSense Earnings & Observations
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- Facebook Goes for World-Wide ToS Changes
- Chrome 66 for Windows, Mac and Linux Rolls Out
- Google Updates and SERP Changes - March 2018
- MauiBot
- How About an RSS Revival
- No way to make money on Adsense now, because of blank ads
Featured Threads
- Alphabet Q1 Revenues $31.1 Billion, up 26pct Year on Year
- Facebook Goes for World-Wide ToS Changes
- Chrome 66 for Windows, Mac and Linux Rolls Out
- Google AdSense: Expands "Funding Choices" To Replace Lost Revenue From Ad Blocking
- What Makes Good Web Content Today
- Mozilla: Internet Health Report 2018
- Facebook Launches Data Abuse Bounty
- How About an RSS Revival
- Webmasters: Distrust of Symantec PKI in Chrome 66
- A Directory is a Utility far more than it is a Business Model