|
|
|
| Forum: Search Engine Spider and User Agent Identification | ||||
| Displaying Topics 1 - 40 (47 total) Sorted by: Date-Last-Post, Direction: reverse | ||||
| # | Subject |  Date | ||
| 1: |  Filtering Out Really Hard To Find Bad BotsWebmasterWorld Members discuss how best to filter out unwanted, bad bots that are tough to find. | Jan 18, 2013 | ||
| 2: | Identifying Fake User Agent Strings User agents come in all shapes and sizes. Some, like the fake Googlebots, are easy to recognize, but what about those really long ones. WebmasterWorld Members help clarify the identification process. | June 11, 2012 | ||
| 3: | How To Block Thousands of Spambot IPs Hitting a Site WebmasterWorld members discuss the best methods of handling and blocking spambots with thousands of unique IP addresses hitting a site, causing bandwidth to rise from 1GB a month to 12GB a day. | Dec 12, 2011 | ||
| 4: | Microsoft Bot 157 Ranges Updated Microsoft's 157. range bots list updated. | Nov 16, 2011 | ||
| 5: | The Best Way to Keep All Spiders/Bots Out Of A Site WebmasterWorld Members discuss the issue of stopping bots from crawling a site, and keeping them out. It seems it's tougher than you might think. | Oct 3, 2011 | ||
| 6: | Yahoo! Slurp Ignoring robots.txt WebmasterWorld Members report that Yahoo's Slurp is ignoring robots.txt | Sept 17, 2011 | ||
| 7: | Google Messing Up Javascript Stats This is the first time in nearly 9 years I've seen G blatantly disregard robots.txt and they're doing it with a GoogleBot UA. | May 16, 2011 | ||
| 8: | Stopping Scrapers From The Start "I'm putting a *huge* number of pages of content online. I'm looking to stop the scraping/copying/bots from the outset and I need bandwidth kept to a minimum." | Feb 25, 2011 | ||
| 9: | Google's Web Preview Spider "WebmasterWorld Members discuss the Web Preview Spider, whether it obeys robots.txt, and how to block it." | Nov 19, 2010 | ||
| 10: | Now Seeing Bingbot "Bingbot is now in the wild." | Sept 29, 2010 | ||
| 11: | Fresh IP's in MSN's Many Cloaked Bot Arsenal "No UA, no robots.txt, no REF, no nothing. Not once. Not twice. Not even three times. Try eleven." | Sept 3, 2010 | ||
| 12: | Casper Bot Search Attempting To Infect Sites "Seen quite a few of these over the past few days, generally in groups of half a dozen-ish." | July 7, 2010 | ||
| 13: | MSNbot Changing to Bingbot on Oct.1, 2010 "we will drop the beta designation from the Bing crawler and change the name of the crawler to reflect Microsoft's new brand for search." | June 29, 2010 | ||
| 14: | The Staggering Number of Tweet Chasing Bots Up to 20 bots now following twitter fire hose feed. | May 8, 2010 | ||
| 15: | Facebook Sues Data Scraper "Warden gathered that data from public profiles using "crawling" software similar to what's commonly available on the Web..." | Apr 4, 2010 | ||
| 16: | Updating The htaccess Bot Ban list "I'm sure most of us are familiar with the classic .httacces bad bot ban list for .htaccess that gets copied and pasted wholesale from web developer forum to forum (e.g.: http://www.webmasterworld.com/forum13/687.htm )" | Dec 29, 2009 | ||
| 17: | Comcast Launches Anti-Botnet Initiative "Constant Guard" "Comcast is taking a leadership role and making a huge step forward in the eradication of botnets." | Oct 12, 2009 | ||
| 18: | IP Banning Primer "I won't ask why you want to block IPs. But supposing you do, here's how to do it." | Sept 21, 2009 | ||
| 19: | Digsby IM Enables Web Crawlers Control of Your PC & Bandwidth Did Digsby just go darkside? | Sept 8, 2009 | ||
| 20: | MJ12bot Implements Ground-Breaking Validation Capability "...first distributed spider to provide validation for webmasters." | Sept 3, 2009 | ||
| 21: | Microsoft Launches Azure, an AWS Competitor | July 15, 2009 | ||
| 22: | Microsoft Disables Live Search "Fake Referrers" "(Microsoft) are working on a fix for this." | Apr 30, 2009 | ||
| 23: | Bot-Blocking Methodology "WebmasterWorld Members discuss various bot-blocking methodologies." | Dec 13, 2008 | ||
| 24: | New Wave of SQL Injection Vulnerability Probes | Aug 29, 2008 | ||
| 25: | AVG Stops Real-Time Scanning | July 7, 2008 | ||
| 26: | Another Phorm Type Ad System Discovered? "The Kindsight service...is funded through an advertising mechanism but without the use of cookies, pop-ups or spam. " | June 20, 2008 | ||
| 27: | AVG - Valid Security Tool or Malware - Part Two AVG anti virus latest update includes a pre-fetch link scanner tool that some are viewing as malware itself. | June 14, 2008 | ||
| 28: | AVG Toolbar Glitch May Be Causing Visitor Loss "Web sites with tight security are turning away AVG visitors with security toolbar broadcasting malformed HTTP headers and user agent strings." | May 10, 2008 | ||
| 29: | Yahoo! Slurp 3.0 Released on New IPs "The new Yahoo! Slurp 3.0 recognizes the same user-agent and all robots.txt directives for 'Yahoo! Slurp,' though it'll identify itself as Slurp 3.0 in your web logs." | Apr 15, 2008 | ||
| 30: | Default User Agents of Programming Libraries and Command Line Tools | Apr 13, 2008 | ||
| 31: | Identifying And Analyzing Hostile & Friendly Bot Activity "The following items can be used to identity bots and slow down and stop most unwanted traffic if applied with proper due care." | Mar 31, 2008 | ||
| 32: | Quick primer on identifying bot activity. | Mar 29, 2008 | ||
| 33: | Secure Sites From Botnet Vulnerability Probes "Blocking these 2 things just may buy you some time from when the vulnerability is posted to the time you or your customers get around to installing the fix without being hacked." | May 11, 2007 | ||
| 34: | Project Honey Pot files $1B+ Lawsuit Against Spammers and Spambots "On Thursday, April 26, 2007 at 10:54am in a court in the Eastern District of Virginia, Project Honey Pot filed the largest anti-spam lawsuit ever. Seeking more than $1B in statutory damages, the suit was brought on behalf of our members. It targets a huge swath of spammers. If you've harvested email addresses or sent spam in the last two years, chances are you're on our radar screen and we're coming after you." | Apr 28, 2007 | ||
| 35: | HoneyNet White Paper : Know your Enemy: Web Application Threats "Web applications present a very high risk, and an attractive target to attackers for the following reasons: Firstly, the quality of the code is often rather poor and many vulnerabilities of commonly used code are published." | Feb 27, 2007 | ||
| 36: | Honeynet Project to Trap Spider Attackers and Name Names "The Web Honeynet Project, an independent group of Honeynet researchers from Securiteam...they plan to name not only the spider attack details, but also to divulge the IP addresses and other tracking information about the attackers themselves." | Jan 27, 2007 | ||
| 37: | Botnet Pandemic Threatens Core of Internet "Up to a quarter of computers on the net may be used by cyber criminals in so-called "botnets", according to Vint Cerf." | Jan 26, 2007 | ||
| 38: | The IRS Bots - Xenon Spiders Web for the Tax Man "A five-nation tax enforcement cartel has been quietly cracking down on suspected internet tax cheats, using a sophisticated web crawling program to monitor transactions on auction sites, and track operators of online shops, poker and porn sites." | Jan 25, 2007 | ||
| 39: | MSN Robot Crawlers Renamed WebmasterWorld Exclusive: MSNDude "These are all Microsoft bots that have been around for a while...." | July 31, 2006 | ||
| 40: | Updated Bot List | Oct 4, 2003 | ||
| # | Subject | Date | ||
|
|
