|Forum: Search Engine Spider and User Agent Identification|
|Displaying Topics 1 - 40 (57 total) Sorted by: Thread-Subject, Direction: forward|
|1:|| AVG - Valid Security Tool or Malware - Part Two|
AVG anti virus latest update includes a pre-fetch link scanner tool that some are viewing as malware itself.
|June 14, 2008|
|2:|| AVG Stops Real-Time Scanning||July 7, 2008|
|3:|| AVG Toolbar Glitch May Be Causing Visitor Loss|
"Web sites with tight security are turning away AVG visitors with security toolbar broadcasting malformed HTTP headers and user agent strings."
|May 10, 2008|
|4:|| Another Phorm Type Ad System Discovered?|
|June 20, 2008|
|5:|| At Home with the Robots: 2015 Edition|
An extensive review of robots, or web crawlers, and behaviour, good, friendly, or unhealthy.
|Feb 10, 2015|
|6:|| Bad Boy Bot of the Week|
Pick a bot, any bot: This weeks fun one is an affiliate bot out walking and not respecting robots.txt.
|Dec 6, 2002|
|7:|| Blocking WebPage Monitoring Services|
Most of these services are unwelcome bandwidth sponges. Lets call them out, identify them, and block them together.
|Apr 9, 2003|
|8:|| Blocking non-North American Traffic Made Simple|
Webmasters discuss how to make an amazingly small optimized IP block list that allows only North American traffic to access a website. The technique can easily be applied to other geographical areas.
|Apr 23, 2014|
|9:|| Bot ID and Blocking GET xmlrpc.php To Avoid Exploit|
WebmasterWorld Members discuss bot blocking to avoid xmlrpc exploit.
|July 10, 2015|
|10:|| Bot-Blocking Methodology|
"WebmasterWorld Members discuss various bot-blocking methodologies."
|Dec 13, 2008|
|11:|| Botnet Pandemic Threatens Core of Internet|
"Up to a quarter of computers on the net may be used by cyber criminals in so-called "botnets", according to Vint Cerf."
|Jan 26, 2007|
|12:|| Casper Bot Search Attempting To Infect Sites|
"Seen quite a few of these over the past few days, generally in groups of half a dozen-ish."
|July 7, 2010|
|13:|| Comcast Launches Anti-Botnet Initiative "Constant Guard"|
"Comcast is taking a leadership role and making a huge step forward in the eradication of botnets."
|Oct 12, 2009|
|14:|| Dealing With WordPress Comment Spam Escalation|
"Just to see what would happen I enabled full comments on my WordPress blog and at first I just let the comments pile up in the WordPress moderation queue as I was curious how bad it would get since nothing ever got published.[br][br]It quickly ramped up from a few a day to 100s a day, peaking currently at over 500 spam posts a day."
|Jan 22, 2014|
|15:|| Default User Agents of Programming Libraries and Command Line Tools||Apr 13, 2008|
|16:|| Digsby IM Enables Web Crawlers Control of Your PC & Bandwidth|
Did Digsby just go darkside?
|Sept 8, 2009|
|17:|| Facebook Sues Data Scraper|
"Warden gathered that data from public profiles using "crawling" software similar to what's commonly available on the Web..."
|Apr 4, 2010|
|18:|| Filtering Out Really Hard To Find Bad Bots|
WebmasterWorld Members discuss how best to filter out unwanted, bad bots that are tough to find.
|Jan 18, 2013|
|19:|| Fresh IP's in MSN's Many Cloaked Bot Arsenal|
"No UA, no robots.txt, no REF, no nothing. Not once. Not twice. Not even three times. Try eleven."
|Sept 3, 2010|
This is the first time in nearly 9 years I've seen G blatantly disregard robots.txt and they're doing it with a GoogleBot UA.
|May 16, 2011|
|21:|| Google's Web Preview Spider|
"WebmasterWorld Members discuss the Web Preview Spider, whether it obeys robots.txt, and how to block it."
|Nov 19, 2010|
|22:|| Googlebot Fails to Pass DNS Verification|
WebmasterWorld members have reported that an apparently valid Googlebot is failing DNS verification. Major impact for sites relying on Googlebot validation.
|Apr 2, 2014|
|23:|| HoneyNet White Paper : Know your Enemy: Web Application Threats|
"Web applications present a very high risk, and an attractive target to attackers for the following reasons: Firstly, the quality of the code is often rather poor and many vulnerabilities of commonly used code are published."
|Feb 27, 2007|
|24:|| Honeynet Project to Trap Spider Attackers and Name Names|
"The Web Honeynet Project, an independent group of Honeynet researchers from Securiteam...they plan to name not only the spider attack details, but also to divulge the IP addresses and other tracking information about the attackers themselves."
|Jan 27, 2007|
|25:|| How To Block Thousands of Spambot IPs Hitting a Site|
WebmasterWorld members discuss the best methods of handling and blocking spambots with thousands of unique IP addresses hitting a site, causing bandwidth to rise from 1GB a month to 12GB a day.
|Dec 12, 2011|
|26:|| How important is the Robots.txt file now?|
"Am I missing out on a lot of traffic by not having a robots.txt file? Or is that a thing of the past?"
|Jan 25, 2002|
|27:|| How to Identify and Block Fake BingBot Visits|
How do you identify and block fake BingBot visits to your sites.
|Apr 2, 2013|
|28:|| IP Banning Primer|
"I won't ask why you want to block IPs. But supposing you do, here's how to do it."
|Sept 21, 2009|
|29:|| Identifying And Analyzing Hostile & Friendly Bot Activity|
"The following items can be used to identity bots and slow down and stop most unwanted traffic if applied with proper due care."
|Mar 31, 2008|
|30:|| Identifying Fake User Agent Strings|
User agents come in all shapes and sizes. Some, like the fake Googlebots, are easy to recognize, but what about those really long ones. WebmasterWorld Members help clarify the identification process.
|June 11, 2012|
|31:|| MJ12bot Implements Ground-Breaking Validation Capability|
"...first distributed spider to provide validation for webmasters."
|Sept 3, 2009|
|32:|| MSN Robot Crawlers Renamed|
WebmasterWorld Exclusive: MSNDude "These are all Microsoft bots that have been around for a while...."
|July 31, 2006|
|33:|| MSNbot Changing to Bingbot on Oct.1, 2010|
"we will drop the beta designation from the Bing crawler and change the name of the crawler to reflect Microsoft's new brand for search."
|June 29, 2010|
|34:|| Microsoft Bot 157 Ranges Updated|
Microsoft's 157. range bots list updated.
|Nov 16, 2011|
|35:|| Microsoft Disables Live Search "Fake Referrers"|
"(Microsoft) are working on a fix for this."
|Apr 30, 2009|
|36:|| Microsoft Launches Azure, an AWS Competitor||July 15, 2009|
|37:|| Mysterious User Agent and 404 Random Strings|
"Altavista and DIIbot use suspicious request methods to test 404 errors."
|June 11, 2002|
|38:|| New Fast/Overture Robot UA||Sept 7, 2003|
|39:|| New Wave of SQL Injection Vulnerability Probes||Aug 29, 2008|
|40:|| Now Seeing Bingbot|
"Bingbot is now in the wild."
|Sept 29, 2010|