Welcome to WebmasterWorld Guest from 220.127.116.11
do you look at page requests or all requests?My figures are coming from "page loads" based on a cross section of popular sites of small to medium size.
*based on 10k daily page loads
it's the rare robot that requests more than the page alone... So even if 2/3 of your page requests are from robots, it might still represent only a small fraction of all requests handled by your server.Absolutely true, but that's not the metric I'm writing about :)
The one thing that stood out for me on a recently analyzed single day is what appears to be a huge increase in bot-net type activity.Botnets can be huge. They come and go. I don't see them as much as I used to. They mostly come from compromised servers at hosting companies but can include the occasional compromised ISP account. These are scripts that are bought/sold in the dark web.
joined:July 23, 2015
if I'm running a promotion, especially on Social Media, bots will hit my server will great vengeance!
If it's mobile, it's likely a bot unless your site specifically targets mobile users.
The great difficulty with mobile visits is that you can't match the IP because it's different from one hour to the nextMobile IPs are no different than Desktop IPs. If you're sering different IPs "from one hour to the next" this is likely a bot, not a human on a mobile device.
joined:June 15, 2001
joined:July 29, 2007
social media ... provide no benefit to the site being crawledSurely that's a matter of individual judgement?
Google Analytics does a not to terrible job of showing real users, although there are some systems, specifically those offered by hosting companies that show just about everything that hits the server as a user.Dunno about other third-party analytics, but a major selling point of piwik--which the present site uses--is that it lives on your own server, so requests are subject to your own access-control rules.
To me, a bot by a news agency or social media company or data reporting company is not a trusted or "good" bot.Of course each site owner needs to determine what is or isn't benneficial to their interests.
if you're talking about the generic “analog stats”, yeah, that's pretty close to being worse than uselessWhat your host does with Analog may well be "generic" and not very useful, but Analog [mirror.reverse.net] is a robust and accurate server traffic analyzer that can be highly customized to return useful data. Best to install on your local machine to process hourly/daily raw access logs. Probably the most accurate logfile analyser I've used in 20 years (but you need to customize it to produce the metrics you use.)
joined:July 29, 2007
Surely that's a matter of individual judgement?
I am seeing over 50% bots. And this is after discounting all Amazon's "cloud", AWS bots as I block ALL of their IP blocks, all of themsmilie - you may want to reevaluate that tactic.
How are bot writers making so much money, who is paying them and why?Most bots are marketing driven, which has many applications, but it all comes down to data retrieval. Information has high value, and can be rolled into many products. Who's paying them? We are.
this is a major difference from keyplr - must be affirmable via rDNS (reverse DNS lookup) for access, which means that many/most cloud hosted bots are nonstarters.iamlost, I think you may have misunderstood the OP.
keyplr, based on various comments, is among the most lenient of active bot blockers I've encounteredPossibly, but I highly doubt it. My comments are usually not about my own sites. However, since you are interested, on my own personal site I block somewhere between 6k to 8k requests per day using the following, but not limited to Blocking Methods [webmasterworld.com]
[edited by: keyplyr at 1:39 am (utc) on Jul 10, 2017]