UA: Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions. Protocol: HTTP/1.1 Robots.txt: No Host: hetzner.de 78.46.0.0 - 78.47.255.255 78.46.0.0/15
Always a treat to find the bot page from the UA is a 404. Russian biz connectivity services, servers. They attempt to block translation on page so a cut'n paste to a translator was necessary.
lucy24
8:38 pm on Oct 22, 2015 (gmt 0)
hetzner.de <snip> no robots.txt <snip> bot page from the UA is a 404 <snip> attempt to block translation on page
Aaaand ... they expect not to meet a resounding 403 because ...?
keyplyr
3:10 am on Oct 23, 2015 (gmt 0)
I've come to the conclusion that webmasters like ourselves (those that write server code based on server requests) are for the most part irrelevant to bot runners. We are a very small number in the overall game.
IMO the majority of bots are purposed for purely egocentric reasons and their drivers could care less about how we may react to the behavior.
Only those that need our support/engagement consider us at all.
tangor
8:17 am on Oct 23, 2015 (gmt 0)
Wonder how many "webmasters" (ourselves at ww excluded) even look at their logs, much less the wasted bandwidth or scrape that occurs?
keyplyr
8:28 am on Oct 23, 2015 (gmt 0)
In 18 years, I have not once been hired by an individual that looked at server logs. One in 20 *might* look at a stats program and I've had relatively few ask me to install Google Analytics (usually only those that publish Adsense.)
Those that did view a stats report (usually supplied by their hosting company) understood little about its limitations. They would see a big surge in traffic and jump for joy. I've never had the heart to spoil their enthusiasm by suggesting it was likely a bot.
