Forum Moderators: open
robots.txt? NO
Sells 'social media metrics,' the new-old coin of the realm.
IP = Host name in UA. Hosted by -- wait for it -- Hurricane Electric.
I encouraged them to join the discussion here.
Also, I said it's mostly because of three issues. Being hosted by HE. Not obeying robots.txt. Not being the kind of site that will send most of us any meaningful traffic in exchange for letting our sites be crawled.
Reading and heeding robots.txt
I do for some, but it's mostly binary data dump for the first 1KB and then 403 ;)
But I don't think they are losing sleep over the loss of indexing your amazing site.
2) Most of us don't consider whether the bot-owner 'cares' or not, we just don't want them wasting our bandwidth or scraping our sites -- intentionally or unintentionally outranking us in the SERPs with our own content.
3) This isn't us 'being mean' to them, it's just self-defense -- especially in cases where the Webmaster is on a limited-bandwidth plan, and the aggregate 'junk-robot' traffic puts the site over that limit and incurs an additonal hosting charge -- with no benefit to the site whatsoever.
4) In many cases, it's simply a matter of the Webmaster not having time to pursue DMCAs and other actions that would otherwise be required to 'keep up' with all of these unknown, useless, and/or malicious requests. It's far easier to just allow legitimate browsers and the top-tier search engines, block the rest, and go on with other more-productive site development activities.
5) To each his own. Each Webmaster is free to protect his/her work and intellectual property if he/she chooses to do so. Or not.
Jim
they send you money via advertisers
6) They send me no money whatsoever by this or any other means.
They could (potentially) cost me money if I allowed them unrestricted access.
But I have no desire to interfere with their sleep.
...
Yes they do, I've seen them do it, turning and wailing like it rains outside, especially that dude with the old orange broom wearing holistically pink bandana.
