Welcome to WebmasterWorld Guest from 188.8.131.52
There's absolutely no way I'm allowing proximic access to my server.Looks like they claim their crawls affect rating of your pages for CPM advertisers, presumably on networks that they then sell info about your site to. If you are not running any third-party ads like AdSense, then that's not really a concern. But if you do, perhaps it warrants additional investigation.
[edited by: motorhaven at 3:12 am (utc) on Mar 4, 2013]
184.108.40.206 Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) /robots.txt
220.127.116.11 Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/) /robots.txt
One is housed at Hetzger
These bots belong to SEO sites. My question is, what's the consequence, server work wise, of blocking them or not blocking them.
Really I'm trying to find the logic in blocking a particular bot or ip (just because). Is it just bandwidth?