Welcome to WebmasterWorld Guest from 18.104.22.168
There's absolutely no way I'm allowing proximic access to my server.Looks like they claim their crawls affect rating of your pages for CPM advertisers, presumably on networks that they then sell info about your site to. If you are not running any third-party ads like AdSense, then that's not really a concern. But if you do, perhaps it warrants additional investigation.
[edited by: motorhaven at 3:12 am (utc) on Mar 4, 2013]
22.214.171.124 Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/) /robots.txt
126.96.36.199 Mozilla/5.0 (compatible; SearchmetricsBot; http://www.searchmetrics.com/en/searchmetrics-bot/) /robots.txt
One is housed at Hetzger
These bots belong to SEO sites. My question is, what's the consequence, server work wise, of blocking them or not blocking them.
Really I'm trying to find the logic in blocking a particular bot or ip (just because). Is it just bandwidth?