Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- QippoBot/1.4+(+http://www.qippo.com/bot/)


qippo - 7:05 am on Dec 5, 2012 (gmt 0)


Ok, that's fine with me. You, as the owner have all rights to block crawling of your website any way you wish. The good way is to use robots.txt, some crawlers don't listen to it - you block it by network. That's ok. It's not about Qippo though. We presume that any e-store wants publicity and getting more links to their products (we don't charge for it and crawl automatically), and if not - they use the most obvious way to express it: using robots.txt (or meta tags for specific web pages). I don't know any other way how any search engine could work otherwise.


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4522779.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com