Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- QippoBot/1.4+(+http://www.qippo.com/bot/)


wilderness - 11:18 pm on Dec 4, 2012 (gmt 0)


No probs, I agree. That's why there are simple ways to block crawlers from crawling - tell them where you don't allow them to go. That's how you block google, right?


FWIW, you have your terminology confused.

A request within robots.txt is exactly that for bots that are compliant.

"Blocking access" (aka denial of access), whether a bot, or any other type of visitor, is a server action, of which the visitor has no choice.

dstiles offered the following, which you apparently overlooked.

many of us block all but the basic googlebot


What's all this pretension comparing yourself to Google?


Ditto.
Why not just deny the 176 Class A, and be down with it ;)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4522779.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com