Page is a not externally linkable
- Search Engines
-- Search Engine Spider and User Agent Identification
---- Twitter for Testing Bot Blocking


incrediBILL - 11:03 pm on Jul 15, 2012 (gmt 0)


Here's a simple Twitter tip for bot blockers.

Thanks to the swarm of link chasing bots that use Twitter's API you can easily test your bot blocker instantly with a single tweet. Just tweet a full URL to the site where you want to test your bot blocking and within seconds bots will start knocking on your door and continue to trickle in for a while. Many seem to cache the results so trying to get them to come to your site repeatedly in s short period of time requires directing them to different pages per tweet.

Hope this little trick helps some people when they're testing some new blocking filters because I've found it to be an invaluable tool to be able summon bots on demand.

Not only that, they've exposed some new hosts I didn't have blocked, a bonus! :)


Thread source:: http://www.webmasterworld.com/search_engine_spiders/4475928.htm
Brought to you by WebmasterWorld: http://www.webmasterworld.com