Welcome to WebmasterWorld Guest from 54.167.40.25

Forum Moderators: Ocean10000 & incrediBILL & phranque

Message Too Old, No Replies

Simulating bots/crawlers ?

     
9:14 am on Jul 13, 2009 (gmt 0)

Junior Member

5+ Year Member

joined:Apr 18, 2006
posts: 115
votes: 0


I run a image heavy site, and Yahoo, Microsoft, Google image bots/crawlers visit me frequently. All 3 of them have indexed many thousand of my images.

I've been thinking about this and I want to replace the images I serve to the bots with watermarked images which include my domain name on the image.

This is simply added by using mod rewrite checking on the useragents extracted from my access log files. However how do I test this ? How can I browse my website with a specific useragent so I can see what a bot will see ?

Thanks in advance.

1:34 pm on July 13, 2009 (gmt 0)

Senior Member

WebmasterWorld Senior Member wilderness is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Nov 11, 2001
posts:5408
votes: 2


Use the Firefox Browser with the UA Plugin.
1:40 pm on July 13, 2009 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


If you are a webmasterworld supporter member - use the SimSpider to change the agent.

Aside from UA checkers, you can run proxies like Proxomitron to run local. (proxo is freeware and no longer supported)