Welcome to WebmasterWorld Guest from 184.108.40.206 , register , login , search , subscribe , help , library , PubCon , announcements , recent posts , open posts Subscribe and Support WebmasterWorld
Disallow: Mozilla Do you do this? nowhere msg:679048 4:54 pm on Feb 13, 2002 (gmt 0) I'm not sure whether or not this has been talked about before. Does anyone try to stop a "shadow" spider posing as a browser by disallowing Mozilla/* in their robots.txt? If so (assuming it works), wouldn't this jeopardize their potential ranking.
Brett_Tabke msg:679049 5:13 pm on Feb 13, 2002 (gmt 0)
Spiders that read robots.txt will only look for their unique id. I can't see Mozilla/ getting many spiders. However, have a look...I do use it here: www.webmasterworld.com/robots.txt Air msg:679050 3:00 am on Feb 14, 2002 (gmt 0)
Most of the spiders fond of using alternate User Agents have been identified, the ususal method is to detect them by IP and feed them the cloaked page.
It's a good thought though, if they did look for and follow robots.txt for Mozilla, or whatever User Agent they happen to be using, it sure would make it easier to just block them.