Welcome to WebmasterWorld Guest from 54.146.217.179

Forum Moderators: open

Message Too Old, No Replies

Disallow: Mozilla

Do you do this?

     
4:54 pm on Feb 13, 2002 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 20, 2001
posts:84
votes: 0


I'm not sure whether or not this has been talked about before. Does anyone try to stop a "shadow" spider posing as a browser by disallowing Mozilla/* in their robots.txt? If so (assuming it works), wouldn't this jeopardize their potential ranking.
5:13 pm on Feb 13, 2002 (gmt 0)

Administrator from US 

WebmasterWorld Administrator brett_tabke is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:Sept 21, 1999
posts:38048
votes: 12


Spiders that read robots.txt will only look for their unique id. I can't see Mozilla/ getting many spiders. However, have a look...I do use it here:
www.webmasterworld.com/robots.txt

Air

3:00 am on Feb 14, 2002 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 10, 2000
posts:1253
votes: 0


Most of the spiders fond of using alternate User Agents have been identified, the ususal method is to detect them by IP and feed them the cloaked page.

It's a good thought though, if they did look for and follow robots.txt for Mozilla, or whatever User Agent they happen to be using, it sure would make it easier to just block them.