homepage Welcome to WebmasterWorld Guest from 23.23.9.5
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Marketing and Biz Dev / Cloaking
Forum Library, Charter, Moderator: open

Cloaking Forum

    
Disallow: Mozilla
Do you do this?
nowhere

10+ Year Member



 
Msg#: 312 posted 4:54 pm on Feb 13, 2002 (gmt 0)

I'm not sure whether or not this has been talked about before. Does anyone try to stop a "shadow" spider posing as a browser by disallowing Mozilla/* in their robots.txt? If so (assuming it works), wouldn't this jeopardize their potential ranking.

 

Brett_Tabke

WebmasterWorld Administrator brett_tabke us a WebmasterWorld Top Contributor of All Time 10+ Year Member



 
Msg#: 312 posted 5:13 pm on Feb 13, 2002 (gmt 0)

Spiders that read robots.txt will only look for their unique id. I can't see Mozilla/ getting many spiders. However, have a look...I do use it here:
www.webmasterworld.com/robots.txt

Air

WebmasterWorld Senior Member 10+ Year Member



 
Msg#: 312 posted 3:00 am on Feb 14, 2002 (gmt 0)

Most of the spiders fond of using alternate User Agents have been identified, the ususal method is to detect them by IP and feed them the cloaked page.

It's a good thought though, if they did look for and follow robots.txt for Mozilla, or whatever User Agent they happen to be using, it sure would make it easier to just block them.

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Marketing and Biz Dev / Cloaking
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved