homepage Welcome to WebmasterWorld Guest from 54.166.228.100
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Microsoft / Bing Search Engine News
Forum Library, Charter, Moderators: mack

Bing Search Engine News Forum

    
Microsoft Details Robots.txt
Brett_Tabke




msg:3665896
 4:40 pm on Jun 3, 2008 (gmt 0)

From the Live Search webmaster Blog [blogs.msdn.com]

The de-facto standard for managing this is the Robots Exclusion Protocol (REP) introduced back in the early 1990's. Over the years, the REP has evolved to support more than "exclusion" directives; it now supports directives controlling what content gets included, how the content is displayed, and how frequently the content is crawled.

[edited by: tedster at 6:58 pm (utc) on June 3, 2008]
[edit reason] fix side scroll [/edit]

 

WiseWebDude




msg:3665908
 4:55 pm on Jun 3, 2008 (gmt 0)

Oh, sweet, they are going to obey wildcards by end of June! Thanks, MSN! That is great to hear.

g1smd




msg:3666195
 11:14 pm on Jun 3, 2008 (gmt 0)

Will it be like Google operates though?

[webmasterworld.com...]

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Microsoft / Bing Search Engine News
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved