Forum Moderators: phranque

Message Too Old, No Replies

Yahoo and MSN to crawl the pages not Google.com

Advice the robots.txt tag for this?

         

johnmich12

10:57 am on May 9, 2008 (gmt 0)

10+ Year Member



I have restricted 150 pages with robots.txt file, but we would like Yahoo, MSN and other search engines to crawl the pages not by the Google.com, please advice us what the tags to be included to crawl by the other search engines?

Waiting for the reply

jimbeetle

3:50 pm on May 9, 2008 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member Top Contributors Of The Month



If what you want to do is stop Google from indexing all pages on your site, simply use this directive in robots.txt:

User-agent: Googlebot
Disallow: /

If you want Googlebot to crawl some pages but not others, say to restrict it from one particular directory, use this directive:

User-agent: Googlebot
Disallow: /directorytorestrict/

Basic information about robots.txt can be found at robotstxt.org [robotstxt.org]. Specific information on how Google has implemented robots.txt can be found here [google.com].