homepage Welcome to WebmasterWorld Guest from 54.196.195.207
register, free tools, login, search, subscribe, help, library, announcements, recent posts, open posts,
Pubcon Platinum Sponsor 2014
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How about these code
abiao




msg:4106159
 11:00 pm on Mar 28, 2010 (gmt 0)

I do not want search engines to craw the certain cotents. How about the following code?

User-agent: *
Disallow: /cheapprescription/
Disallow: /cheap-glasses/?Cheap-Glasses-
Disallow: /cheap-glasses/?discount-glasses-
Disallow: /cheap-glasses/?Men-Glasses-
Disallow: /cheap-glasses/?Women-Glasses-
Disallow: /cheap-glasses/?Kids-Glasses-
Disallow: /cheap-glasses/?Eyeglasses-Glasses-
Disallow: /cheap-glasses/?Men's-Glasses-
Disallow: /cheap-glasses/?Women's-Glasses-
Disallow: /cheap-glasses/?kids-Glasses-
Disallow: /cheap-glasses/?Fashion-glasses-
Disallow: /cheap-glasses/?Prescription-Eyeglasses-
Disallow: /cheap-glasses/?Bifocal-Glasses-
Disallow: /cheap-glasses/?Reading-Glasses-
Disallow: /cheap-glasses/?Plastic-Eyeglasses-
Disallow: /cheap-glasses/?Prescription-glasses-
Disallow: /cheap-glasses/?Classic-glasses-
Disallow: /cheap-glasses/?Pure-Titanium-Eyeglasses-
Disallow: /cheap-glasses/?Semi-Rimless-Eyeglasses-
Disallow: /cheap-glasses/?Rimless-Glasses-
Disallow: /cheap-glasses/?Full-Rim-Glasses-
Disallow: /cheap-glasses/?Bendable-Titanium-Eyeglasses-
Disallow: /cheap-glasses/?Progressive-Glasses-
Disallow: /cheap-glasses/?eyeglasses-Progressives-
Disallow: /cheap-glasses/?eyeglasses-Bifocals-
Disallow: /cheap-glasses/?Full-Rim-Eyeglasses-
Disallow: /cheap-glasses/?Rimless-Eyeglasses-
Disallow: /cheap-glasses/?size0-
Disallow: /cheap-glasses/?size1-
Disallow: /cheap-glasses/?size2-
Disallow: /cheap-glasses/?size3-
Disallow: /cheap-glasses/?size4-
Disallow: /cheap-glasses/?price0-
Disallow: /cheap-glasses/?price1-
Disallow: /cheap-glasses/?price2-
Disallow: /cheap-glasses/?price3-
Disallow: /cheap-glasses/?Full-Rim-Sunglasses-
Disallow: /cheap-glasses/?Rimless-Sunglasses-
Disallow: /cheap-glasses/?Semi-Rimless-Sunglasses-
Disallow: /cheap-glasses/?Pure-Titanium-Sunglasses-
Disallow: /cheap-glasses/?Metal-Sunglasses-
Disallow: /cheap-glasses/?Bendable-Titanium-Sunglasses-
Disallow: /cheap-glasses/?Kids-Plastic-Glasses-
Disallow: /cheap-glasses/?Kids-Metal-Glasses-
Disallow: /cheap-glasses/?Kids-Full-Rim-Glasses-
Disallow: /cheap-glasses/?Kids-Bendable-Titanium-Glasses-

 

g1smd




msg:4108639
 12:09 am on Apr 2, 2010 (gmt 0)

This would remove more than half of the above code:

Disallow: /cheap*glasses
Disallow: /cheap*Glasses

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved