homepage Welcome to WebmasterWorld Guest from 54.227.215.139
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
How to disallow search results
How to disallow search results
spiritualseo

5+ Year Member



 
Msg#: 3700172 posted 7:56 pm on Jul 16, 2008 (gmt 0)

I have a website that produces search URLs that look like this:

example.com/Products?search=term

How do I disallow this?

The site also has a sort function that sorts products on a page. The URL generated is as follows:

example.com/Products?search=term&sort1desc=F&sort1=Item_NAME&range=

how do I disallow robots from crawling these pages?

Please let me know what I should add to the robots.txt page.

I would be really obliged for the help.

 

htdawg

10+ Year Member



 
Msg#: 3700172 posted 8:20 am on Jul 17, 2008 (gmt 0)

Hi,

I had a similar problem. Not sure if this will help but check this thread out

[webmasterworld.com...]

goodroi

WebmasterWorld Administrator goodroi us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month



 
Msg#: 3700172 posted 12:29 pm on Jul 17, 2008 (gmt 0)

htdawg is referring to using wildcards. wildcards aka pattern matching are supported by google, yahoo and msn. they are not officially part of the robots.txt protocol so many of the smaller bots do not handle them well. be careful with using wildcards with those other bots.

spiritualseo

5+ Year Member



 
Msg#: 3700172 posted 1:28 pm on Jul 17, 2008 (gmt 0)

Thanks for the replies..

The sort URL generated looks something like this: product-name?sort1desc=F&sort1=Item_NAME&range=

Would it help if I disallow:

Disallow sort1desc=F&sort1=Item_NAME&range=

I just need to disallow this single page

htdawg

10+ Year Member



 
Msg#: 3700172 posted 2:00 pm on Jul 17, 2008 (gmt 0)

I think so unless you have urls like this for example

sort1desc=F&sort1=Item_NAME&range=productname1
sort1desc=F&sort1=Item_NAME&range=productname2
sort1desc=F&sort1=Item_NAME&range=productname3 ect..

cause if you just disallow only sort1desc=F&sort1=Item_NAME&range=

you will still have urls like above I think.

so you might have to try the solution in the link I posted before.

Receptional Andy



 
Msg#: 3700172 posted 2:01 pm on Jul 17, 2008 (gmt 0)

Disallow: /Products?search will block any url starting with that string (presumably, that would cover all search result pages).
Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved