Robots Disallow

How can I stop robots from hitting "add to cart"

[Alliee]

Hi Everyone. I have an ecommerce site and I track items that are put into a cart, but not purchased. The problem is that the robots seem to be hitting the "add to cart" button and I wish to stop them from doing this, however I want them to index the rest of the page, which contains info about the product, just not that link.

Two questions,

Where would the robot exclusion code go, on the home page, or on the actual page where the "add to cart" button exists?

Which of these is the correct syntax to accomplish this:

User-agent: *
Disallow: /~searchresults.asp/addtocart.html

The people I'm hosting with, on an NT machine, say the answer is simply to post the following on the page containing the addtocart:

<meta name="robots" content="index,nofollow">
This doesn't seem correct to me.

Any help would be greatly appreciated!

[tedster]

The index, nofollow meta tag placed on the product page would work, but it would also disallow bots from following any other the links on that page -- and that result may not be what you want.

The first syntax you suggest is better, but it doesn't belong in an HTML page at all -- neither the product page nor the cart page. It belongs in a separate file named robots.txt, placed in the root directory of your site.

By the way, we have an entire forum about the robots.txt file [webmasterworld.com] should your needs become more complex.

[Alliee]

Thanks so much Tedster!

Your answer makes perfect sense. I think I'll just disallow the whole Cart folder. I read through some of the other posts on the robots.txt forum and that seems the way to go.

Alliee

[DrDoc]

Just remember that not all robots obey robots.txt :(

[g1smd]

I put <meta name="robots" content="noindex,follow"> on the pages that I don't want indexed (though Google will already have followed the link to get there by then).

[dcrombie]

You could change your "add to cart" links to use the "POST" method (make each one a form button). That's the right way to do it according to the standards - but can be a bit tricky to set up.

[Alliee]

Decrombie, thanks for the tip. Now I just have to find a smart, dependable developer to implement, which is the most tricky part of all!