Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

robots and dynamic url

How to ban robots from a dynamic url

4:33 am on Nov 22, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 17, 2003
votes: 0

I have dynamically generated product pages (have 600 products) and on each page it lists the following:

1. A purchase button
2. A tell a friend button (loads a email script along with product info)
3. A add to wishlist button
4. Enlarge product image button
5. Complimentary product the customer may be interested in, say 5 or 6 for each product.

I have a problem in that the robots don't want to index my dynamic content and I think it may be because indexing product pages overwhelms it, yes? I mean your talking each product x at least 10 more links for a total of about 5400 links. Of course the complimentary products are repeats, but still it seems like to much.

Can I ban robots from a specific file, say tellafriend.asp or addwishlist.asp and does it matter if the file has a query string on the end of it?

Hopefully I explained this well enough, if I didn't let me know and I'll try and clarify.


3:10 pm on Nov 23, 2003 (gmt 0)

Administrator from GB 

WebmasterWorld Administrator engine is a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

joined:May 9, 2000
votes: 685

The robots.txt file will enable you to control which spiders and which areas of your site are traversed and which ones are not. It does not allow you to block the link.

Use the Meta noindex or nofollow on specific pages you want to control.

With dynamic pages, part of the problem is the complexity of the url. The sophisticated crawlers will follow dynamic urls as long as it is not too complex.

In addition, chech the file size of the generated pages. You don't want to end up with a single page much over 100k. Ideally, a lot under 100k in size.

Watch out for a spider trap. A site that feeds an infinite loop will cause the spider to eventually stop indexing altogether.

Check to see how much of your site is already indexed in the engines and look at how the link was followed from the home page. Sometimes that gives clues.

8:09 pm on Nov 23, 2003 (gmt 0)

Full Member

10+ Year Member

joined:Nov 17, 2003
votes: 0


Thank you, you may have lead me to my problem!

Thank again.