homepage Welcome to WebmasterWorld Guest from
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member
Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

robots and dynamic url
How to ban robots from a dynamic url

10+ Year Member

Msg#: 203 posted 4:33 am on Nov 22, 2003 (gmt 0)

I have dynamically generated product pages (have 600 products) and on each page it lists the following:

1. A purchase button
2. A tell a friend button (loads a email script along with product info)
3. A add to wishlist button
4. Enlarge product image button
5. Complimentary product the customer may be interested in, say 5 or 6 for each product.

I have a problem in that the robots don't want to index my dynamic content and I think it may be because indexing product pages overwhelms it, yes? I mean your talking each product x at least 10 more links for a total of about 5400 links. Of course the complimentary products are repeats, but still it seems like to much.

Can I ban robots from a specific file, say tellafriend.asp or addwishlist.asp and does it matter if the file has a query string on the end of it?

Hopefully I explained this well enough, if I didn't let me know and I'll try and clarify.




WebmasterWorld Administrator engine us a WebmasterWorld Top Contributor of All Time 10+ Year Member Top Contributors Of The Month

Msg#: 203 posted 3:10 pm on Nov 23, 2003 (gmt 0)

The robots.txt file will enable you to control which spiders and which areas of your site are traversed and which ones are not. It does not allow you to block the link.

Use the Meta noindex or nofollow on specific pages you want to control.

With dynamic pages, part of the problem is the complexity of the url. The sophisticated crawlers will follow dynamic urls as long as it is not too complex.

In addition, chech the file size of the generated pages. You don't want to end up with a single page much over 100k. Ideally, a lot under 100k in size.

Watch out for a spider trap. A site that feeds an infinite loop will cause the spider to eventually stop indexing altogether.

Check to see how much of your site is already indexed in the engines and look at how the link was followed from the home page. Sometimes that gives clues.


10+ Year Member

Msg#: 203 posted 8:09 pm on Nov 23, 2003 (gmt 0)


Thank you, you may have lead me to my problem!

Thank again.

Global Options:
 top home search open messages active posts  

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved