Forum Moderators: Robert Charlton & goodroi
Example:
www.widget.com/index.html?layout=1&color=red&something=1
www.widget.com/index.html?layout=2&color=black&otherparameter=1
this gives evidently virtually infinite posibilities of URLs. A significant part of the load on our server is though coming from googlebot, spidering virtually any of this URLs.
We thought now about including in this links a nofollow, to avoid this useless load, as 90% of the page is anyway the same, and if there is a parameter there is a Meta="follow,noindex" included.
BUT, having only about 100'000 genuine pages, and site:mysite.com shows about 800'000 pages, might that not have a significant impact, as Google likes aparently big sites?
Any ideas on this issue or experiences would be highly apreciated! We would like to get rid of this useless load, but not of our pages in Googles Index! ;)
Thanks a lot!
It could be that you are better off letting Google filter out whichever urls they choose to filter, as they currently must be doing. There may be no consistent pattern that you can discover, and then playing around with rel="nofollow" and robots meta tags would mess you up. Sometimes Google does a good job sorting things out.
As long as your site is thriving, then I would say don't risk dramatic changes, you know? But definitely keep this issue in mind. If things do start to get wobbly for your site, this would be one good place to start looking.