Forum Moderators: Robert Charlton & goodroi
While faceted navigation can be useful for users, by helping them find what they are looking for quickly, it can also be problematic when it comes to crawling. Each selectable option can generate a new URL on the website by adding parameters such as colour or size. For example, the following pages would look very similar to search engines:
***com/category
***com/category?colour=blue
***com/category?colour=blue&size=small
***com/category?colour=blue&size=small&order=price-asc
Having just 5 unique filtering options could generate as many as 3,125 different URLs per category page depending on how your faceted navigation is setup. Stopping search engines from crawling these additional pages is relatively simple, only requiring two changes:
Use nofollow on all facet links
Deploy meta robots or x-robots noindex, nofollow on all additional pages
Many recommends adding a noindex in robot.txtWhile noindex can be helpful, it does not belong in robots.txt. You can use a metatag such as:
<meta name="robots" content="noindex,noarchive"> in the html headers or as x-robots elements within directories or URL structures via .htaccess. If you block directories or URL structures in robots.txt then Google will not be able to know that they are noindexed. Make sure that noindexed URLs are not submitted in your sitemaps.