Crawling of non-linked pages

Over 18's area

[theleveller]

Hi Our site has several hundred crawlable pages, and we see googlebot, slurp etc everyday..

I have created over 18's area for some of my more adult products which is accessible from an Over 18 checkbox which forwards the visitor to the over 18 area and sets a cookie to allow over 18 content.

Question is there are over 100 products and I would be mad not to have google and all the other bots crawl these pages.. but as no direct url links on the site exist how do I entice the bots to index these pages..

Thanks in advance..

Kind regards
TheLeveller

[bcolflesh]

...how do I entice the bots to index these pages

Maybe some cloaking is in order to serve engines the 18+ content - you'd have to identify the visitor and redirect based on a list of useragents and IPs for each engine you wanted to target - even then, it can get you banned.

[theleveller]

I thought as much... dont want to get into cloaking...
I think the best way is to provide the links in my site map which wont be classified... at least the products will be crawled...

I was just attempting to protect the under 16's from some content, but try you best and looks like you get penalised... Oh well one for google to think on..

Cheers