Welcome to WebmasterWorld Guest from

Forum Moderators: goodroi

Message Too Old, No Replies

Crawling of non-linked pages

Over 18's area



4:43 pm on Nov 29, 2004 (gmt 0)

10+ Year Member

Hi Our site has several hundred crawlable pages, and we see googlebot, slurp etc everyday..

I have created over 18's area for some of my more adult products which is accessible from an Over 18 checkbox which forwards the visitor to the over 18 area and sets a cookie to allow over 18 content.

Question is there are over 100 products and I would be mad not to have google and all the other bots crawl these pages.. but as no direct url links on the site exist how do I entice the bots to index these pages..

Thanks in advance..

Kind regards


2:16 pm on Dec 1, 2004 (gmt 0)

WebmasterWorld Senior Member 10+ Year Member

...how do I entice the bots to index these pages

Maybe some cloaking is in order to serve engines the 18+ content - you'd have to identify the visitor and redirect based on a list of useragents and IPs for each engine you wanted to target - even then, it can get you banned.


2:43 pm on Dec 1, 2004 (gmt 0)

10+ Year Member

I thought as much... dont want to get into cloaking...
I think the best way is to provide the links in my site map which wont be classified... at least the products will be crawled...

I was just attempting to protect the under 16's from some content, but try you best and looks like you get penalised... Oh well one for google to think on..



Featured Threads

Hot Threads This Week

Hot Threads This Month