Welcome to WebmasterWorld Guest from 54.145.58.37

Forum Moderators: goodroi

Message Too Old, No Replies

Crawling of non-linked pages

Over 18's area

     
4:43 pm on Nov 29, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 4, 2003
posts:63
votes: 0


Hi Our site has several hundred crawlable pages, and we see googlebot, slurp etc everyday..

I have created over 18's area for some of my more adult products which is accessible from an Over 18 checkbox which forwards the visitor to the over 18 area and sets a cookie to allow over 18 content.

Question is there are over 100 products and I would be mad not to have google and all the other bots crawl these pages.. but as no direct url links on the site exist how do I entice the bots to index these pages..

Thanks in advance..

Kind regards
TheLeveller

2:16 pm on Dec 1, 2004 (gmt 0)

Senior Member

WebmasterWorld Senior Member 10+ Year Member

joined:Feb 21, 2003
posts:2355
votes: 0


...how do I entice the bots to index these pages

Maybe some cloaking is in order to serve engines the 18+ content - you'd have to identify the visitor and redirect based on a list of useragents and IPs for each engine you wanted to target - even then, it can get you banned.

2:43 pm on Dec 1, 2004 (gmt 0)

Junior Member

10+ Year Member

joined:Feb 4, 2003
posts:63
votes: 0


I thought as much... dont want to get into cloaking...
I think the best way is to provide the links in my site map which wont be classified... at least the products will be crawled...

I was just attempting to protect the under 16's from some content, but try you best and looks like you get penalised... Oh well one for google to think on..

Cheers

 

Join The Conversation

Moderators and Top Contributors

Hot Threads This Week

Featured Threads

Free SEO Tools

Hire Expert Members