homepage Welcome to WebmasterWorld Guest from 184.72.69.79
register, free tools, login, search, pro membership, help, library, announcements, recent posts, open posts,
Become a Pro Member

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
Forum Library, Charter, Moderators: goodroi

Sitemaps, Meta Data, and robots.txt Forum

    
Crawling of non-linked pages
Over 18's area
theleveller




msg:1526880
 4:43 pm on Nov 29, 2004 (gmt 0)

Hi Our site has several hundred crawlable pages, and we see googlebot, slurp etc everyday..

I have created over 18's area for some of my more adult products which is accessible from an Over 18 checkbox which forwards the visitor to the over 18 area and sets a cookie to allow over 18 content.

Question is there are over 100 products and I would be mad not to have google and all the other bots crawl these pages.. but as no direct url links on the site exist how do I entice the bots to index these pages..

Thanks in advance..

Kind regards
TheLeveller

 

bcolflesh




msg:1526881
 2:16 pm on Dec 1, 2004 (gmt 0)

...how do I entice the bots to index these pages

Maybe some cloaking is in order to serve engines the 18+ content - you'd have to identify the visitor and redirect based on a list of useragents and IPs for each engine you wanted to target - even then, it can get you banned.

theleveller




msg:1526882
 2:43 pm on Dec 1, 2004 (gmt 0)

I thought as much... dont want to get into cloaking...
I think the best way is to provide the links in my site map which wont be classified... at least the products will be crawled...

I was just attempting to protect the under 16's from some content, but try you best and looks like you get penalised... Oh well one for google to think on..

Cheers

Global Options:
 top home search open messages active posts  
 

Home / Forums Index / Search Engines / Sitemaps, Meta Data, and robots.txt
rss feed

All trademarks and copyrights held by respective owners. Member comments are owned by the poster.
Home ¦ Free Tools ¦ Terms of Service ¦ Privacy Policy ¦ Report Problem ¦ About ¦ Library ¦ Newsletter
WebmasterWorld is a Developer Shed Community owned by Jim Boykin.
© Webmaster World 1996-2014 all rights reserved